DRBD + Pacemaker + SSHFS on CentOS 7 #1 – DRBD

VIP – 10.19.10.195

Master 서버
eth0 – none (VIP용)
eth1 – 10.19.10.196

Slave 서버
eth0 – none (VIP용)
eth1 – 10.19.10.197

// NIC을 eth0, eth1 두개로 나눈 이유는 편의상 복제 트래픽과 서비스 트래픽을 분리하기 위함이므로 반드시 NIC을 나눌 필요는 없다.
// 위 설정에 따라 pacemaker의 cib 설정도 바뀔수 있다.

DRBD 구성

Master / Slave 모두 적용

1. data 디스크 생성
// lvm 으로 생성하되 파티션 포맷은 하지 않는다.

[root@10-19-10-196 /]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xaf4647e8.

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 1048576 bytes
Disk label type: dos
Disk identifier: 0xaf4647e8

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-20971519, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):  
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 1048576 bytes
Disk label type: dos
Disk identifier: 0xaf4647e8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    20971519    10484736   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@10-19-10-196 /]# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created.

[root@10-19-10-196 /]# vgcreate vg_data /dev/sdb1
  Volume group "vg_data" successfully created

[root@10-19-10-196 /]# lvcreate -n lv_data -l 100%FREE vg_data
  Logical volume "lv_data" created.

[root@10-19-10-196 /]# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda2  centos  lvm2 a--   <9.00g 4.00m
  /dev/sdb1  vg_data lvm2 a--  <10.00g    0 

[root@10-19-10-196 /]# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  centos    1   2   0 wz--n-  <9.00g 4.00m
  vg_data   1   1   0 wz--n- <10.00g    0 

[root@10-19-10-196 /]# lvs
  LV      VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root    centos  -wi-ao----   6.99g                                                    
  swap    centos  -wi-ao----   2.00g                                                    
  lv_data vg_data -wi-a----- <10.00g

 

2. 의존성 패키지 설치

[root@10-19-10-196 /]# yum install -y gcc gcc-c++ make autoconf openssl openssl-devel kernel kernel-devel patch flex libxslt po4a

 

3. drbd 설치

[root@10-19-10-196 /]# cd /usr/local/src
[root@10-19-10-196 src]# wget https://www.linbit.com/downloads/drbd/9.0/drbd-9.0.22-1.tar.gz
[root@10-19-10-196 src]# tar xvzf drbd-9.0.22-1.tar.gz
[root@10-19-10-196 src]# cd drbd-9.0.22-1
[root@10-19-10-196 drbd-9.0.22-1]# make KDIR=/usr/src/kernels/$(uname -r)
[root@10-19-10-196 drbd-9.0.22-1]# make install
[root@10-19-10-196 drbd-9.0.22-1]# modprobe drbd
[root@10-19-10-196 drbd-9.0.22-1]# cat /proc/drbd
[root@10-19-10-196 drbd-9.0.22-1]# echo drbd > /etc/modules-load.d/drbd.conf

 

4. drbd-util 설치

[root@10-19-10-196 drbd-9.0.22-1]# cd /usr/local/src
[root@10-19-10-196 src]# wget https://www.linbit.com/downloads/drbd/utils/drbd-utils-9.12.0.tar.gz
[root@10-19-10-196 src]# tar xvzf drbd-utils-9.12.0.tar.gz
[root@10-19-10-196 src]# cd drbd-utils-9.12.0
[root@10-19-10-196 drbd-utils-9.12.0]# ./configure
[root@10-19-10-196 drbd-utils-9.12.0]# make
[root@10-19-10-196 drbd-utils-9.12.0]# make install

 

5. 설정파일 링크

[root@10-19-10-196 drbd-9.0.22-1]# ln -s /usr/local/etc/drbd.conf /etc/drbd.conf
[root@10-19-10-196 drbd-9.0.22-1]# ln -s /usr/local/etc/drbd.d /etc/drbd.d

 

6. drbd 설정
// 호스트네임은 반드시 [uname -n] 커맨드로 확인된 호스트네임으로만 인식한다.
// drbd0은 가상 device 이며, 위에서 생성한 lvm 과 맵핑된다.
// meta-disk를 internal로 설정하면 drbd device내에 metadata를 저장하고 별도의 device(ex. /dev/sdc1)로 저장하기 원한다면 지정이 가능하다.
// metadata의 크기는 device 크기에 따라 다르다.

:vi /etc/drbd.d/file_storage.res

resource "file_storage" {
    protocol C;
    disk {
        on-io-error detach;
    }

    on 10-19-10-196 {
        device    /dev/drbd0;
        disk      /dev/mapper/vg_data-lv_data;
        address   10.19.10.196:7788;
        meta-disk internal;
    }

    on 10-19-10-197 {
        device    /dev/drbd0;
        disk      /dev/mapper/vg_data-lv_data;
        address   10.19.10.197:7788;
        meta-disk internal;
    }
}

 

4. drbd metadata 생성
// metadata를 생성하면 res 파일에 설정되어 있는 device 값을 참조하여 /dev/drbd0이 생성된다.

[root@10-19-10-196 /]# drbdadm create-md file_storage --force
[root@10-19-10-196 /]# drbdadm up file_storage
[root@10-19-10-196 /]# mkdir /data

 

 

Master만 적용

1. drbd primary 설정

[root@10-19-10-196 /]# drbdadm primary file_storage --force
[root@10-19-10-196 /]# drbdadm -- --overwrite-data-of-peer primary file_storage

 

2. 디스크 포맷 및 마운트

[root@10-19-10-196 /]# mkfs.xfs /dev/drbd0
[root@10-19-10-196 /]# mount /dev/drbd0 /data

// Slave 는 디스크가 마운트되지 않기 때문에 복제가 잘 되고 있는지 확인하기 위해서는 커맨드를 통해 상태확인만 가능하다.
// drbdadm 커맨드는 리소스 네임대신 all 인자를 주어서 모든 리소스에 대한 제어가 가능하다.
// ex) drbdadm up all, drbdadm down all, drbdadm status all …

[root@10-19-10-196 /]# cat /proc/drbd
version: 9.0.22-1 (api:2/proto:86-116)
GIT-hash: 449d6bf22b01af7d14a297a4ed3e281aa84c94a5 build by root@10-19-10-196, 2020-03-20 14:58:43
Transports (api:16): tcp (9.0.22-1)

[root@10-19-10-197 /]# cat /proc/drbd
version: 9.0.22-1 (api:2/proto:86-116)
GIT-hash: 449d6bf22b01af7d14a297a4ed3e281aa84c94a5 build by root@10-19-10-197, 2020-03-20 14:58:43
Transports (api:16): tcp (9.0.22-1)

[root@10-19-10-196 /]# drbdadm status file_storage
file_storage role:Primary
  disk:UpToDate
  10-19-10-197 role:Secondary
    peer-disk:UpToDate

[root@10-19-10-197 /]# drbdadm status file_storage
file_storage role:Secondary
  disk:UpToDate
  10-19-10-196 role:Primary
    peer-disk:UpToDate

 

TIP 1. drbd master / slave 교체 방법

// 위에서 언급했듯이 Slave 의 경우 복제여부를 눈으로 직접 보기 힘들기 때문에 아래와 같이 master 와 slave를 교체하여 복제가 잘 되는지 확인 할 수있다.

1. master 를 slave로 변경
umount /data
drbdadm secondary file_storage

2. slave를 master로 변경
drbdadm primary file_storage
mount /dev/drbd0 /data

 

TIP 2. metadata 크기를 계산하는 스크립트
// 출처 : https://serverfault.com/questions/433999/calculating-drbd-meta-size
// 아래 내용으로 스크립트파일을 만들고 실행시에 인수값으로 device를 입력해 준다.
// bc 패키지가 설치되어 있어야 한다.
// ex) metacalc.sh /dev/mapper/vg_data-lv_data

#!/bin/bash

which bc >/dev/null 2>&1
if [ ! $? -eq 0 ]; then
    echo "Error: bc is not installed"
    exit 1
fi

if [ $# -lt 1 ]; then
    echo "Error: Please supply block device path"
    echo "Eg. /dev/vg1/backups"
    exit 1
fi

DEVICE=$1

SECTOR_SIZE=$( blockdev --getss $DEVICE )
SECTORS=$( blockdev --getsz $DEVICE )
MD_SIZE=$( echo "((($SECTORS + (2^18)-1) / 262144 * 8) + 72)" | bc )
FS_SIZE=$( echo "$SECTORS - $MD_SIZE" | bc )

MD_SIZE_MB=$( echo "($MD_SIZE / 4 / $SECTOR_SIZE) + 1" | bc )
FS_SIZE_MB=$( echo "($FS_SIZE / 4 / $SECTOR_SIZE)" | bc )

echo "Filesystem: $FS_SIZE_MB MiB"
echo "Filesystem: $FS_SIZE Sectors"
echo "Meta Data:  $MD_SIZE_MB MiB"
echo "Meta Data:  $MD_SIZE Sectors"
echo "--"
echo "Resize commands: resize2fs -p "$DEVICE $FS_SIZE_MB"M; drbdadm create-md res"
[root@10-19-10-196 src]# bash metacalc.sh /dev/mapper/vg_data-lv_data 
Filesystem: 511980 MiB
Filesystem: 1048535736 Sectors
Meta Data:  16 MiB
Meta Data:  32072 Sectors
--
Resize commands: resize2fs -p /dev/mapper/vg_data-lv_data 511980M; drbdadm create-md res

 

You may also like...