lost and found ( for me ? )

CentOS 6 DRBD : how to set up DRBD


Here is an explanation of how to set up DRBD on CentOS6.
# cat /etc/redhat-release
CentOS release 6.2 (Final)

# uname -ri
2.6.32-220.7.1.el6.i686 i386

# rpm -qa | grep drbd
kmod-drbd84-8.4.1-1.el6.elrepo.i686
drbd84-utils-8.4.1-1.el6.elrepo.i686


Hostname ( uname –n ) of DBRD nodes are:
centos6-drbd1.localdomain
centos6-drbd2.localdomain

These nodes are running as VMs on a KVM host machine ( Linux Mint 12 )

[ install DRBD on CentOS 6 ]

CentOS 6 doesn’t seem to have DRBD repo.

add the DRBD repo
[root@centos6-drbd1 ~]# rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm


edit the DRBD repo.
[root@centos6-drbd1 ~]# vi /etc/yum.repos.d/elrepo.repo

[elrepo]
name=ELRepo.org Community Enterprise Linux Repository - el6
baseurl=http://elrepo.org/linux/elrepo/el6/$basearch/
mirrorlist=http://elrepo.org/mirrors-elrepo.el6
#enabled=1
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0


confirm you can search DRBD packages via yum
[root@centos6-drbd1 ~]# yum --enablerepo=elrepo search drbd
Loaded plugins: fastestmirror, refresh-packagekit
Loading mirror speeds from cached hostfile
* base: www.ftp.ne.jp
* elrepo: elrepo.org
* extras: www.ftp.ne.jp
* updates: www.ftp.ne.jp
elrepo                                                   | 1.9 kB     00:00
elrepo/primary_db                                        | 409 kB     00:01
============================== N/S Matched: drbd ===============================
drbd83-utils.i686 : Management utilities for DRBD %{version}
drbd84-utils.i686 : Management utilities for DRBD
kmod-drbd83.i686 : drbd83 kernel module(s)
kmod-drbd84.i686 : drbd84 kernel module(s)


install drbd
[root@centos6-drbd1 ~]# yum --enablerepo=elrepo install –y drbd84-utils.i686 kmod-d
rbd84.i686

<snip>
Installed:
 drbd84-utils.i686 0:8.4.1-1.el6.elrepo  kmod-drbd84.i686 0:8.4.1-1.el6.elrepo

Complete!


do the same things on the other machine

[ when not using LVM ]

Before setting up DRBD , I’m going to add a storage for DRBD to both machines which are running on KVM.

- create a storage with kvm-image for DRBD nodes

for centos6-drbd1
mint-1 images # qemu-img create -f qcow2 /var/disk1/libvirt/images/centos6-drbd1
-1.img 500M
Formatting '/var/disk1/libvirt/images/centos1-drbd1-1.img', fmt=qcow2 size=524288000 encryption=off cluster_size=0


for centos6-drbd2
mint-1 images # qemu-img create -f qcow2 /var/disk1/libvirt/images/centos6-drbd2-1.img 500M
Formatting '/var/disk1/libvirt/images/centos1-drbd2-1.img', fmt=qcow2 size=524288000 encryption=off cluster_size=0


I have turned off apparomor on the KVM host.

add the centos6-drbd1-1.img to the DRBD machine centos6-drbd1 on the fly

on the KVM host
virsh # qemu-monitor-command centos6-32-drbd1 'pci_add auto storage file=/var/disk1/libvirt/images/centos6-drbd1-1.img,if=scsi'
OK domain 0, bus 0, slot 7, function 0


on the KVM host
virsh # qemu-monitor-command centos6-32-drbd1 'info block'
drive-virtio-disk0: type=hd removable=0 file=/var/disk1/libvirt/images/centos6-32-drbd1.img ro=0 drv=raw encrypted=0
scsi0-hd0: type=hd removable=0 file=/var/disk1/libvirt/images/centos6-drbd1-1.img ro=0 drv=qcow2 encrypted=0


on the VM ( dmesg )
/dev/sda has been added
sd 2:0:0:0: [sda] Attached SCSI disk
sd 2:0:0:0: Attached scsi generic sg0 type 0


do the same things on the VM centos6-drbd2

- add a storage device on both DRBD nodes.

on the KVM host , add the storage to centos6-32-drbd2 VM on the fly.
virsh # qemu-monitor-command centos6-32-drbd2 'pci_add auto storage file=/var/di
sk1/libvirt/images/centos6-drbd2-1.img,if=scsi'
OK domain 0, bus 0, slot 7, function 0

virsh # qemu-monitor-command centos6-32-drbd2 'info block'
drive-virtio-disk0: type=hd removable=0 file=/var/disk1/libvirt/images/centos6-32-drbd2.img ro=0 drv=raw encrypted=0
scsi0-hd0: type=hd removable=0 file=/var/disk1/libvirt/images/centos6-drbd2-1.img ro=0 drv=qcow2 encrypted=0


make a partition for DRBD on both VMs with fdisk.
[root@centos6-drbd1 ~]# LANG=C fdisk -l | grep sda
Disk /dev/sda: 524 MB, 524288000 bytes
/dev/sda1               1        1020      511500+  83  Linux


Sample config has been installed under :
[root@centos6-drbd1 ~]# head -10 /usr/share/doc/drbd84-utils-8.4.1/drbd.conf.example
resource example {
       options {
               on-no-data-accessible suspend-io;
       }

       net {
               cram-hmac-alg "sha1";
               shared-secret "secret_string";
       }


make a config file for DRBD , called disk0.res under /etc/drbd.d/ directory.

drbd.conf file reads the following files
[root@centos6-drbd1 ~]# cat /etc/drbd.conf
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";

[root@centos6-drbd1 ~]# uname -n
centos6-drbd1.localdomain

[root@centos6-drbd1 ~]# ifconfig eth0 | grep "inet addr" | awk '{print $2}'
addr:192.168.10.50

[root@centos6-drbd2 ~]# uname -n
centos6-drbd2.localdomain
[root@centos6-drbd2 ~]# ifconfig eth0 | grep "inet addr" | awk '{print $2}'
addr:192.168.10.60


/etc/drbd.d/disk0.res file.
[root@centos6-drbd1 ~]# cat /etc/drbd.d/disk0.res
resource disk0 {
       protocol C;
       net {
               cram-hmac-alg sha1;
               shared-secret "test";
       }

on centos6-drbd1.localdomain {
       device /dev/drbd0;
       disk /dev/sda1;
       address 192.168.10.50:7788; # centos6-drbd1’s IP
       meta-disk internal;
}

on centos6-drbd2.localdomain {
       device /dev/drbd0;
       disk /dev/sda1;
       address 192.168.10.60:7788; # centos6-drbd2’s IP
       meta-disk internal;
}


copy same config to centos6-drbd2 node.
[root@centos6-drbd1 ~]# scp /etc/drbd.d/disk0.res root@192.168.10.60:/etc/drbd.d/disk0.res


- create meta data on both nodes

on centos6-drbd1
[root@centos6-drbd1 ~]# drbdadm create-md disk0

 --==  Thank you for participating in the global usage survey  ==--
The server's response is:

you are the 2135th user to install this version
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success


on centos6-drbd2
[root@centos6-drbd2 ~]# drbdadm create-md disk0


- start drbd daemon on both nodes
[root@centos6-drbd1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [
    create res: disk0
  prepare disk: disk0
   adjust disk: disk0
    adjust net: disk0
]
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
  reboot the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
  expire after 0 seconds. [wfc-timeout]
  (These values are for resource 'disk0'; 0 sec -> wait forever)
To abort waiting enter 'yes' [  10]:
.
[root@centos6-drbd1 ~]# echo $?
0

[root@centos6-drbd2 ~]# /etc/init.d/drbd start
Starting DRBD resources: [
    create res: disk0
  prepare disk: disk0
   adjust disk: disk0
    adjust net: disk0
]
.
[root@centos6-drbd2 ~]# echo $?
0


check the status of DRBD
[root@centos6-drbd1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res    cs         ro                   ds                         p  mounted  fstype
0:disk0  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C

[root@centos6-drbd2 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res    cs         ro                   ds                         p  mounted  fstype
0:disk0  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C


status is secondary/secondary , inconsistent.

To solve this , have the centos6-drbd1 node primary.

on centos6-drbd1
[root@centos6-drbd1 ~]# drbdadm -- --overwrite-data-of-peer primary all


check the status
[root@centos6-drbd1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res    cs          ro                 ds                     p  mounted  fstype
0:disk0  SyncSource  Primary/Secondary  UpToDate/Inconsistent  C
...      sync'ed:    9.6%               (465368/511448)K


get things done
[root@centos6-drbd1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
3GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res    cs         ro                 ds                 p  mounted  fstype
0:disk0  Connected  Primary/Secondary  UpToDate/UpToDate  C


centos6-drbd1 is primary and centos6-drbd2 is secondary.
[root@centos6-drbd1 ~]# drbdadm role all
Primary/Secondary

[root@centos6-drbd2 ~]# drbdadm role all
Secondary/Primary


- create a filesystem

make a filesystem on primary node ( not secondary !! )
[root@centos6-drbd1 ~]# drbdadm role all
Primary/Secondary

[root@centos6-drbd1 ~]# mkfs.ext4 /dev/drbd0

[root@centos6-drbd1 ~]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
   ns:610747 nr:0 dw:99299 dr:517056 al:68 bm:32 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
   ns:0 nr:74335 dw:74335 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0


- test

mount /dev/drbd0 to a directory.

on primary node
[root@centos6-drbd1 ~]# mount -t ext4 /dev/drbd0 data/


on primary ( centos6-drbd1 )
[root@centos6-drbd1 ~]# echo hello > data/hello.txt
[root@centos6-drbd1 ~]# cat data/hello.txt
hello


unmount the directory on the primary
[root@centos6-drbd1 ~]# umount /root/data/


have the centos6-drbd1 secondary
Primary/Secondary
[root@centos6-drbd1 ~]# drbdadm secondary all

[root@centos6-drbd1 ~]# drbdadm role all
Secondary/Secondary


have the centos6-drbd2 primary
[root@centos6-drbd2 ~]# drbdadm role all
Secondary/Secondary
[root@centos6-drbd2 ~]# drbdadm primary all
[root@centos6-drbd2 ~]# drbdadm role all
Primary/Secondary


mount
[root@centos6-drbd2 ~]# mount -t ext4 /dev/drbd0 /root/data/
[root@centos6-drbd2 ~]# cat /root/data/hello.txt
hello


[ when using LVM ]

add two storage devices for LVM on both nodes.
mint-1 images # qemu-img create -f qcow2 centos6-drbd1-lvm1.img 100M
Formatting 'centos6-drbd1-lvm1.img', fmt=qcow2 size=104857600 encryption=off cluster_size=0

mint-1 images # qemu-img create -f qcow2 centos6-drbd1-lvm2.img 100M
Formatting 'centos6-drbd1-lvm2.img', fmt=qcow2 size=104857600 encryption=off cluster_size=0

mint-1 images # qemu-img create -f qcow2 centos6-drbd2-lvm1.img 100M
Formatting 'centos6-drbd2-lvm1.img', fmt=qcow2 size=104857600 encryption=off cluster_size=0

mint-1 images # qemu-img create -f qcow2 centos6-drbd2-lvm2.img 100M
Formatting 'centos6-drbd2-lvm2.img', fmt=qcow2 size=104857600 encryption=off cluster_size=0

mint-1 images # ls *lvm*
centos6-drbd1-lvm1.img  centos6-drbd2-lvm1.img
centos6-drbd1-lvm2.img  centos6-drbd2-lvm2.img


add these devices to DRBD nodes.

on KVM host , add two storages to centos6-drbd1 node.
virsh # qemu-monitor-command centos6-32-drbd1 'pci_add auto storage file=/var/disk1/libvirt/images/centos6-drbd1-lvm1.img,if=scsi'
OK domain 0, bus 0, slot 8, function 0

virsh # qemu-monitor-command centos6-32-drbd1 'pci_add auto storage file=/var/disk1/libvirt/images/centos6-drbd1-lvm2.img,if=scsi'
OK domain 0, bus 0, slot 9, function 0

virsh # qemu-monitor-command centos6-32-drbd2 'pci_add auto storage file=/var/di
sk1/libvirt/images/centos6-drbd2-lvm1.img,if=scsi'
OK domain 0, bus 0, slot 8, function 0


virsh # qemu-monitor-command centos6-32-drbd2 'pci_add auto storage file=/var/disk1/libvirt/images/centos6-drbd2-lvm2.img,if=scsi'
OK domain 0, bus 0, slot 9, function 0


/dev/sdb , /dev/sdc are for LVM
[root@centos6-drbd1 ~]# LANG=C fdisk -l | grep "/dev/sd*" | grep -v sda
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdb: 104 MB, 104857600 bytes
Disk /dev/sdc: 104 MB, 104857600 bytes


- create LVM partitions

on centos6-drbd1
[root@centos6-drbd1 ~]# fdisk /dev/sdb
Command (m for help): n
Command action
  e   extended
  p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1024, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1024, default 1024):
Using default value 1024

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sdb: 104 MB, 104857600 bytes
4 heads, 50 sectors/track, 1024 cylinders
Units = cylinders of 200 * 512 = 102400 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x518bd64a

  Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1024      102375   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


do the same thing to /dev/sdc
[root@centos6-drbd1 ~]# LANG=C fdisk -l | grep "/dev/sd*" | grep -v sda
Disk /dev/sdb: 104 MB, 104857600 bytes
/dev/sdb1               1        1024      102375   8e  Linux LVM
Disk /dev/sdc: 104 MB, 104857600 bytes
/dev/sdc1               1        1024      102375   8e  Linux LVM


make LVM partitions ( /dev/sdb , sdc ) on centos6-drbd2 as well.

centos6-drbd2
[root@centos6-drbd2 ~]# LANG=C fdisk -l | grep "/dev/sd*" | grep -v sda
Disk /dev/sdb: 104 MB, 104857600 bytes
/dev/sdb1               1        1024      102375   8e  Linux LVM
Disk /dev/sdc: 104 MB, 104857600 bytes
/dev/sdc1               1        1024      102375   8e  Linux LVM


- create PV ( physical volume )

there’s no pvcreate command.
install lvm2 via yum
[root@centos6-drbd1 ~]# yum install -y lvm2
[root@centos6-drbd2 ~]# yum install -y lvm2


on centos6-drbd1
[root@centos6-drbd1 ~]# pvcreate /dev/sdb1 /dev/sdc1
 Writing physical volume data to disk "/dev/sdb1"
 Physical volume "/dev/sdb1" successfully created
 Writing physical volume data to disk "/dev/sdc1"
 Physical volume "/dev/sdc1" successfully created

[root@centos6-drbd1 ~]# pvscan
 PV /dev/sdb1                      lvm2 [99.98 MiB]
 PV /dev/sdc1                      lvm2 [99.98 MiB]
 Total: 2 [199.95 MiB] / in use: 0 [0   ] / in no VG: 2 [199.95 MiB]


do the same thing on centos6-drbd2
[root@centos6-drbd2 ~]# pvcreate /dev/sdb1 /dev/sdc1


- create VG ( volume group )

on centos6-drbd1
[root@centos6-drbd1 ~]# vgcreate -s 32m VolGroup01 /dev/sdb1 /dev/sdc1
 Volume group "VolGroup01" successfully created

[root@centos6-drbd1 ~]# vgscan
 Reading all physical volumes.  This may take a while...
 Found volume group "VolGroup01" using metadata type lvm2

[root@centos6-drbd1 ~]# vgdisplay
 --- Volume group ---
 VG Name               VolGroup01
 System ID
 Format                lvm2
 Metadata Areas        2
 Metadata Sequence No  1
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                0
 Open LV               0
 Max PV                0
 Cur PV                2
 Act PV                2
 VG Size               192.00 MiB
 PE Size               32.00 MiB
 Total PE              6
 Alloc PE / Size       0 / 0
 Free  PE / Size       6 / 192.00 MiB
 VG UUID               ZbTytp-s0st-HG2A-hB49-JuEX-f13I-829dgh


on centos6-drbd2
[root@centos6-drbd2 ~]#  vgcreate -s 32m VolGroup01 /dev/sdb1 /dev/sdc1
 Volume group "VolGroup01" successfully created


- create LV ( logical volume )

on centos6-drbd1
[root@centos6-drbd1 ~]# lvcreate -n LogVol01 -L 100M VolGroup01
 Rounding up size to full physical extent 128.00 MiB
 Logical volume "LogVol01" created

[root@centos6-drbd1 ~]# lvscan
 ACTIVE            '/dev/VolGroup01/LogVol01' [128.00 MiB] inherit

[root@centos6-drbd1 ~]# lvdisplay
 --- Logical volume ---
 LV Name                /dev/VolGroup01/LogVol01
 VG Name                VolGroup01
 LV UUID                0bCwiL-kMDE-sHyI-au1a-8Wwl-z67d-23KOkX
 LV Write Access        read/write
 LV Status              available
 # open                 0
 LV Size                128.00 MiB
 Current LE             4
 Segments               2
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:0



do the same thing on centos6-drbd2
[root@centos6-drbd2 ~]# lvcreate -n LogVol01 -L 100M VolGroup01
 Rounding up size to full physical extent 128.00 MiB
 Logical volume "LogVol01" created


- make ext4 filesystem on the LV

on centos6-drbd1
[root@centos6-drbd1 ~]# mkfs.ext4 /dev/VolGroup01/LogVol01


on centos6-drbd2
[root@centos6-drbd2 ~]# mkfs.ext4 /dev/VolGroup01/LogVol01

mount

on centos6-drbd1
[root@centos6-drbd1 ~]# mount -t ext4 /dev/VolGroup01/LogVol01 /root/lvm_mnt/

[root@centos6-drbd1 ~]# echo hello > /root/lvm_mnt/hello.txt
[root@centos6-drbd1 ~]# cat /root/lvm_mnt/hello.txt
hello

[root@centos6-drbd1 ~]# rm /root/lvm_mnt/hello.txt


If you want to manage LVM via GUI , you can use “system-config-lvm” which you can install via yum.
[root@centos6-drbd1 ~]# yum install system-config-lvm


start GUI tool
# system-config-lvm



- set up DRBD configuration

at first stop DRBD , unmount LV.

on centos6-drbd1
[root@centos6-drbd1 ~]# umount /root/lvm_mnt/
[root@centos6-drbd1 ~]# /etc/init.d/drbd stop


on centos6-drbd2
[root@centos6-drbd2 ~]# /etc/init.d/drbd stop


make a DRBD cofnig file for this LVM.

on centos6-drbd1
[root@centos6-drbd1 ~]# vi /etc/drbd.d/lvm_disk1.res

[root@centos6-drbd1 ~]# cat /etc/drbd.d/lvm_disk1.res
resource lvm_disk1 {
       protocol C;
       net {
               cram-hmac-alg sha1;
               shared-secret "test";
       }

on centos6-drbd1.localdomain {
       device /dev/drbd1;
       disk /dev/VolGroup01/LogVol01;
       address 192.168.10.50:7788; # centos6-drbd1's IP
       meta-disk internal;
}

on centos6-drbd2.localdomain {
       device /dev/drbd1;
       disk /dev/VolGroup01/LogVol01;
       address 192.168.10.60:7788; # centos6-drbd2's IP
       meta-disk internal;
}


copy it to centos6-drbd2
[root@centos6-drbd1 ~]# scp /etc/drbd.d/lvm_disk1.res root@192.168.10.60:/etc/dr
bd.d/lvm_disk1.res


nnn , error
[root@centos6-drbd1 ~]# drbdadm create-md lvm_disk1
drbd.d/lvm_disk1.res:11: conflicting use of IP '192.168.10.50:7788' ...
drbd.d/disk0.res:11: IP '192.168.10.50:7788' first used here.


I need to change the port # per DRBD resource.

Okay , I’ll use 7789 for this LVM’s DRBD

confirm 7789 is not used
[root@centos6-drbd1 ~]# lsof -ni:7789
[root@centos6-drbd1 ~]#

[root@centos6-drbd1 ~]# cat /etc/drbd.d/lvm_disk1.res
resource lvm_disk1 {
       protocol C;
       net {
               cram-hmac-alg sha1;
               shared-secret "test";
       }

on centos6-drbd1.localdomain {
       device /dev/drbd1;
       disk /dev/VolGroup01/LogVol01;
#        address 192.168.10.50:7788; # centos6-drbd1's IP
       address 192.168.10.50:7789; # centos6-drbd1's IP
       meta-disk internal;
}

on centos6-drbd2.localdomain {
       device /dev/drbd1;
       disk /dev/VolGroup01/LogVol01;
#        address 192.168.10.60:7788; # centos6-drbd2's IP
       address 192.168.10.60:7789; # centos6-drbd2's IP
       meta-disk internal;
}


copy it to centos6-drbd2
[root@centos6-drbd1 ~]# !877
scp /etc/drbd.d/lvm_disk1.res root@192.168.10.60:/etc/drbd.d/lvm_disk1.res
root@192.168.10.60's password:


try again.

nnn , erro ,, erro code is 40.
[root@centos6-drbd1 ~]# drbdadm create-md lvm_disk1
md_offset 134213632
al_offset 134180864
bm_offset 134176768

Found ext3 filesystem
     131072 kB data area apparently used
     131032 kB left usable by current configuration

Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
  * use external meta data (recommended)
  * shrink that filesystem first
  * zero out the device (destroy the filesystem)
Operation refused.

Command 'drbdmeta 1 v08 /dev/VolGroup01/LogVol01 internal create-md' terminated with exit code 40


zero out the LV.
[root@centos6-drbd1 ~]# dd if=/dev/zero of=/dev/VolGroup01/LogVol01 bs=1M count=1

[root@centos6-drbd2 ~]# dd if=/dev/zero of=/dev/VolGroup01/LogVol01 bs=1M count=1


try again

on centos-drbd1
[root@centos6-drbd1 ~]# drbdadm create-md lvm_disk1
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success


do the same thing on centos-drbd2
[root@centos6-drbd2 ~]# drbdadm create-md lvm_disk1


- start DRBD on both nodes
[root@centos6-drbd1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [
    create res: disk0 lvm_disk1
  prepare disk: disk0 lvm_disk1
   adjust disk: disk0 lvm_disk1
    adjust net: disk0 lvm_disk1
]
.........
[root@centos6-drbd1 ~]#

[root@centos6-drbd1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res        cs         ro                   ds                         p  mounted  fstype
0:disk0      Connected  Secondary/Secondary  UpToDate/UpToDate          C
1:lvm_disk1  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C

[root@centos6-drbd1 ~]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
   ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
   ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:131032


on centos6-drbd1 ( initialize DRBD disk )
[root@centos6-drbd1 ~]# drbdadm -- --overwrite-data-of-peer primary all

[root@centos6-drbd1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res        cs         ro                 ds                 p  mounted  fstype
0:disk0      Connected  Primary/Secondary  UpToDate/UpToDate  C
1:lvm_disk1  Connected  Primary/Secondary  UpToDate/UpToDate  C

[root@centos6-drbd1 ~]# drbdadm role all
Primary/Secondary
Primary/Secondary

[root@centos6-drbd1 ~]# drbd-overview
 0:disk0/0      Connected Primary/Secondary UpToDate/UpToDate C r-----
 1:lvm_disk1/0  Connected Primary/Secondary UpToDate/UpToDate C r----- /root/lvm_mnt ext4 124M 5.6M 113M 5%

[root@centos6-drbd2 ~]# drbdadm role all
Secondary/Primary
Secondary/Primary

[root@centos6-drbd2 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by dag@Build32R6, 2011-12-21 06:07:17
m:res        cs         ro                 ds                 p  mounted  fstype
0:disk0      Connected  Secondary/Primary  UpToDate/UpToDate  C
1:lvm_disk1  Connected  Secondary/Primary  UpToDate/UpToDate  C


- make filesystem on the primary node
[root@centos6-drbd1 ~]# mkfs.ext4 /dev/drbd1


mount
[root@centos6-drbd1 ~]# mount -t ext4 /dev/drbd1 /root/lvm_mnt/


make a file
[root@centos6-drbd1 ~]# echo "hi" > /root/lvm_mnt/hi.txt

[root@centos6-drbd1 ~]# cat /root/lvm_mnt/hi.txt
hi


unmount
[root@centos6-drbd1 ~]# umount /root/lvm_mnt/


make centos6-drbd1 secondary
[root@centos6-drbd1 ~]# drbdadm secondary all

[root@centos6-drbd1 ~]# drbdadm role all
Secondary/Secondary
Secondary/Secondary


make centos6-drbd2 priamry

on centos6-drbd2
[root@centos6-drbd2 ~]# drbdadm role all
Secondary/Secondary
Secondary/Secondary
[root@centos6-drbd2 ~]# drbdadm primary all
[root@centos6-drbd2 ~]#

[root@centos6-drbd2 ~]# drbd-overview
 0:disk0/0      Connected Primary/Secondary UpToDate/UpToDate C r-----
 1:lvm_disk1/0  Connected Primary/Secondary UpToDate/UpToDate C r-----

[root@centos6-drbd2 ~]# mkdir /root/lvm_mnt

[root@centos6-drbd2 ~]# mount -t ext4 /dev/drbd1 /root/lvm_mnt/

[root@centos6-drbd2 ~]# ls /root/lvm_mnt/
hi.txt  lost+found
[root@centos6-drbd2 ~]# cat /root/lvm_mnt/hi.txt
hi


disable resources
[root@centos6-drbd1 ~]# drbd-overview
 0:disk0/0      Connected Secondary/Primary UpToDate/UpToDate C r-----
 1:lvm_disk1/0  Connected Secondary/Primary UpToDate/UpToDate C r-----

[root@centos6-drbd1 ~]# drbdadm down disk0
[root@centos6-drbd1 ~]# drbdadm down lvm_disk1

[root@centos6-drbd1 ~]# drbd-overview
 0:disk0/0      Unconfigured . . . .
 1:lvm_disk1/0  Unconfigured . . . .

1 comment:

  1. Thanks, a very nice tutorial for us that does not work with drbd on a daily basis. After following this Iḿ off to read up a bit on pacemaker.

    Cheers!
    Claes

    ReplyDelete

Note: Only a member of this blog may post a comment.