lost and found ( for me ? )

Openfiler 2.99 : Can’t configure active/passive redundancy with peacemaker , corosync and drbd ..


I referred the following instructions. ( many thx XD )

[ install Openfiler on KVM ]

At first , prepare two openfiler nodes.
I installed openfiler as VMs on KMV.

Host OS ( KVM host )
# tail -1 /etc/lsb-release
DISTRIB_DESCRIPTION="Linux Mint 12 Lisa"

# uname -r
3.0.0-16-generic

# libvirtd --version
libvirtd (libvirt) 0.9.2

# kvm --version
QEMU emulator version 0.14.1 (qemu-kvm-0.14.1), Copyright (c) 2003-2008 Fabrice Bellard


Seen from openfiler’s documentation , it seems that open filer is rPath linux base.
OS : Linux
Version : Generic 2.6.x kernel

filer01.localdomiain ( uname –n )
eth0 : 192.168.100.10
eth1 : 192.168.101.10 ( heart beat segment )

filer02.loccaldomain ( uname –n )
eth0 : 192.168.100.20
eth2 : 192.168.101.20 ( heart beat segment )

VIP : 192.168.100.100

I prepared the filer01 VM.
To make filer02 VM , it’s easy to make a clone VM from filer01 by virt-clone.

The original VM filer01 has three disks.
# ls /var/disk1/libvirt/images/filer01*
/var/disk1/libvirt/images/filer01-1.img  <- meta data
/var/disk1/libvirt/images/filer01.img <- system
/var/disk1/libvirt/images/filer01-2.img <- data


When cloning a VM ( filer02 ) , specify multiple disks with –f option.
# virt-clone -o filer01 -n filer02 -f /var/disk1/libvirt/images/filer02.img -f /var/disk1/libvirt/images/filer02-1.img -f /var/disk1/libvirt/images/filer02-2.img

# LANG=C virsh list --all | grep filer
 - filer01              shut off
 - filer02              shut off


before starting filer02 VM , change hostname and IP address by guestfish.

mount filer02’s diks
# guestfish -i -d filer02

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
     'man' to read the manual
     'quit' to quit the shell

/dev/vda1 mounted on /


change IP address and MAC address
><fs> edit /etc/sysconfig/network-scripts/ifcfg-eth0
><fs> edit /etc/sysconfig/network-scripts/ifcfg-eth1


You can check the clone VM’s MAC from *.xml file.
# LANG=C virsh dumpxml filer02 | grep -i "mac addr"


change hostname
><fs> edit /etc/hosts
><fs> edit /etc/sysconfig/network


change MAC address from the original VM’s to the clone VM’s
><fs> edit /etc/udev/rules.d/70-persistent-net.rules
<fs> quit


[ set up active/passive redundancy ]

Okay , let’s start two open filer nodes.
virsh # start filer01
virsh # start filer02


Here is my test environment
filer01
eth0 : 192.168.100.10
eth1 : 192.168.101.10 ( heart beat segment )
/dev/sda1 ( system ) : 4.5GB
/dev/sda2 ( swap ) : 0.5GB
/dev/sdb1 ( meta ) : 512MB
/dev/sdc1 ( data ) : 1GB

filer02
eth0 : 192.168.100.20
eth2 : 192.168.101.20 ( heart beat segment )
/dev/sda1 ( system ) : 4.5GB
/dev/sda2 ( swap ) : 0.5GB
/dev/sdb1 ( meta ) : 512MB
/dev/sdc1 ( data ) : 1GB


   Clients
      |
   L2SW
    |   |  service segment
filer01   filer02
    |   |  heart beat segment
    L2SW

add the hostname and the IP address of heart beat segment IP in /etc/hosts for easier access

on filer01 , register filer02’s IP
# grep filer02 /etc/hosts
192.168.101.20 filer02.localdomain


on filer02
# egrep filer01 /etc/hosts
192.168.101.10 filer01.localdomain


make SSH keys and exchange that to make filer01/filer02 exchange files without password authentication.

on filer01
# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.


on filer02
# ssh-keygen -t dsa


exchange SSH keys.

on filer01
# scp /root/.ssh/id_dsa.pub root@filer02.localdomain:/root/.ssh/authorized_keys


on filer02
# scp /root/.ssh/id_dsa.pub root@filer01.localdomain:/root/.ssh/authorized_keys


create partitions for meta and data

do the same thing on both filer01 and filer02

make the partition for meta ( /dev/sdb1 )
# fdisk /dev/sdb

Disk /dev/sdb: 536 MB, 536870912 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb134de93

  Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
  e   extended
  p   primary partition (1-4)
p
Partition number (1-4, default 1): 1
First sector (2048-1048575, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1048575, default 1048575):
Using default value 1048575

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


make the partition for data ( /dev/sdc1 )
# fdisk /dev/sdc
Command (m for help): n
Command action
  e   extended
  p   primary partition (1-4)
p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Please note that I have not mounted the partition for meta and data yet.
# mount
/dev/sda1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)


edit /etc/drbd.conf on both nodes.
Both servers have the same drbd.conf.
# cat /etc/drbd.conf
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";

# meta
resource meta {
device /dev/drbd_meta minor 0;
disk /dev/sdc1;
meta-disk internal;
protocol C;
syncer {
rate 100M;
}
disk {
on-io-error pass_on;
}
on filer01.localdomain {
address 192.168.101.10:7788; # filer01’s IP
}
on filer02.localdomain {
address 192.168.101.20:7788; # filer02's IP
}
}

# data
resource data {
device /dev/drbd_data minor 1;
disk /dev/sdb1;
meta-disk internal;
protocol C;
syncer {
rate 100M;
after "meta";
}
disk {
on-io-error pass_on;
}
on filer01.localdomain {
address 192.168.101.10:7789; # filer01's IP
}
on filer02.localdomain {
address 192.168.101.20:7789; # filer02's IP
}
}


check both 7788 and 7789 ports are unused by other processes.
# lsof -ni:7789
# lsof -ni:7788


copy drbd.conf to filer02 from filer01
# scp /etc/drbd.conf root@filer02.localdomain:/etc/drbd.conf


check the syntax
# drbdadm dump all
# /etc/drbd.conf
common {
   protocol               C;
   handlers {
       pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
       pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
       local-io-error   "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
   }
}

# resource meta on filer01.localdomain: not ignored, not stacked
resource meta {
   protocol               C;
   on filer01.localdomain {
       device           /dev/drbd_meta minor 0;
       disk             /dev/sdc1;
       address          ipv4 192.168.101.10:7788;
       meta-disk        internal;
   }
   on filer02.localdomain {
       device           /dev/drbd_meta minor 0;
       disk             /dev/sdc1;
       address          ipv4 192.168.101.20:7788;
       meta-disk        internal;
   }
   disk {
       on-io-error      pass_on;
   }
   syncer {
       rate             100M;
   }
}

# resource data on filer01.localdomain: not ignored, not stacked
resource data {
   protocol               C;
   on filer01.localdomain {
       device           /dev/drbd_data minor 1;
       disk             /dev/sdb1;
       address          ipv4 192.168.101.10:7789;
       meta-disk        internal;
   }
   on filer02.localdomain {
       device           /dev/drbd_data minor 1;
       disk             /dev/sdb1;
       address          ipv4 192.168.101.20:7789;
       meta-disk        internal;
   }
   disk {
       on-io-error      pass_on;
   }
   syncer {
       rate             100M;
       after            meta;
   }
}


issue drbdadm on both servers.
# drbdadm create-md meta
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

# drbdadm create-md data
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.


start drbd process
# /etc/init.d/drbd start

# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre, 2011-01-28 12:17:35
m:res   cs         ro                   ds                         p  mounted  fstype
0:meta  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C
1:data  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C

# ls /dev/drbd_*
/dev/drbd_data  /dev/drbd_meta


make the filer01 primary

on filer01
# drbd-overview
 0:meta  Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
 1:data  Connected Secondary/Secondary Inconsistent/Inconsistent C r-----

[root@filer01 ~]# drbdsetup /dev/drbd_meta primary -o
[root@filer01 ~]# drbdsetup /dev/drbd_data primary -o


confirm that filer01 is primary , filer02 is secondary and status is UpToDate.
# drbd-overview
 0:meta  Connected Primary/Secondary UpToDate/UpToDate C r-----
 1:data  Connected Primary/Secondary UpToDate/UpToDate C r-----


prepare meta partition
[root@filer01 ~]# mkfs.ext3 /dev/drbd_meta


copy openfiler , samba , nfs etc configuration to meta partition.

at first stop openfiler process
[root@filer01 ~]# /etc/init.d/openfiler stop


copy openfiler’s config
[root@filer01 ~]# mkdir /meta
[root@filer01 ~]# mount -t ext3 /dev/drbd_meta /meta
[root@filer01 ~]# mv /opt/openfiler/ /opt/openfiler.local
[root@filer01 ~]# mkdir /meta/opt
[root@filer01 ~]# cp -a /opt/openfiler.local/ /meta/opt/openfiler
[root@filer01 ~]# ln -s /meta/opt/openfiler /opt/openfiler
[root@filer01 ~]# rm /meta/opt/openfiler/sbin/openfiler
[root@filer01 ~]# ln -s /usr/sbin/httpd /meta/opt/openfiler/sbin/openfiler
[root@filer01 ~]# rm /meta/opt/openfiler/etc/rsync.xml
[root@filer01 ~]# ln -s /opt/openfiler.local/etc/rsync.xml /meta/opt/openfiler/etc/
[root@filer01 ~]# mkdir -p /meta/etc/httpd/conf.d


copy samba,nfs etc config to meta partition
[root@filer01 ~]# /etc/init.d/nfslock stop
Stopping NFS statd:                                        [  OK  ]
[root@filer01 ~]# umount -a -t rpc-pipefs
[root@filer01 ~]# mv /etc/samba/ /meta/etc/
[root@filer01 ~]# ln -s /meta/etc/samba /etc/samba
[root@filer01 ~]# mkdir -p /meta/var/spool
[root@filer01 ~]# mv /var/spool/samba/ /meta/var/spool/
[root@filer01 ~]# ln -s /meta/var/spool/samba/ /var/spool/samba
[root@filer01 ~]# mkdir -p /meta/var/lib
[root@filer01 ~]# mv /var/lib/nfs/ /meta/var/lib/
[root@filer01 ~]# ln -s /meta/var/lib/nfs/ /var/lib/nfs
[root@filer01 ~]# mv /etc/exports /meta/etc/
[root@filer01 ~]# ln -s /meta/etc/exports /etc/exports
[root@filer01 ~]# mv /etc/ietd.conf /meta/etc/
[root@filer01 ~]# ln -s /meta/etc/ietd.conf /etc/ietd.conf
[root@filer01 ~]# mv /etc/initiators.allow /meta/etc/
[root@filer01 ~]# ln -s /meta/etc/initiators.allow /etc/initiators.allow
[root@filer01 ~]# mv /etc/initiators.deny /meta/etc/
[root@filer01 ~]# ln -s /meta/etc/initiators.deny /etc/initiators.deny
[root@filer01 ~]# mv /etc/proftpd /meta/etc/
[root@filer01 ~]# ln -s /meta/etc/proftpd/ /etc/proftpd
[root@filer01 ~]# rm /opt/openfiler/etc/httpd/modules
[root@filer01 ~]# ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules


start openfiler on filer01
[root@filer01 ~]# /etc/init.d/openfiler start


on filer02 , copy configuration files to meta partition.
[root@filer02 ~]# /etc/init.d/openfiler stop
Stopping openfiler:                                        [  OK  ]
[root@filer02 ~]# mkdir /meta
[root@filer02 ~]# mv /opt/openfiler/ /opt/openfiler.local
[root@filer02 ~]# ln -s /meta/opt/openfiler /opt/openfiler
[root@filer02 ~]# /etc/init.d/nfslock stop
Stopping NFS statd:                                        [  OK  ]
[root@filer02 ~]# umount -a -t rpc-pipefs
[root@filer02 ~]# rm -rf /etc/samba/
[root@filer02 ~]# ln -s /meta/etc/samba/ /etc/samba
[root@filer02 ~]# rm -rf /var/spool/samba/
[root@filer02 ~]# ln -s /meta/var/spool/samba /var/spool/samba
[root@filer02 ~]# rm -rf /var/lib/nfs/
[root@filer02 ~]# ln -s /meta/var/lib/nfs/ /var/lib/nfs
[root@filer02 ~]# rm -rf /etc/exports
[root@filer02 ~]# ln -s /meta/etc/exports /etc/exports
[root@filer02 ~]# rm -rf /etc/ietd.conf
[root@filer02 ~]# ln -s /meta/etc/ietd.conf /etc/ietd.conf
[root@filer02 ~]# rm -f /etc/initiators.allow
[root@filer02 ~]# rm -f /etc/initiators.deny
[root@filer02 ~]# ln -s /meta/etc/initiators.allow /etc/initiators.allow
[root@filer02 ~]# ln -s /meta/etc/initiators.deny /etc/initiators.deny
[root@filer02 ~]# rm -rf /etc/proftpd/
[root@filer02 ~]# ln -s /meta/etc/proftpd/ /etc/proftpd


edit lvm filter ( lvm.conf )
   # By default we accept every block device:
   #filter = [ "a/.*/" ]
   filter = [ "a|^/dev/drbd_.*|", "r|.*|" ]


copy lvm.conf to filer02 from filer01
# scp /etc/lvm/lvm.conf root@filer02.localdomain:/etc/lvm/
lvm.conf                                     100%   24KB  24.0KB/s   00:00


create physical/volume/logical volume on filer01 for data partition.
[root@filer01 ~]# pvcreate /dev/drbd_data
 Physical volume "/dev/drbd_data" successfully created
[root@filer01 ~]# vgcreate data /dev/drbd_data
 Volume group "data" successfully created
[root@filer01 ~]# lvcreate -L 100M -n filer data
 Logical volume "filer" created


set up corosync ( cluster engine )
[root@filer01 ~]# corosync-keygen
Writing corosync key to /etc/corosync/authkey.


copy auth key to filer02
[root@filer01 ~]# scp /etc/corosync/authkey root@filer02.localdomain:/etc/corosync/authkey


change permission.
[root@filer02 ~]# chmod 400 /etc/corosync/authkey


make pcmk file.
[root@filer01 ~]# cat /etc/corosync/service.d/pcmk
service {
       # Load the Pacemaker Cluster Resource Manager
       name: pacemaker
       ver:  0
}


copy it to filer02.
[root@filer01 ~]# scp /etc/corosync/service.d/pcmk root@filer02.localdomain:/etc/corosync/service.d/pcmk


edit corosync.conf
# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

[root@filer01 ~]# cat /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank

totem {
       version: 2
       secauth: off
       threads: 0
       interface {
               ringnumber: 0
               bindnetaddr: 192.168.101.0
               mcastaddr: 226.94.1.1
               mcastport: 5405
               ttl: 1
       }
}

logging {
       fileline: off
       to_stderr: no
       to_logfile: yes
       to_syslog: yes
       logfile: /var/log/cluster/corosync.log
       debug: off
       timestamp: on
       logger_subsys {
               subsys: AMF
               debug: off
       }
}

amf {
       mode: disabled
}


copy it to filer02
[root@filer01 ~]# scp /etc/corosync/corosync.conf root@filer02.localdomain:/etc/corosync/corosync.conf


disable openfiler and nfs-lock to run after booting , because corosync manages these processes.
[root@filer01 ~]# chkconfig openfiler off
[root@filer01 ~]# chkconfig nfslock off
[root@filer01 ~]# chkconfig corosync on
[root@filer01 ~]# chkconfig drbd on

[root@filer02 ~]# chkconfig openfiler off;chkconfig nfslock off;chkconfig corosync on;drbd on


restart both nodes.
[root@filer01 ~]# reboot
[root@filer02 ~]# reboot


check the status

filer01
[root@filer01 ~]# crm_mon -f -A
============
Last updated: Wed Mar  7 20:29:05 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ filer02.localdomain filer01.localdomain ]


Node Attributes:
* Node filer02.localdomain:
* Node filer01.localdomain:

Migration summary:
* Node filer02.localdomain:
* Node filer01.localdomain:

[root@filer01 ~]# crm configure show
node filer01.localdomain
node filer02.localdomain
property $id="cib-bootstrap-options" \
       dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
       cluster-infrastructure="openais" \
       expected-quorum-votes="2"


filer02
[root@filer02 ~]# crm_mon -f –A
============
Last updated: Wed Mar  7 20:29:28 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ filer02.localdomain filer01.localdomain ]


Node Attributes:
* Node filer02.localdomain:
* Node filer01.localdomain:

Migration summary:
* Node filer02.localdomain:
* Node filer01.localdomain:

[root@filer02 ~]# crm configure show
node filer01.localdomain
node filer02.localdomain
property $id="cib-bootstrap-options" \
       dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
       cluster-infrastructure="openais" \
       expected-quorum-votes="2"


configure corosync
I’m not familiar with crm …

on filer01
[root@filer01 ~]# crm configure
crm(live)configure# property stonith-enabled="false"
crm(live)configure# property no-quorum-policy="ignore"
crm(live)configure# rsc_defaults $id="rsc-options" resource-stickiness="100"
crm(live)configure# primitive ClusterIP ocf:heartbeat:IPaddr2 params ip="192.168.100.100" cidr_netmask="32" op monitor interval="30s"
crm(live)configure# primitive MetaFS ocf:heartbeat:Filesystem params device="/dev/drbd_meta" directory="/meta" fstype="ext3"
WARNING: MetaFS: default timeout 20s for start is smaller than the advised 60
WARNING: MetaFS: default timeout 20s for stop is smaller than the advised 60
crm(live)configure# primitive lvmdata ocf:heartbeat:LVM params volgrpname="data"
WARNING: lvmdata: default timeout 20s for start is smaller than the advised 30
WARNING: lvmdata: default timeout 20s for stop is smaller than the advised 30
crm(live)configure# primitive drbd_meta ocf:linbit:drbd params drbd_resource="meta" op monitor interval="15s"
WARNING: drbd_meta: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_meta: default timeout 20s for stop is smaller than the advised 100
crm(live)configure# primitive drbd_data ocf:linbit:drbd params drbd_resource="data" \op monitor interval="15s"
WARNING: drbd_data: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_data: default timeout 20s for stop is smaller than the advised 100
crm(live)configure# primitive openfiler lsb:openfiler
crm(live)configure# primitive iscsi lsb:iscsi-target
crm(live)configure# primitive samba lsb:smb
crm(live)configure# primitive nfs lsb:nfs
crm(live)configure# primitive nfslock lsb:nfslock
crm(live)configure# group g_drbd drbd_meta drbd_data
crm(live)configure# group g_services MetaFS lvmdata openfiler ClusterIP iscsi samba nfs nfslock
crm(live)configure# ms ms_g_drbd g_drbd meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
crm(live)configure# colocation c_g_services_on_g_drbd inf: g_services ms_g_drbd:Master
crm(live)configure# order o_g_services_after_g_drbd inf: ms_g_drbd:promote g_services:start
crm(live)configure#
crm(live)configure# commit
WARNING: MetaFS: default timeout 20s for start is smaller than the advised 60
WARNING: MetaFS: default timeout 20s for stop is smaller than the advised 60
WARNING: lvmdata: default timeout 20s for start is smaller than the advised 30
WARNING: lvmdata: default timeout 20s for stop is smaller than the advised 30
WARNING: drbd_meta: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_meta: default timeout 20s for stop is smaller than the advised 100
WARNING: drbd_data: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_data: default timeout 20s for stop is smaller than the advised 100
crm(live)configure# quit
bye

[root@filer01 ~]# crm configure show
node filer01.localdomain
node filer02.localdomain
primitive ClusterIP ocf:heartbeat:IPaddr2 \
       params ip="192.168.100.100" cidr_netmask="32" \
       op monitor interval="30s"
primitive MetaFS ocf:heartbeat:Filesystem \
       params device="/dev/drbd_meta" directory="/meta" fstype="ext3"
primitive drbd_data ocf:linbit:drbd \
       params drbd_resource="data" \
       op monitor interval="15s"
primitive drbd_meta ocf:linbit:drbd \
       params drbd_resource="meta" \
       op monitor interval="15s"
primitive iscsi lsb:iscsi-target
primitive lvmdata ocf:heartbeat:LVM \
       params volgrpname="data"
primitive nfs lsb:nfs
primitive nfslock lsb:nfslock
primitive openfiler lsb:openfiler
primitive samba lsb:smb
group g_drbd drbd_meta drbd_data
group g_services MetaFS lvmdata openfiler ClusterIP iscsi samba nfs nfslock
ms ms_g_drbd g_drbd \
       meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation c_g_services_on_g_drbd inf: g_services ms_g_drbd:Master
order o_g_services_after_g_drbd inf: ms_g_drbd:promote g_services:start
property $id="cib-bootstrap-options" \
       dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
       cluster-infrastructure="openais" \
       expected-quorum-votes="2" \
       stonith-enabled="false" \
       no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
       resource-stickiness="100"


reboot both nodes
[root@filer01 ~]# reboot
[root@filer02 ~]# reboot


ok, get things done.

Let’s confirm whether or not the cluster setting is correct

check which one is master.

filer01 is master and filer02 is slave.

on filer01
[root@filer01 ~]# crm_mon --one-shot -V
============
Last updated: Wed Mar  7 23:34:33 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ filer02.localdomain filer01.localdomain ]

Resource Group: g_services
    MetaFS     (ocf::heartbeat:Filesystem):    Started filer01.localdomain
    lvmdata    (ocf::heartbeat:LVM):   Started filer01.localdomain
    openfiler  (lsb:openfiler):        Started filer01.localdomain
    ClusterIP  (ocf::heartbeat:IPaddr2):       Started filer01.localdomain
    iscsi      (lsb:iscsi-target):     Started filer01.localdomain
    samba      (lsb:smb):      Started filer01.localdomain
    nfs        (lsb:nfs):      Started filer01.localdomain
    nfslock    (lsb:nfslock):  Started filer01.localdomain
Master/Slave Set: ms_g_drbd
    Masters: [ filer01.localdomain ]
    Slaves: [ filer02.localdomain ]


on filer02
[root@filer02 ~]# crm_mon --one-shot -V
============
Last updated: Wed Mar  7 23:35:19 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ filer02.localdomain filer01.localdomain ]

Resource Group: g_services
    MetaFS     (ocf::heartbeat:Filesystem):    Started filer01.localdomain
    lvmdata    (ocf::heartbeat:LVM):   Started filer01.localdomain
    openfiler  (lsb:openfiler):        Started filer01.localdomain
    ClusterIP  (ocf::heartbeat:IPaddr2):       Started filer01.localdomain
    iscsi      (lsb:iscsi-target):     Started filer01.localdomain
    samba      (lsb:smb):      Started filer01.localdomain
    nfs        (lsb:nfs):      Started filer01.localdomain
    nfslock    (lsb:nfslock):  Started filer01.localdomain
Master/Slave Set: ms_g_drbd
    Masters: [ filer01.localdomain ]
    Slaves: [ filer02.localdomain ]


check drbd status on both nodes

on filer01
[root@filer01 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre, 2011-01-28 12:17:35
m:res   cs         ro                 ds                 p  mounted  fstype
0:meta  Connected  Primary/Secondary  UpToDate/UpToDate  C  /meta    ext3
1:data  Connected  Primary/Secondary  UpToDate/UpToDate  C


on filer02
[root@filer02 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre, 2011-01-28 12:17:35
m:res   cs         ro                 ds                 p  mounted  fstype
0:meta  Connected  Secondary/Primary  UpToDate/UpToDate  C
1:data  Connected  Secondary/Primary  UpToDate/UpToDate  C


You can operate openfiler on the master  and can’t operate on the slave because the openfiler’s process on the slave is not running.

on filer01
[root@filer01 ~]# /etc/init.d/openfiler status
openfiler (pid  3503) is running...


on filer02
[root@filer02 ~]# /etc/init.d/openfiler status
openfiler is stopped


reboot filer01 ( master ) to check whether atvie/passive redundancy is correct or not.

reboot filer01
[root@filer01 ~]# reboot


filer02 become the master.
[root@filer02 ~]# crm_mon --one-shot -V
============
Last updated: Wed Mar  7 23:44:32 2012
Stack: openais
Current DC: filer02.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ filer02.localdomain filer01.localdomain ]

Resource Group: g_services
    MetaFS     (ocf::heartbeat:Filesystem):    Started filer02.localdomain
    lvmdata    (ocf::heartbeat:LVM):   Started filer02.localdomain
    openfiler  (lsb:openfiler):        Started filer02.localdomain
    ClusterIP  (ocf::heartbeat:IPaddr2):       Started filer02.localdomain
    iscsi      (lsb:iscsi-target):     Started filer02.localdomain
    samba      (lsb:smb):      Started filer02.localdomain
    nfs        (lsb:nfs):      Started filer02.localdomain
    nfslock    (lsb:nfslock):  Started filer02.localdomain
Master/Slave Set: ms_g_drbd
    Masters: [ filer02.localdomain ]


You can check the cluster status on Master node via GUI , however I think it’s easy to check the status via CLI. IMO

GUI on the master node

[ configure openfiler ]

VIP 192.168.100.100 ( the master node grabs this IP )
filer01 : eth0 : 192.168.100.10 , eth1: 192.168.101.10 ( heart beat )
filer02 : eth0 : 192.168.100.20 , eth1 : 192.168.101.20 ( heart beat )

Assumption :
filer01 is master and filer02 is slave.
I have already created a volume group called “data” when configuring active/passive cluster.

log into filer01

https://192.168.100.10:446
user : openfiler
Sharcredentials : password

before configuring openfiler , please make sure that CIFS service etc is running
GUI -> Services


GUI -> Shares -> create a new filesystem volume



create a sub-folder in “File-Share”


create a sub-folder in “File Share”

create ACL
System -> Network Access Configuration
Associate this ACL with “Shared-folder”




smb.conf
[root@filer01 ~]# cat /meta/etc/samba/smb.conf

# PLEASE DO NOT MODIFY THIS CONFIGURATION FILE!
#       This configuration file was autogenerated
#       by Openfiler. Any manual changes will be overwritten
#       Generated at: Thu Mar 8 3:38:13 EST 2012

# Global settings
[global]

workgroup =
server string = Openfiler NAS
netbios name = FILER01
password server =
realm =
; interfaces = 192.168.12.2/24 192.168.13.2/24
; remote announce = 92.168.1.255 192.168.2.44
; domain logons = yes
log file = /var/log/samba/%m.log
max log size = 0
; hosts deny = all
map to guest = Bad User
guest account = ofguest
display charset = LOCALE
unix charset = UTF-8
dos charset = CP850
encrypt passwords = yes
security = user
smb passwd file = /etc/samba/smbpasswd
unix password sync = yes
passwd program = /usr/bin/passwd %u
passwd chat = *New*password* %n\n *Retype*new*password* %n\n *passwd:*all*authentication*tokens*updated*successfully*
pam password change = yes
; username map = /etc/samba/smbusers
obey pam restrictions = yes
load printers = no
domain master = no
local master = no
preferred master = no
os level = 0


[data.file-share.shared-folder]
       comment = shared-folder
       path = /mnt/data/file-share/shared-folder
       read only = no
       writeable = yes
       oplocks = yes
       level2 oplocks = yes
       force security mode = 0
       dos filemode = no
       dos filetime resolution = no
       dos filetimes = no
       fake directory create times = no
  browseable = yes
  csc policy = manual
  share modes = yes
  veto oplock files = /*.mdb/*.MDB/*.dbf/*.DBF/
  veto files = /*:Zone.Identifier:*/
  store dos attributes = yes
  map acl inherit = yes
       create mode = 0777
       directory mode = 2777
       printable = no
       guest ok = yes
       hosts allow = 192.168.100.0/24
       hosts readonly allow =

# End of Openfiler configuration

[root@filer01 ~]#


access to the VIP(192.168.100.100) node from Windows XP.

create a file in this folder.

reboot the filer01 to make the filer02 primary.
[root@filer01 ~]# reboot


confirm filer02 node becomes master.
[root@filer02 ~]# crm_mon --one-shot
============
Last updated: Thu Mar  8 04:38:40 2012
Stack: openais
Current DC: filer02.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ filer02.localdomain filer01.localdomain ]

Resource Group: g_services
    MetaFS     (ocf::heartbeat:Filesystem):    Started filer02.localdomain
    lvmdata    (ocf::heartbeat:LVM):   Started filer02.localdomain
    openfiler  (lsb:openfiler):        Started filer02.localdomain
    ClusterIP  (ocf::heartbeat:IPaddr2):       Started filer02.localdomain
    iscsi      (lsb:iscsi-target):     Started filer02.localdomain
    samba      (lsb:smb):      Started filer02.localdomain
    nfs        (lsb:nfs):      Started filer02.localdomain
    nfslock    (lsb:nfslock):  Started filer02.localdomain
Master/Slave Set: ms_g_drbd
    Masters: [ filer02.localdomain ]
    Slaves: [ filer01.localdomain ]


access to the VIP from Windows XP again.
there’s no “shared-folder” in the filer02…

when filer02 is master

filer02’s GUI.  There’s no “shared-folder” sub-folder under “File-Share” folder.


nnnn , I’m tired … :(
I’ll locate the root cause of this issue next time…
I think I miss configure DRBD + LVM..
meta data is synchronized which is not LVM. On the other hand , data partition is LVM...

1 comment:

  1. Have you solved this problem? I'm getting the same thing https://forums.openfiler.com/index.php?/topic/6343-created-folder-not-showing-in-the-other-node/

    ReplyDelete

Note: Only a member of this blog may post a comment.