I referred the following instructions. ( many thx XD )
[ install Openfiler on KVM ]
At first , prepare two openfiler nodes.
I installed openfiler as VMs on KMV.
Host OS ( KVM host )
# tail -1 /etc/lsb-release
DISTRIB_DESCRIPTION="Linux Mint 12 Lisa"
# uname -r
3.0.0-16-generic
# libvirtd --version
libvirtd (libvirt) 0.9.2
# kvm --version
QEMU emulator version 0.14.1 (qemu-kvm-0.14.1), Copyright (c) 2003-2008 Fabrice Bellard
|
Seen from openfiler’s documentation , it seems that open filer is rPath linux base.
OS : Linux
Version : Generic 2.6.x kernel
filer01.localdomiain ( uname –n )
eth0 : 192.168.100.10
eth1 : 192.168.101.10 ( heart beat segment )
filer02.loccaldomain ( uname –n )
eth0 : 192.168.100.20
eth2 : 192.168.101.20 ( heart beat segment )
VIP : 192.168.100.100
I prepared the filer01 VM.
To make filer02 VM , it’s easy to make a clone VM from filer01 by virt-clone.
The original VM filer01 has three disks.
# ls /var/disk1/libvirt/images/filer01*
/var/disk1/libvirt/images/filer01-1.img <- meta data
/var/disk1/libvirt/images/filer01.img <- system
/var/disk1/libvirt/images/filer01-2.img <- data
|
When cloning a VM ( filer02 ) , specify multiple disks with –f option.
| # virt-clone -o filer01 -n filer02 -f /var/disk1/libvirt/images/filer02.img -f /var/disk1/libvirt/images/filer02-1.img -f /var/disk1/libvirt/images/filer02-2.img |
# LANG=C virsh list --all | grep filer
- filer01 shut off
- filer02 shut off
|
before starting filer02 VM , change hostname and IP address by guestfish.
mount filer02’s diks
# guestfish -i -d filer02
Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.
Type: 'help' for help on commands
'man' to read the manual
'quit' to quit the shell
/dev/vda1 mounted on /
|
change IP address and MAC address
><fs> edit /etc/sysconfig/network-scripts/ifcfg-eth0
><fs> edit /etc/sysconfig/network-scripts/ifcfg-eth1
|
You can check the clone VM’s MAC from *.xml file.
# LANG=C virsh dumpxml filer02 | grep -i "mac addr"
|
change hostname
><fs> edit /etc/hosts
><fs> edit /etc/sysconfig/network
|
change MAC address from the original VM’s to the clone VM’s
><fs> edit /etc/udev/rules.d/70-persistent-net.rules
<fs> quit
|
[ set up active/passive redundancy ]
Okay , let’s start two open filer nodes.
virsh # start filer01
virsh # start filer02
|
Here is my test environment
filer01
eth0 : 192.168.100.10
eth1 : 192.168.101.10 ( heart beat segment )
/dev/sda1 ( system ) : 4.5GB
/dev/sda2 ( swap ) : 0.5GB
/dev/sdb1 ( meta ) : 512MB
/dev/sdc1 ( data ) : 1GB
filer02
eth0 : 192.168.100.20
eth2 : 192.168.101.20 ( heart beat segment )
/dev/sda1 ( system ) : 4.5GB
/dev/sda2 ( swap ) : 0.5GB
/dev/sdb1 ( meta ) : 512MB
/dev/sdc1 ( data ) : 1GB
|
Clients
|
L2SW
| | service segment
filer01 filer02
| | heart beat segment
L2SW
add the hostname and the IP address of heart beat segment IP in /etc/hosts for easier access
on filer01 , register filer02’s IP
# grep filer02 /etc/hosts
192.168.101.20 filer02.localdomain
|
on filer02
# egrep filer01 /etc/hosts
192.168.101.10 filer01.localdomain
|
make SSH keys and exchange that to make filer01/filer02 exchange files without password authentication.
on filer01
# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
|
on filer02
# ssh-keygen -t dsa
|
exchange SSH keys.
on filer01
# scp /root/.ssh/id_dsa.pub root@filer02.localdomain:/root/.ssh/authorized_keys
|
on filer02
# scp /root/.ssh/id_dsa.pub root@filer01.localdomain:/root/.ssh/authorized_keys
|
create partitions for meta and data
do the same thing on both filer01 and filer02
make the partition for meta ( /dev/sdb1 )
# fdisk /dev/sdb
Disk /dev/sdb: 536 MB, 536870912 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb134de93
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4, default 1): 1
First sector (2048-1048575, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1048575, default 1048575):
Using default value 1048575
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
|
make the partition for data ( /dev/sdc1 )
# fdisk /dev/sdc
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
|
Please note that I have not mounted the partition for meta and data yet.
# mount
/dev/sda1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
|
edit /etc/drbd.conf on both nodes.
Both servers have the same drbd.conf.
# cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
# meta
resource meta {
device /dev/drbd_meta minor 0;
disk /dev/sdc1;
meta-disk internal;
protocol C;
syncer {
rate 100M;
}
disk {
on-io-error pass_on;
}
on filer01.localdomain {
address 192.168.101.10:7788; # filer01’s IP
}
on filer02.localdomain {
address 192.168.101.20:7788; # filer02's IP
}
}
# data
resource data {
device /dev/drbd_data minor 1;
disk /dev/sdb1;
meta-disk internal;
protocol C;
syncer {
rate 100M;
after "meta";
}
disk {
on-io-error pass_on;
}
on filer01.localdomain {
address 192.168.101.10:7789; # filer01's IP
}
on filer02.localdomain {
address 192.168.101.20:7789; # filer02's IP
}
}
|
check both 7788 and 7789 ports are unused by other processes.
# lsof -ni:7789
# lsof -ni:7788
|
copy drbd.conf to filer02 from filer01
# scp /etc/drbd.conf root@filer02.localdomain:/etc/drbd.conf
|
check the syntax
# drbdadm dump all
# /etc/drbd.conf
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
}
}
# resource meta on filer01.localdomain: not ignored, not stacked
resource meta {
protocol C;
on filer01.localdomain {
device /dev/drbd_meta minor 0;
disk /dev/sdc1;
address ipv4 192.168.101.10:7788;
meta-disk internal;
}
on filer02.localdomain {
device /dev/drbd_meta minor 0;
disk /dev/sdc1;
address ipv4 192.168.101.20:7788;
meta-disk internal;
}
disk {
on-io-error pass_on;
}
syncer {
rate 100M;
}
}
# resource data on filer01.localdomain: not ignored, not stacked
resource data {
protocol C;
on filer01.localdomain {
device /dev/drbd_data minor 1;
disk /dev/sdb1;
address ipv4 192.168.101.10:7789;
meta-disk internal;
}
on filer02.localdomain {
device /dev/drbd_data minor 1;
disk /dev/sdb1;
address ipv4 192.168.101.20:7789;
meta-disk internal;
}
disk {
on-io-error pass_on;
}
syncer {
rate 100M;
after meta;
}
}
|
issue drbdadm on both servers.
# drbdadm create-md meta
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
# drbdadm create-md data
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
|
start drbd process
# /etc/init.d/drbd start
|
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre, 2011-01-28 12:17:35
m:res cs ro ds p mounted fstype
0:meta Connected Secondary/Secondary Inconsistent/Inconsistent C
1:data Connected Secondary/Secondary Inconsistent/Inconsistent C
|
# ls /dev/drbd_*
/dev/drbd_data /dev/drbd_meta
|
make the filer01 primary
on filer01
# drbd-overview
0:meta Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
1:data Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
|
[root@filer01 ~]# drbdsetup /dev/drbd_meta primary -o
[root@filer01 ~]# drbdsetup /dev/drbd_data primary -o
|
confirm that filer01 is primary , filer02 is secondary and status is UpToDate.
# drbd-overview
0:meta Connected Primary/Secondary UpToDate/UpToDate C r-----
1:data Connected Primary/Secondary UpToDate/UpToDate C r-----
|
prepare meta partition
[root@filer01 ~]# mkfs.ext3 /dev/drbd_meta
|
copy openfiler , samba , nfs etc configuration to meta partition.
at first stop openfiler process
[root@filer01 ~]# /etc/init.d/openfiler stop
|
copy openfiler’s config
[root@filer01 ~]# mkdir /meta
[root@filer01 ~]# mount -t ext3 /dev/drbd_meta /meta
[root@filer01 ~]# mv /opt/openfiler/ /opt/openfiler.local
[root@filer01 ~]# mkdir /meta/opt
[root@filer01 ~]# cp -a /opt/openfiler.local/ /meta/opt/openfiler
[root@filer01 ~]# ln -s /meta/opt/openfiler /opt/openfiler
[root@filer01 ~]# rm /meta/opt/openfiler/sbin/openfiler
[root@filer01 ~]# ln -s /usr/sbin/httpd /meta/opt/openfiler/sbin/openfiler
[root@filer01 ~]# rm /meta/opt/openfiler/etc/rsync.xml
[root@filer01 ~]# ln -s /opt/openfiler.local/etc/rsync.xml /meta/opt/openfiler/etc/
[root@filer01 ~]# mkdir -p /meta/etc/httpd/conf.d
|
copy samba,nfs etc config to meta partition
| [root@filer01 ~]# /etc/init.d/nfslock stop Stopping NFS statd: [ OK ] [root@filer01 ~]# umount -a -t rpc-pipefs [root@filer01 ~]# mv /etc/samba/ /meta/etc/ [root@filer01 ~]# ln -s /meta/etc/samba /etc/samba [root@filer01 ~]# mkdir -p /meta/var/spool [root@filer01 ~]# mv /var/spool/samba/ /meta/var/spool/ [root@filer01 ~]# ln -s /meta/var/spool/samba/ /var/spool/samba [root@filer01 ~]# mkdir -p /meta/var/lib [root@filer01 ~]# mv /var/lib/nfs/ /meta/var/lib/ [root@filer01 ~]# ln -s /meta/var/lib/nfs/ /var/lib/nfs [root@filer01 ~]# mv /etc/exports /meta/etc/ [root@filer01 ~]# ln -s /meta/etc/exports /etc/exports [root@filer01 ~]# mv /etc/ietd.conf /meta/etc/ [root@filer01 ~]# ln -s /meta/etc/ietd.conf /etc/ietd.conf [root@filer01 ~]# mv /etc/initiators.allow /meta/etc/ [root@filer01 ~]# ln -s /meta/etc/initiators.allow /etc/initiators.allow [root@filer01 ~]# mv /etc/initiators.deny /meta/etc/ [root@filer01 ~]# ln -s /meta/etc/initiators.deny /etc/initiators.deny [root@filer01 ~]# mv /etc/proftpd /meta/etc/ [root@filer01 ~]# ln -s /meta/etc/proftpd/ /etc/proftpd [root@filer01 ~]# rm /opt/openfiler/etc/httpd/modules [root@filer01 ~]# ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules |
start openfiler on filer01
[root@filer01 ~]# /etc/init.d/openfiler start
|
on filer02 , copy configuration files to meta partition.
[root@filer02 ~]# /etc/init.d/openfiler stop
Stopping openfiler: [ OK ]
[root@filer02 ~]# mkdir /meta
[root@filer02 ~]# mv /opt/openfiler/ /opt/openfiler.local
[root@filer02 ~]# ln -s /meta/opt/openfiler /opt/openfiler
[root@filer02 ~]# /etc/init.d/nfslock stop
Stopping NFS statd: [ OK ]
[root@filer02 ~]# umount -a -t rpc-pipefs
[root@filer02 ~]# rm -rf /etc/samba/
[root@filer02 ~]# ln -s /meta/etc/samba/ /etc/samba
[root@filer02 ~]# rm -rf /var/spool/samba/
[root@filer02 ~]# ln -s /meta/var/spool/samba /var/spool/samba
[root@filer02 ~]# rm -rf /var/lib/nfs/
[root@filer02 ~]# ln -s /meta/var/lib/nfs/ /var/lib/nfs
[root@filer02 ~]# rm -rf /etc/exports
[root@filer02 ~]# ln -s /meta/etc/exports /etc/exports
[root@filer02 ~]# rm -rf /etc/ietd.conf
[root@filer02 ~]# ln -s /meta/etc/ietd.conf /etc/ietd.conf
[root@filer02 ~]# rm -f /etc/initiators.allow
[root@filer02 ~]# rm -f /etc/initiators.deny
[root@filer02 ~]# ln -s /meta/etc/initiators.allow /etc/initiators.allow
[root@filer02 ~]# ln -s /meta/etc/initiators.deny /etc/initiators.deny
[root@filer02 ~]# rm -rf /etc/proftpd/
[root@filer02 ~]# ln -s /meta/etc/proftpd/ /etc/proftpd
|
edit lvm filter ( lvm.conf )
# By default we accept every block device:
#filter = [ "a/.*/" ]
filter = [ "a|^/dev/drbd_.*|", "r|.*|" ]
|
copy lvm.conf to filer02 from filer01
# scp /etc/lvm/lvm.conf root@filer02.localdomain:/etc/lvm/
lvm.conf 100% 24KB 24.0KB/s 00:00
|
create physical/volume/logical volume on filer01 for data partition.
[root@filer01 ~]# pvcreate /dev/drbd_data
Physical volume "/dev/drbd_data" successfully created
[root@filer01 ~]# vgcreate data /dev/drbd_data
Volume group "data" successfully created
[root@filer01 ~]# lvcreate -L 100M -n filer data
Logical volume "filer" created
|
set up corosync ( cluster engine )
[root@filer01 ~]# corosync-keygen
Writing corosync key to /etc/corosync/authkey.
|
copy auth key to filer02
[root@filer01 ~]# scp /etc/corosync/authkey root@filer02.localdomain:/etc/corosync/authkey
|
change permission.
[root@filer02 ~]# chmod 400 /etc/corosync/authkey
|
make pcmk file.
[root@filer01 ~]# cat /etc/corosync/service.d/pcmk
service {
# Load the Pacemaker Cluster Resource Manager
name: pacemaker
ver: 0
}
|
copy it to filer02.
| [root@filer01 ~]# scp /etc/corosync/service.d/pcmk root@filer02.localdomain:/etc/corosync/service.d/pcmk |
edit corosync.conf
# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
|
[root@filer01 ~]# cat /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
interface {
ringnumber: 0
bindnetaddr: 192.168.101.0
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
|
copy it to filer02
[root@filer01 ~]# scp /etc/corosync/corosync.conf root@filer02.localdomain:/etc/corosync/corosync.conf
|
disable openfiler and nfs-lock to run after booting , because corosync manages these processes.
[root@filer01 ~]# chkconfig openfiler off
[root@filer01 ~]# chkconfig nfslock off
[root@filer01 ~]# chkconfig corosync on
[root@filer01 ~]# chkconfig drbd on
|
[root@filer02 ~]# chkconfig openfiler off;chkconfig nfslock off;chkconfig corosync on;drbd on
|
restart both nodes.
[root@filer01 ~]# reboot
[root@filer02 ~]# reboot
|
check the status
filer01
[root@filer01 ~]# crm_mon -f -A
============
Last updated: Wed Mar 7 20:29:05 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
0 Resources configured.
============
Online: [ filer02.localdomain filer01.localdomain ]
Node Attributes:
* Node filer02.localdomain:
* Node filer01.localdomain:
Migration summary:
* Node filer02.localdomain:
* Node filer01.localdomain:
[root@filer01 ~]# crm configure show
node filer01.localdomain
node filer02.localdomain
property $id="cib-bootstrap-options" \
dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
cluster-infrastructure="openais" \
expected-quorum-votes="2"
|
filer02
[root@filer02 ~]# crm_mon -f –A
============
Last updated: Wed Mar 7 20:29:28 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
0 Resources configured.
============
Online: [ filer02.localdomain filer01.localdomain ]
Node Attributes:
* Node filer02.localdomain:
* Node filer01.localdomain:
Migration summary:
* Node filer02.localdomain:
* Node filer01.localdomain:
[root@filer02 ~]# crm configure show
node filer01.localdomain
node filer02.localdomain
property $id="cib-bootstrap-options" \
dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
cluster-infrastructure="openais" \
expected-quorum-votes="2"
|
configure corosync
I’m not familiar with crm …
on filer01
[root@filer01 ~]# crm configure
crm(live)configure# property stonith-enabled="false"
crm(live)configure# property no-quorum-policy="ignore"
crm(live)configure# rsc_defaults $id="rsc-options" resource-stickiness="100"
crm(live)configure# primitive ClusterIP ocf:heartbeat:IPaddr2 params ip="192.168.100.100" cidr_netmask="32" op monitor interval="30s"
crm(live)configure# primitive MetaFS ocf:heartbeat:Filesystem params device="/dev/drbd_meta" directory="/meta" fstype="ext3"
WARNING: MetaFS: default timeout 20s for start is smaller than the advised 60
WARNING: MetaFS: default timeout 20s for stop is smaller than the advised 60
crm(live)configure# primitive lvmdata ocf:heartbeat:LVM params volgrpname="data"
WARNING: lvmdata: default timeout 20s for start is smaller than the advised 30
WARNING: lvmdata: default timeout 20s for stop is smaller than the advised 30
crm(live)configure# primitive drbd_meta ocf:linbit:drbd params drbd_resource="meta" op monitor interval="15s"
WARNING: drbd_meta: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_meta: default timeout 20s for stop is smaller than the advised 100
crm(live)configure# primitive drbd_data ocf:linbit:drbd params drbd_resource="data" \op monitor interval="15s"
WARNING: drbd_data: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_data: default timeout 20s for stop is smaller than the advised 100
crm(live)configure# primitive openfiler lsb:openfiler
crm(live)configure# primitive iscsi lsb:iscsi-target
crm(live)configure# primitive samba lsb:smb
crm(live)configure# primitive nfs lsb:nfscrm(live)configure# primitive nfslock lsb:nfslock crm(live)configure# group g_drbd drbd_meta drbd_data crm(live)configure# group g_services MetaFS lvmdata openfiler ClusterIP iscsi samba nfs nfslock crm(live)configure# ms ms_g_drbd g_drbd meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" crm(live)configure# colocation c_g_services_on_g_drbd inf: g_services ms_g_drbd:Master crm(live)configure# order o_g_services_after_g_drbd inf: ms_g_drbd:promote g_services:start
crm(live)configure#
crm(live)configure# commit
WARNING: MetaFS: default timeout 20s for start is smaller than the advised 60
WARNING: MetaFS: default timeout 20s for stop is smaller than the advised 60
WARNING: lvmdata: default timeout 20s for start is smaller than the advised 30
WARNING: lvmdata: default timeout 20s for stop is smaller than the advised 30
WARNING: drbd_meta: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_meta: default timeout 20s for stop is smaller than the advised 100
WARNING: drbd_data: default timeout 20s for start is smaller than the advised 240
WARNING: drbd_data: default timeout 20s for stop is smaller than the advised 100
crm(live)configure# quit
bye
[root@filer01 ~]# crm configure show
node filer01.localdomain
node filer02.localdomain
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.100.100" cidr_netmask="32" \
op monitor interval="30s"
primitive MetaFS ocf:heartbeat:Filesystem \
params device="/dev/drbd_meta" directory="/meta" fstype="ext3"
primitive drbd_data ocf:linbit:drbd \
params drbd_resource="data" \
op monitor interval="15s"
primitive drbd_meta ocf:linbit:drbd \
params drbd_resource="meta" \
op monitor interval="15s"
primitive iscsi lsb:iscsi-target
primitive lvmdata ocf:heartbeat:LVM \
params volgrpname="data"
primitive nfs lsb:nfs
primitive nfslock lsb:nfslock
primitive openfiler lsb:openfiler
primitive samba lsb:smb
group g_drbd drbd_meta drbd_data
group g_services MetaFS lvmdata openfiler ClusterIP iscsi samba nfs nfslock
ms ms_g_drbd g_drbd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation c_g_services_on_g_drbd inf: g_services ms_g_drbd:Master
order o_g_services_after_g_drbd inf: ms_g_drbd:promote g_services:start
property $id="cib-bootstrap-options" \
dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
|
reboot both nodes
[root@filer01 ~]# reboot
[root@filer02 ~]# reboot
|
ok, get things done.
Let’s confirm whether or not the cluster setting is correct
check which one is master.
filer01 is master and filer02 is slave.
on filer01
[root@filer01 ~]# crm_mon --one-shot -V
============
Last updated: Wed Mar 7 23:34:33 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ filer02.localdomain filer01.localdomain ]
Resource Group: g_services
MetaFS (ocf::heartbeat:Filesystem): Started filer01.localdomain
lvmdata (ocf::heartbeat:LVM): Started filer01.localdomain
openfiler (lsb:openfiler): Started filer01.localdomain
ClusterIP (ocf::heartbeat:IPaddr2): Started filer01.localdomain
iscsi (lsb:iscsi-target): Started filer01.localdomain
samba (lsb:smb): Started filer01.localdomain
nfs (lsb:nfs): Started filer01.localdomain
nfslock (lsb:nfslock): Started filer01.localdomain
Master/Slave Set: ms_g_drbd
Masters: [ filer01.localdomain ]
Slaves: [ filer02.localdomain ]
|
on filer02
[root@filer02 ~]# crm_mon --one-shot -V
============
Last updated: Wed Mar 7 23:35:19 2012
Stack: openais
Current DC: filer01.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ filer02.localdomain filer01.localdomain ]
Resource Group: g_services
MetaFS (ocf::heartbeat:Filesystem): Started filer01.localdomain
lvmdata (ocf::heartbeat:LVM): Started filer01.localdomain
openfiler (lsb:openfiler): Started filer01.localdomain
ClusterIP (ocf::heartbeat:IPaddr2): Started filer01.localdomain
iscsi (lsb:iscsi-target): Started filer01.localdomain
samba (lsb:smb): Started filer01.localdomain
nfs (lsb:nfs): Started filer01.localdomain
nfslock (lsb:nfslock): Started filer01.localdomain
Master/Slave Set: ms_g_drbd
Masters: [ filer01.localdomain ]
Slaves: [ filer02.localdomain ]
|
check drbd status on both nodes
on filer01
[root@filer01 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre, 2011-01-28 12:17:35
m:res cs ro ds p mounted fstype
0:meta Connected Primary/Secondary UpToDate/UpToDate C /meta ext3
1:data Connected Primary/Secondary UpToDate/UpToDate C
|
on filer02
[root@filer02 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre, 2011-01-28 12:17:35
m:res cs ro ds p mounted fstype
0:meta Connected Secondary/Primary UpToDate/UpToDate C
1:data Connected Secondary/Primary UpToDate/UpToDate C
|
You can operate openfiler on the master and can’t operate on the slave because the openfiler’s process on the slave is not running.
on filer01
[root@filer01 ~]# /etc/init.d/openfiler status
openfiler (pid 3503) is running...
|
on filer02
[root@filer02 ~]# /etc/init.d/openfiler status
openfiler is stopped
|
reboot filer01 ( master ) to check whether atvie/passive redundancy is correct or not.
reboot filer01
[root@filer01 ~]# reboot
|
filer02 become the master.
[root@filer02 ~]# crm_mon --one-shot -V
============
Last updated: Wed Mar 7 23:44:32 2012
Stack: openais
Current DC: filer02.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ filer02.localdomain filer01.localdomain ]
Resource Group: g_services
MetaFS (ocf::heartbeat:Filesystem): Started filer02.localdomain
lvmdata (ocf::heartbeat:LVM): Started filer02.localdomain
openfiler (lsb:openfiler): Started filer02.localdomain
ClusterIP (ocf::heartbeat:IPaddr2): Started filer02.localdomain
iscsi (lsb:iscsi-target): Started filer02.localdomain
samba (lsb:smb): Started filer02.localdomain
nfs (lsb:nfs): Started filer02.localdomain
nfslock (lsb:nfslock): Started filer02.localdomain
Master/Slave Set: ms_g_drbd
Masters: [ filer02.localdomain ]
|
You can check the cluster status on Master node via GUI , however I think it’s easy to check the status via CLI. IMO
GUI on the master node
[ configure openfiler ]
VIP 192.168.100.100 ( the master node grabs this IP )
filer01 : eth0 : 192.168.100.10 , eth1: 192.168.101.10 ( heart beat )
filer02 : eth0 : 192.168.100.20 , eth1 : 192.168.101.20 ( heart beat )
Assumption :
filer01 is master and filer02 is slave.
I have already created a volume group called “data” when configuring active/passive cluster.
log into filer01
https://192.168.100.10:446
user : openfiler
Sharcredentials : password
before configuring openfiler , please make sure that CIFS service etc is running
GUI -> Services
GUI -> Shares -> create a new filesystem volume
create a sub-folder in “File-Share”
create a sub-folder in “File Share”
create ACL
System -> Network Access Configuration
Associate this ACL with “Shared-folder”
smb.conf
| [root@filer01 ~]# cat /meta/etc/samba/smb.conf # PLEASE DO NOT MODIFY THIS CONFIGURATION FILE! # This configuration file was autogenerated # by Openfiler. Any manual changes will be overwritten # Generated at: Thu Mar 8 3:38:13 EST 2012 # Global settings [global] workgroup = server string = Openfiler NAS netbios name = FILER01 password server = realm = ; interfaces = 192.168.12.2/24 192.168.13.2/24 ; remote announce = 92.168.1.255 192.168.2.44 ; domain logons = yes log file = /var/log/samba/%m.log max log size = 0 ; hosts deny = all map to guest = Bad User guest account = ofguest display charset = LOCALE unix charset = UTF-8 dos charset = CP850 encrypt passwords = yes security = user smb passwd file = /etc/samba/smbpasswd unix password sync = yes passwd program = /usr/bin/passwd %u passwd chat = *New*password* %n\n *Retype*new*password* %n\n *passwd:*all*authentication*tokens*updated*successfully* pam password change = yes ; username map = /etc/samba/smbusers obey pam restrictions = yes load printers = no domain master = no local master = no preferred master = no os level = 0 [data.file-share.shared-folder] comment = shared-folder path = /mnt/data/file-share/shared-folder read only = no writeable = yes oplocks = yes level2 oplocks = yes force security mode = 0 dos filemode = no dos filetime resolution = no dos filetimes = no fake directory create times = no browseable = yes csc policy = manual share modes = yes veto oplock files = /*.mdb/*.MDB/*.dbf/*.DBF/ veto files = /*:Zone.Identifier:*/ store dos attributes = yes map acl inherit = yes create mode = 0777 directory mode = 2777 printable = no guest ok = yes hosts allow = 192.168.100.0/24 hosts readonly allow = # End of Openfiler configuration [root@filer01 ~]# |
access to the VIP(192.168.100.100) node from Windows XP.
create a file in this folder.
reboot the filer01 to make the filer02 primary.
[root@filer01 ~]# reboot
|
confirm filer02 node becomes master.
[root@filer02 ~]# crm_mon --one-shot
============
Last updated: Thu Mar 8 04:38:40 2012
Stack: openais
Current DC: filer02.localdomain - partition with quorum
Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ filer02.localdomain filer01.localdomain ]
Resource Group: g_services
MetaFS (ocf::heartbeat:Filesystem): Started filer02.localdomain
lvmdata (ocf::heartbeat:LVM): Started filer02.localdomain
openfiler (lsb:openfiler): Started filer02.localdomain
ClusterIP (ocf::heartbeat:IPaddr2): Started filer02.localdomain
iscsi (lsb:iscsi-target): Started filer02.localdomain
samba (lsb:smb): Started filer02.localdomain
nfs (lsb:nfs): Started filer02.localdomain
nfslock (lsb:nfslock): Started filer02.localdomain
Master/Slave Set: ms_g_drbd
Masters: [ filer02.localdomain ]
Slaves: [ filer01.localdomain ]
|
access to the VIP from Windows XP again.
there’s no “shared-folder” in the filer02…
when filer02 is master
filer02’s GUI. There’s no “shared-folder” sub-folder under “File-Share” folder.
nnnn , I’m tired … :(
I’ll locate the root cause of this issue next time…
I think I miss configure DRBD + LVM..
meta data is synchronized which is not LVM. On the other hand , data partition is LVM...
Have you solved this problem? I'm getting the same thing https://forums.openfiler.com/index.php?/topic/6343-created-folder-not-showing-in-the-other-node/
ReplyDelete