CentOS7 : install and set up GlusterFS

Here are logs when setting up GlusterFS on CentOS7.

Reference

http://www.linuxtechi.com/setup-glusterfs-storage-on-centos-7-rhel-7/

Five VMs are running within KVM.

Four VMs are for Gluster FS servers and one for client.

$ virsh list --all | grep gluster 26    cent7-gluster02                running 27    cent7-gluster03                running 29    cent7-gluster01                running 30    cent7-gluster04                running 31    cent7-gluster-client           running

Edit /etc/hosts so that gluster servers can communicate with each other.

[root@gluster01 ~]# cat /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.130.110 gluster01 gluster01.localdomain 192.168.130.111 gluster02 gluster02.localdomain 192.168.130.112 gluster03 gluster03.localdomain 192.168.130.113 gluster04 gluster04.localdomain

add a block storage to the Gluster servers.

Now gluster servers have only one storage device, /dev/vda.

root@gluster01 ~]# ls /dev/vd* /dev/vda  /dev/vda1 [root@gluster01 ~]#

on the KVM machine, create a block storage for three gluster servers.

$ sudo dd if=/dev/zero of=/var/lib/libvirt/images/gluster01-block.img bs=1M seek=4096 count=0 $ sudo dd if=/dev/zero of=/var/lib/libvirt/images/gluster02-block.img bs=1M seek=4096 count=0 $ sudo dd if=/dev/zero of=/var/lib/libvirt/images/gluster03-block.img bs=1M seek=4096 count=0 $ sudo dd if=/dev/zero of=/var/lib/libvirt/images/gluster04-block.img bs=1M seek=4096 count=0

Attach a disk to a VM.

$ sudo virsh attach-disk cent7-gluster01 /var/lib/libvirt/images/gluster01-block.img vdb --cache none Disk attached successfully $ sudo virsh attach-disk cent7-gluster02 /var/lib/libvirt/images/gluster02-block.img vdb --cache noneDisk attached successfully $ sudo virsh attach-disk cent7-gluster03 /var/lib/libvirt/images/gluster03-block.img vdb --cache none Disk attached successfully $ sudo virsh attach-disk cent7-gluster04 /var/lib/libvirt/images/gluster04-block.img vdb --cache none Disk attached successfully

Confirm vdb has been added.

$ ansible gluster -m shell -a 'ls /dev/vd*' 192.168.130.112 | success | rc=0 >> /dev/vda /dev/vda1 /dev/vdb 192.168.130.110 | success | rc=0 >> /dev/vda /dev/vda1 /dev/vdb 192.168.130.111 | success | rc=0 >> /dev/vda /dev/vda1 /dev/vdb 192.168.130.113 | success | rc=0 >> /dev/vda /dev/vda1 /dev/vdb

install gluster server

$ ansible gluster -m shell -a 'yum install -y centos-release-gluster epel-release' $ ansible gluster -m shell -a 'yum install -y glusterfs-server' $ ansible gluster -m shell -a 'yum install -y glusterfs-server' $ ansible gluster -m shell -a 'systemctl start glusterd' $ ansible gluster -m shell -a 'systemctl enable glusterd'

[ volume setup : distributed volume ]

create a trusted storage pool between gluster01 and gluster02.

on the gluster01

[root@gluster01 ~]# gluster peer probe gluster02 peer probe: success. [root@gluster01 ~]# gluster peer status Number of Peers: 1 Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected)

Create a brick on the gluster01

[root@gluster01 ~]# pvcreate /dev/vdb [root@gluster01 ~]# vgcreate vg-bricks /dev/vdb [root@gluster01 ~]# lvcreate -L 2G -n lv-brick01 vg-bricks [root@gluster01 ~]# lvdisplay  --- Logical volume ---  LV Path                /dev/vg-bricks/lv-brick01  LV Name                lv-brick01  VG Name                vg-bricks  LV UUID                jOFkXI-h13U-tVju-a416-bHFX-xlZO-CF3Uo3  LV Write Access        read/write  LV Creation host, time gluster01.localdomain, 2016-09-26 13:26:28 +0900  LV Status              available  # open                 0  LV Size                2.00 GiB  Current LE             512  Segments               1  Allocation             inherit  Read ahead sectors     auto  - currently set to     8192  Block device           252:0

[root@gluster01 ~]# mkfs.xfs -i size=512 /dev/vg-bricks/lv-brick01 [root@gluster01 ~]# mkdir -p /bricks/dist_brick1 [root@gluster01 ~]# mount /dev/vg-bricks/lv-brick01 /bricks/dist_brick1

on the gluster02

[root@gluster02 ~]# pvcreate /dev/vdb  Physical volume "/dev/vdb" successfully created [root@gluster02 ~]# vgcreate vg-bricks /dev/vdb  Volume group "vg-bricks" successfully created [root@gluster02 ~]# lvcreate -L 2G -n lv-brick01 vg-bricks  Logical volume "lv-brick01" created. [root@gluster02 ~]# mkfs.xfs -i size=512 /dev/vg-bricks/lv-brick01 meta-data=/dev/vg-bricks/lv-brick01 isize=512    agcount=4, agsize=131072 blks         =                       sectsz=512   attr=2, projid32bit=1         =                       crc=0        finobt=0 data     =                       bsize=4096   blocks=524288, imaxpct=25         =                       sunit=0      swidth=0 blks naming   =version 2              bsize=4096   ascii-ci=0 ftype=0 log      =internal log           bsize=4096   blocks=2560, version=2         =                       sectsz=512   sunit=0 blks, lazy-count=1 realtime =none                   extsz=4096   blocks=0, rtextents=0 [root@gluster02 ~]# mkdir -p /bricks/dist_brick1 [root@gluster02 ~]# mount /dev/vg-bricks/lv-brick01 /bricks/dist_brick1 [root@gluster02 ~]#

Create a distributed volume

on the gluster01

error, needs to specify sub directory.

[root@gluster01 ~]# gluster volume create distvol gluster01:/bricks/dist_brick1 gluster02:/bricks/dist_brick1 volume create: distvol: failed: The brick gluster01:/bricks/dist_brick1 is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior.

create a distributed volume.

[root@gluster01 ~]# gluster volume create distvol gluster01:/bricks/dist_brick1/brick gluster02:/bricks/dist_brick1/brick volume create: distvol: success: please start the volume to access data [root@gluster01 ~]#

start the volume

[root@gluster01 ~]# gluster volume start distvol volume start: distvol: success

[root@gluster01 ~]# gluster volume info distvol Volume Name: distvol Type: Distribute Volume ID: 0f5e64e6-3219-4ab3-be57-c3c9da049e7e Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gluster01:/bricks/dist_brick1/brick Brick2: gluster02:/bricks/dist_brick1/brick Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on

mount the volume via client.

on the client

[root@gluster-client ~]# yum install -y centos-release-gluster epel-release [root@gluster-client ~]#  yum install glusterfs-fuse –y [root@gluster-client ~]# mount -t glusterfs -o acl gluster01:/distvol /mnt/distvol/ [root@gluster-client ~]# mount| grep gluster gluster01:/distvol on /mnt/distvol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)

create some files on the client.

[root@gluster-client ~]# cd /mnt/distvol/ [root@gluster-client distvol]# pwd /mnt/distvol [root@gluster-client distvol]# for i in {1..10};do echo $i > $i.txt;done [root@gluster-client distvol]# ls 1.txt  10.txt  2.txt  3.txt  4.txt  5.txt  6.txt  7.txt  8.txt  9.txt [root@gluster-client distvol]#

on the gluster01

[root@gluster01 ~]# ls /bricks/dist_brick1/brick/ 4.txt  8.txt  9.txt

on the gluster02

[root@gluster02 ~]# ls /bricks/dist_brick1/brick/ 1.txt  10.txt  2.txt  3.txt  5.txt  6.txt  7.txt

some files are stored in gluster01 and the others are in gluster02.

suspend gluster01 VM

on the KVM host.

virsh # suspend cent7-gluster01 Domain cent7-gluster01 suspended

on the gluster02

[root@gluster02 ~]# gluster peer status Number of Peers: 1 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Disconnected)

on the client, create 10 files ( a_0.txt .. a_10.txt )

[root@gluster-client distvol]# for i in {1..10};do echo $i > a_$i.txt;done -bash: a_1.txt: Transport endpoint is not connected -bash: a_2.txt: Transport endpoint is not connected -bash: a_5.txt: Transport endpoint is not connected [root@gluster-client distvol]# ls 1.txt  10.txt  2.txt  3.txt  5.txt  6.txt  7.txt  a_10.txt  a_3.txt  a_4.txt  a_6.txt  a_7.txt  a_8.txt  a_9.txt

[root@gluster02 ~]# ls /bricks/dist_brick1/brick/ 1.txt  10.txt  2.txt  3.txt  5.txt  6.txt  7.txt  a_10.txt  a_3.txt  a_4.txt  a_6.txt  a_7.txt  a_8.txt  a_9.txt [root@gluster02 ~]#

on the client

[root@gluster-client distvol]# pwd /mnt/distvol [root@gluster-client distvol]# rm -f *.txt [root@gluster-client distvol]# ls [root@gluster-client distvol]# [root@gluster-client distvol]# for i in {1..10};do echo $i > a_$i.txt;done -bash: a_1.txt: Transport endpoint is not connected -bash: a_2.txt: Transport endpoint is not connected -bash: a_5.txt: Transport endpoint is not connected [root@gluster-client distvol]#

resume gluster01 VM

virsh # resume cent7-gluster01 Domain cent7-gluster01 resumed virsh #

status is connected

[root@gluster02 ~]# gluster peer status Number of Peers: 1 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Connected) [root@gluster01 ~]# gluster peer status Number of Peers: 1 Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected)

[root@gluster-client distvol]# rm *.txt -f [root@gluster-client distvol]# for i in {1..10};do echo $i > a_$i.txt;done [root@gluster-client distvol]#

[ replicate volume ]

set up a replicate volume on gluster03 and 04 servers.

on the gluster01

[root@gluster01 ~]# gluster peer probe gluster03 peer probe: success. [root@gluster01 ~]# gluster peer probe gluster04 peer probe: success. [root@gluster01 ~]#

$ ansible gluster -m shell -a 'gluster peer status' 192.168.130.112 | success | rc=0 >> Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Connected) Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster04 Uuid: 905205fb-291b-4d54-b786-00b22a4cdcd9 State: Peer in Cluster (Connected) 192.168.130.111 | success | rc=0 >> Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Connected) Hostname: gluster03 Uuid: 0008cf97-7ed1-46d4-b795-9d7bd03c2667 State: Peer in Cluster (Connected) Hostname: gluster04 Uuid: 905205fb-291b-4d54-b786-00b22a4cdcd9 State: Peer in Cluster (Connected) 192.168.130.113 | success | rc=0 >> Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Connected) Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster03 Uuid: 0008cf97-7ed1-46d4-b795-9d7bd03c2667 State: Peer in Cluster (Connected) 192.168.130.110 | success | rc=0 >> Number of Peers: 3 Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster03 Uuid: 0008cf97-7ed1-46d4-b795-9d7bd03c2667 State: Peer in Cluster (Connected) Hostname: gluster04 Uuid: 905205fb-291b-4d54-b786-00b22a4cdcd9 State: Peer in Cluster (Connected)

on the gluster03, create a volume.

[root@gluster03 ~]# pvcreate /dev/vdb ; vgcreate vg_bricks /dev/vdb [root@gluster03 ~]# lvcreate -L 2G -n lv-brick01 vg_bricks [root@gluster03 ~]# mkfs.xfs -i size=512 /dev/vg_bricks/lv-brick01 [root@gluster03 ~]# mkdir -p /bricks/shadow_brick1 [root@gluster03 ~]# mount /dev/vg_bricks/lv-brick01 /bricks/shadow_brick1 [root@gluster03 ~]# mkdir /bricks/shadow_brick1/brick

on the gluster04

[root@gluster04 ~]# pvcreate /dev/vdb ; vgcreate vg_bricks /dev/vdb  Physical volume "/dev/vdb" successfully created  Volume group "vg_bricks" successfully created [root@gluster04 ~]#  lvcreate -L 2G -n lv-brick01 vg_bricks  Logical volume "lv-brick01" created. [root@gluster04 ~]# mkfs.xfs -i size=512 /dev/vg_bricks/lv-brick01 meta-data=/dev/vg_bricks/lv-brick01 isize=512    agcount=4, agsize=131072 blks         =                       sectsz=512   attr=2, projid32bit=1         =                       crc=0        finobt=0 data     =                       bsize=4096   blocks=524288, imaxpct=25         =                       sunit=0      swidth=0 blks naming   =version 2              bsize=4096   ascii-ci=0 ftype=0 log      =internal log           bsize=4096   blocks=2560, version=2         =                       sectsz=512   sunit=0 blks, lazy-count=1 realtime =none                   extsz=4096   blocks=0, rtextents=0 [root@gluster04 ~]# mkdir -p /bricks/shadow_brick1 [root@gluster04 ~]# mount /dev/vg_bricks/lv-brick01 /bricks/shadow_brick1 [root@gluster04 ~]# mkdir /bricks/shadow_brick1/brick [root@gluster04 ~]#

create a replicate volume on the gluster03

[root@gluster03 ~]# gluster volume create shadowvol replica 2 gluster03:/bricks/shadow_brick1/brick gluster04:/bricks/shado w_brick1/brick volume create: shadowvol: success: please start the volume to access data [root@gluster03 ~]# [root@gluster03 ~]# gluster volume start shadowvol volume start: shadowvol: success [root@gluster03 ~]# gluster volume info shadowvol Volume Name: shadowvol Type: Replicate Volume ID: 145c0bb8-0fe9-45b8-861e-3f67a099db82 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gluster03:/bricks/shadow_brick1/brick Brick2: gluster04:/bricks/shadow_brick1/brick Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on [root@gluster03 ~]#

on the client

[root@gluster-client ~]# mkdir /mnt/shadowvol [root@gluster-client ~]# mount -t glusterfs gluster03:/shadowvol /mnt/shadowvol/ [root@gluster-client ~]# oot@gluster-client ~]# mount| grep glusterfs gluster01:/distvol on /mnt/distvol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072) gluster03:/shadowvol on /mnt/shadowvol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

create files

[root@gluster-client ~]# cd /mnt/shadowvol/ [root@gluster-client shadowvol]# for i in {1..10};do echo $i > a_$i.txt;done [root@gluster-client shadowvol]# ls a_1.txt  a_10.txt  a_2.txt  a_3.txt  a_4.txt  a_5.txt  a_6.txt  a_7.txt  a_8.txt  a_9.txt [root@gluster-client shadowvol]#

all files are replicated on both gluster03 and gluster04.

[root@gluster03 ~]# ls /bricks/shadow_brick1/brick/ a_1.txt  a_10.txt  a_2.txt  a_3.txt  a_4.txt  a_5.txt  a_6.txt  a_7.txt  a_8.txt  a_9.txt [root@gluster03 ~]# [root@gluster04 ~]# ls /bricks/shadow_brick1/brick/ a_1.txt  a_10.txt  a_2.txt  a_3.txt  a_4.txt  a_5.txt  a_6.txt  a_7.txt  a_8.txt  a_9.txt [root@gluster04 ~]#

suspend gluster04 VM.

virsh # suspend cent7-gluster04 Domain cent7-gluster04 suspended virsh #

on the gluster03

[root@gluster03 ~]# gluster peer status Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Connected) Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster04 Uuid: 905205fb-291b-4d54-b786-00b22a4cdcd9 State: Peer in Cluster (Disconnected) [root@gluster03 ~]#

create files on the client.

[root@gluster-client shadowvol]# for i in {1..10};do echo $i > b_$i.txt;done [root@gluster-client shadowvol]# ls b_* b_1.txt  b_10.txt  b_2.txt  b_3.txt  b_4.txt  b_5.txt  b_6.txt  b_7.txt  b_8.txt  b_9.txt [root@gluster-client shadowvol]#

resume gluster04 VM.

virsh # resume cent7-gluster04 Domain cent7-gluster04 resumed virsh #

on the gluster04

all files start with b_*.txt were replicated on the gluster04.

[root@gluster04 ~]# gluster peer status Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Connected) Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster03 Uuid: 0008cf97-7ed1-46d4-b795-9d7bd03c2667 State: Peer in Cluster (Connected) [root@gluster04 ~]# [root@gluster04 ~]# ls /bricks/shadow_brick1/brick/ a_1.txt   a_2.txt  a_4.txt  a_6.txt  a_8.txt  b_1.txt   b_2.txt  b_4.txt  b_6.txt  b_8.txt a_10.txt  a_3.txt  a_5.txt  a_7.txt  a_9.txt  b_10.txt  b_3.txt  b_5.txt  b_7.txt  b_9.txt [root@gluster04 ~]#

[ distributed and replicated volume ]

gluster01

[root@gluster01 ~]# lvcreate -L 100M -n prod_brick1 vg-bricks [root@gluster01 ~]# mkfs.xfs -i size=512 /dev/vg-bricks/prod_brick1 [root@gluster01 ~]# mkdir -p /bricks/prod_brick1 [root@gluster01 ~]# mount /dev/vg-bricks/prod_brick1 /bricks/prod_brick1/ [root@gluster01 ~]# mkdir /bricks/prod_brick1/brick

gluster02

[root@gluster02 ~]# lvcreate -L 100M -n prod_brick1 vg-bricks [root@gluster02 ~]# mkfs.xfs -i size=512 /dev/vg-bricks/prod_brick1 [root@gluster02 ~]# mkdir -p /bricks/prod_brick1 [root@gluster02 ~]#  mount /dev/vg-bricks/prod_brick1 /bricks/prod_brick1/ [root@gluster02 ~]# mkdir /bricks/prod_brick1/brick [root@gluster02 ~]#

gluster03

38  lvcreate -L 100M -n prod_brick1 vg_bricks   40  mkfs.xfs -i size=512 /dev/vg_bricks/prod_brick1   41  mkdir -p /bricks/prod_brick1  45  mount /dev/vg_bricks/prod_brick1 /bricks/prod_brick1/   46  mkdir /bricks/prod_brick1/brick

gluster04

[root@gluster04 ~]#  lvcreate -L 100M -n prod_brick1 vg_bricks  Logical volume "prod_brick1" created. [root@gluster04 ~]# mkfs.xfs -i size=512 /dev/vg_bricks/prod_brick1 [root@gluster04 ~]# mkdir -p /bricks/prod_brick1 [root@gluster04 ~]# mount /dev/vg_bricks/prod_brick1 /bricks/prod_brick1/ [root@gluster04 ~]# mkdir /bricks/prod_brick1/brick [root@gluster04 ~]#

create a volume.

[root@gluster01 ~]# gluster volume create dist-rep-vol replica 2 gluster01:/bricks/prod_brick1/brick gluster02:/bricks/prod_brick1/brick gluster03:/bricks/prod_brick1/brick gluster04:/bricks/prod_brick1/brick force volume create: dist-rep-vol: success: please start the volume to access data [root@gluster01 ~]# [root@gluster01 ~]# gluster volume start dist-rep-vol volume start: dist-rep-vol: success [root@gluster01 ~]# [root@gluster01 ~]# gluster volume info dist-rep-vol Volume Name: dist-rep-vol Type: Distributed-Replicate Volume ID: d3114800-9ab1-4c9d-99a2-160ae7dd5fc9 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gluster01:/bricks/prod_brick1/brick Brick2: gluster02:/bricks/prod_brick1/brick Brick3: gluster03:/bricks/prod_brick1/brick Brick4: gluster04:/bricks/prod_brick1/brick Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on [root@gluster01 ~]#

on the client

[root@gluster-client ~]# mkdir  /mnt/dist-rep-vol [root@gluster-client ~]# mount.glusterfs gluster01:/dist-rep-vol /mnt/dist-rep-vol/ [root@gluster-client ~]# mount| grep glusterfs gluster01:/distvol on /mnt/distvol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072) gluster03:/shadowvol on /mnt/shadowvol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) gluster01:/dist-rep-vol on /mnt/dist-rep-vol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [root@gluster-client ~]#

create files

[root@gluster-client ~]# cd /mnt/dist-rep-vol/ [root@gluster-client dist-rep-vol]# [root@gluster-client dist-rep-vol]# for i in {1..10};do echo $i > a_$i.txt;done [root@gluster-client dist-rep-vol]# ls a_1.txt  a_10.txt  a_2.txt  a_3.txt  a_4.txt  a_5.txt  a_6.txt  a_7.txt  a_8.txt  a_9.txt [root@gluster-client dist-rep-vol]#

110 : gluster01

111 : gluster02

112 : gluster03

113 : gluster04

$ ansible gluster -m shell -a 'ls /bricks/prod_brick1/brick' 192.168.130.110 | success | rc=0 >> a_1.txt a_2.txt a_5.txt 192.168.130.112 | success | rc=0 >> a_10.txt a_3.txt a_4.txt a_6.txt a_7.txt a_8.txt a_9.txt 192.168.130.113 | success | rc=0 >> a_10.txt a_3.txt a_4.txt a_6.txt a_7.txt a_8.txt a_9.txt 192.168.130.111 | success | rc=0 >> a_1.txt a_2.txt a_5.txt

[root@gluster-client dist-rep-vol]# rm a_*.txt -f [root@gluster-client dist-rep-vol]#

suspend gluster01

virsh # suspend cent7-gluster01 Domain cent7-gluster01 suspended virsh #

$ ansible gluster -m shell -a 'gluster peer status' 192.168.130.113 | success | rc=0 >> Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Disconnected) Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster03 Uuid: 0008cf97-7ed1-46d4-b795-9d7bd03c2667 State: Peer in Cluster (Connected) 192.168.130.112 | success | rc=0 >> Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Disconnected) Hostname: gluster02 Uuid: e8a4bb2e-3289-4002-9f33-0775f39e9588 State: Peer in Cluster (Connected) Hostname: gluster04 Uuid: 905205fb-291b-4d54-b786-00b22a4cdcd9 State: Peer in Cluster (Connected) 192.168.130.111 | success | rc=0 >> Number of Peers: 3 Hostname: gluster01 Uuid: 04666978-f467-497f-9d7a-86eec03e0ed7 State: Peer in Cluster (Disconnected) Hostname: gluster03 Uuid: 0008cf97-7ed1-46d4-b795-9d7bd03c2667 State: Peer in Cluster (Connected) Hostname: gluster04 Uuid: 905205fb-291b-4d54-b786-00b22a4cdcd9 State: Peer in Cluster (Connected) 192.168.130.110 | FAILED => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

confirm client can access to the other gluster servers, gluster02,03,04.

on the client

[root@gluster-client dist-rep-vol]# pwd /mnt/dist-rep-vol [root@gluster-client dist-rep-vol]# for i in {1..10};do echo $i > b_$i.txt;done [root@gluster-client dist-rep-vol]# ls b_1.txt  b_10.txt  b_2.txt  b_3.txt  b_4.txt  b_5.txt  b_6.txt  b_7.txt  b_8.txt  b_9.txt [root@gluster-client dist-rep-vol]#

$ ansible gluster -m shell -a 'ls /bricks/prod_brick1/brick' 192.168.130.111 | success | rc=0 >> b_10.txt b_4.txt b_5.txt b_6.txt b_8.txt b_9.txt 192.168.130.113 | success | rc=0 >> b_1.txt b_2.txt b_3.txt b_7.txt 192.168.130.112 | success | rc=0 >> b_1.txt b_2.txt b_3.txt b_7.txt 192.168.130.110 | FAILED => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

resume gluster01

virsh # resume  cent7-gluster01 Domain cent7-gluster01 resumed virsh #

[root@gluster02 ~]# gluster volume status dist-rep-vol Status of volume: dist-rep-vol Gluster process                             TCP Port  RDMA Port  Online  Pid ------------------------------------------------------------------------------ Brick gluster01:/bricks/prod_brick1/brick   49153     0          Y       3193 Brick gluster02:/bricks/prod_brick1/brick   49153     0          Y       11277 Brick gluster03:/bricks/prod_brick1/brick   49153     0          Y       11299 Brick gluster04:/bricks/prod_brick1/brick   49153     0          Y       2923 Self-heal Daemon on localhost               N/A       N/A        Y       11458 Self-heal Daemon on gluster03               N/A       N/A        Y       11442 Self-heal Daemon on gluster04               N/A       N/A        Y       3067 Self-heal Daemon on gluster01               N/A       N/A        Y       3280 Task Status of Volume dist-rep-vol ------------------------------------------------------------------------------ There are no active volume tasks