Ubuntu10.10 Server 64bit ver : KVM Live Migration w/ virsh

Here’s an example how to do Live migration on KVM

[ Network Topology ]

ubuntu-1  : Ubuntu 10.10 server    ---- L2SW ---- shared storage ( NFS : CentOS5.5 )
ubuntu-2  : Ubuntu 10.10 server    -------|

NFS server : 192.168.10.13
ubuntu-1 ：192.168.10.14
ubuntu-2  : 192.168.10.15
VM for live migration ( ubuntu-vm2 ) : 192.168.10.101
both servers (ubuntu-1 n’ ubuntu-2 ) can resolve hostname each other not IP address

[ To do KVM live migration ]

you need to do:

Both KVM machines have same type of CPU
prepare shared storage such as NFS , iSCSI , GFS2.
prepare a VM which is stored in shared storage
VM must run w/ same KVM version
VM must have same network configuration ( I configure VM’s network as bridge )

[ NFS Server ( CentOS) ]

1. set up NFS server on CentOS5.5 for live migration

Usually NFS server is installed.
If you haven’t installed NFS , just type “yum install nfs-utils”

# yum install nfs-utils

create a folder for KVM machines

# mkdir /shared_storage

edit /etc/exports file

# cat /etc/exports /shared_storage 192.168.10.0/255.255.255.0(rw,no_root_squash,async)

start NFS

# /etc/init.d/nfs start NFS サービスを起動中:                                      [  OK  ] NFS クォータを起動中:                                      [  OK  ] NFS デーモンを起動中:                                      [  OK  ] NFS mountd を起動中:                                       [  OK  ]

check exported directories through NSF

# exportfs /shared_storage                192.168.10.0/255.255.255.0

[ NFS Client ( Ubuntu ) ]

install NFS client

# apt-get install nfs-common

see the list of all shared directories

# showmount -e 192.168.10.13 Export list for 192.168.10.13: /shared_storage 192.168.10.0/255.255.255.0

confirm NFS client can mount NFS server ( OK )

# mount.nfs 192.168.10.13:/shared_storage /root/tmp

# cat /proc/mounts | grep shared 192.168.10.13:/shared_storage /root/tmp nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0

unmount

# umount.nfs /root/tmp/

how to have NFS client mount NFS server automatically.
edit /etc/fstab like below.

# mount NFS server 192.168.10.13:/shared_storage /root/tmp nfs rw,soft 0 0

Now I could confirm NFS client mount NFS server.
Next , I’ll set up the shared storage settings for live migration.

[ shared storage settings for KVM ]

assume shared storage path like below.

NFS server’s shared storage path : /var/lib/libvirt/shared_storage
KVM’s storage path : /var/lib/libvirt/shared_storage

on NFS server ( CentOS )

make a directory for shared_storage n’ export that directory

# mkdir -p /var/lib/libvirt/shared_storage

edit /etc/exports

# cat /etc/exports /var/lib/libvirt/shared_storage 192.168.10.0/255.255.255.0(rw,no_root_squash,async)

do exportfs -ra to reflect on configuration

# exportfs -ra

show the list of all directories exported

# showmount -e Export list for centos-5-3.localdomain: /var/lib/libvirt/shared_storage 192.168.10.0/255.255.255.0

on KVM machines ( Ubuntu )

make a directory to mount NFS server on two KVM machines

root@ubuntu-1:~# mkdir -p /var/lib/libvirt/shared_storage root@ubuntu-2:~# mkdir -p /var/lib/libvirt/shared_storage

mount

root@ubuntu-1:~# mount.nfs 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage root@ubuntu-2:~# mount.nfs 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage

check KVM machines could mount

root@ubuntu-1:~# egrep shared /proc/mounts 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0

root@ubuntu-2:~# egrep shared /proc/mounts 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0

registre /var/lib/libvirt/shared_storage as KVM storage

edit xml file

on ubuntu-1

root@ubuntu-1:/etc/libvirt/storage# pwd /etc/libvirt/storage root@ubuntu-1:/etc/libvirt/storage# cat shared_storage.xml <pool type='dir'>  <name>shared_storage</name>  <capacity>0</capacity>  <allocation>0</allocation>  <available>0</available>  <source>  </source>  <target>    <path>/var/lib/libvirt/shared_storage</path>    <permissions>      <mode>0700</mode>      <owner>-1</owner>      <group>-1</group>    </permissions>  </target> </pool> root@ubuntu-1:/etc/libvirt/storage#

on ubuntu-2

root@ubuntu-2:/etc/libvirt/storage# pwd /etc/libvirt/storage root@ubuntu-2:/etc/libvirt/storage# cat shared_storage.xml <pool type='dir'>  <name>shared_storage</name>  <capacity>0</capacity>  <allocation>0</allocation>  <available>0</available>  <source>  </source>  <target>    <path>/var/lib/libvirt/shared_storage</path>    <permissions>      <mode>0700</mode>      <owner>-1</owner>      <group>-1</group>    </permissions>  </target> </pool>

register /var/lib/libvirt/shared_storage as KVM’s pool

on ubuntu-1

root@ubuntu-1:~# virsh virsh # pool-define /etc/libvirt/storage/shared_storage.xml プール shared_storage が /etc/libvirt/storage/shared_storage.xml から定義されま した virsh # pool-list --all 名前               状態     自動起動 ----------------------------------------- default              動作中  yes disk1                動作中  yes shared_storage       停止状態 no virsh #

do same operation on ubuntu-2

root@ubuntu-2:~# virsh virsh # pool-define /etc/libvirt/storage/shared_storage.xml プール shared_storage が /etc/libvirt/storage/shared_storage.xml から定義されま した virsh # pool-list --all 名前               状態     自動起動 ----------------------------------------- default              動作中  yes disk1                動作中  yes shared_storage       停止状態 no virsh #

start shared_storage pool

ubuntu-1

virsh # pool-start shared_storage プール shared_storage が起動されました virsh # pool-list --all 名前               状態     自動起動 ----------------------------------------- default              動作中  yes disk1                動作中  yes shared_storage       動作中  no virsh #

ubuntu-2

virsh # pool-list --all 名前               状態     自動起動 ----------------------------------------- default              動作中  yes disk1                動作中  yes shared_storage       停止状態 no virsh # pool-start shared_storage プール shared_storage が起動されました virsh # pool-list --all 名前               状態     自動起動 ----------------------------------------- default              動作中  yes disk1                動作中  yes shared_storage       動作中  no virsh #

prepare VM for live migration n’ store it in shared storage.
I created a clone VM of ubuntu-vm1 and named it ubuntu-vm2 which is stored in shared_storage ( NFS )  

root@ubuntu-1:~# virt-clone --connect qemu:///system --original ubuntu-vm1 --name ubuntu-vm2 -f /var/lib/libvirt/shared_storage/ubuntu-vm2.img Cloning tmpEf1a8J.qcow2   100% |=========================|  107 B    00:05 Clone 'ubuntu-vm2' created successfully. root@ubuntu-1:~#

root@ubuntu-1:~# virsh list --all Id 名前               状態 ----------------------------------  - ubuntu-vm1           シャットオフ  - ubuntu-vm2           シャットオフ root@ubuntu-1:~# virsh dumpxml ubuntu-vm2 | grep shared      <source file='/var/lib/libvirt/shared_storage/ubuntu-vm2.img'/>

There’s a VM image I copied on NFS server storage. ( on NFS Server )

[root@centos-5-3 ~]# ls /var/lib/libvirt/shared_storage/* /var/lib/libvirt/shared_storage/ubuntu-vm2.img

confirm I can start ubuntu-vm2 from shared storage.

root@ubuntu-1:~# virsh virsh にようこそ、仮想化対話式ターミナルです。 入力方法: 'help' コマンドに関するヘルプ          'quit' 終了します virsh # start ubuntu-vm2 ドメイン ubuntu-vm2 が起動されました

[ confitm ubuntu-1 can connect to ubuntu-2 n’ vice versa ]

confirm ubuntu-1 connect to ubuntu-2

on ubuntu-1

root@ubuntu-1:~# virsh -c qemu+ssh://192.168.10.15/system root@192.168.10.15's password: virsh にようこそ、仮想化対話式ターミナルです。 入力方法: 'help' コマンドに関するヘルプ          'quit' 終了します virsh # list --all Id 名前               状態 ----------------------------------  - SL6-vm1              シャットオフ virsh # quit

on ubuntu-2

root@ubuntu-2:~# virsh -c qemu+ssh://192.168.10.14/system root@192.168.10.14's password: virsh にようこそ、仮想化対話式ターミナルです。 入力方法: 'help' コマンドに関するヘルプ          'quit' 終了します virsh # list --all Id 名前               状態 ----------------------------------  - ubuntu-vm1           シャットオフ  - ubuntu-vm2           シャットオフ virsh # quit root@ubuntu-2:~#

[ do live migration ]

I’ll move ubuntu-vm2 VM that is running from KVM1(ubuntu-1) to KVM2(ubuntu-2)

fly ubuntu-vm2 on KVM1

root@ubuntu-1:~# virsh start ubuntu-vm2 ドメイン ubuntu-vm2 が起動されました root@ubuntu-1:~# virsh list --all Id 名前               状態 ----------------------------------  7 ubuntu-vm2           実行中  - ubuntu-vm1           シャットオフ

ping to www.google.co.jp on ubuntu-vm2

root@ubuntu-1:~# virsh console ubuntu-vm2 Connected to domain ubuntu-vm2 エスケープ文字は  ^] です Ubuntu 10.04.2 LTS ubuntu-vm2 ttyS0 ubuntu-vm2 login: root Password: Last login: Tue Mar 29 09:25:16 UTC 2011 on ttyS0 Linux ubuntu-vm2 2.6.32-30-generic-pae #59-Ubuntu SMP Tue Mar 1 23:01:33 UTC 2011 i686 GNU/Linux Ubuntu 10.04.2 LTS Welcome to Ubuntu! * Documentation:  https://help.ubuntu.com/ root@ubuntu-vm2:~# ping www.google.co.jp PING www.l.google.com (74.125.235.82) 56(84) bytes of data. 64 bytes from 74.125.235.82: icmp_seq=1 ttl=53 time=6.84 ms 64 bytes from 74.125.235.82: icmp_seq=2 ttl=53 time=9.05 ms

do live migration ( migrate ubunbu-vm2 from ubuntu-1 to ubuntu-2 )

failed ..

root@ubuntu-1:~# virsh migrate --live ubuntu-vm2 qemu+ssh://192.168.10.15/system root@192.168.10.15's password: エラー: operation failed: migration to 'tcp:ubuntu-2:49152' failed: migration failed

After registering hostname in /etc/hosts file , Live migration suceeded.

on ubuntu-1

root@ubuntu-1:~# egrep ubuntu-* /etc/hosts 127.0.0.1       ubuntu-1        localhost.localdomain   localhost ::1     ubuntu-1        localhost6.localdomain6 localhost6 127.0.1.1       ubuntu-1 192.168.10.15   ubuntu-2

on ubuntu-2

root@ubuntu-2:~# egrep ubuntu-* /etc/hosts 127.0.0.1       ubuntu-2        localhost.localdomain   localhost ::1     ubuntu-2        localhost6.localdomain6 localhost6 127.0.1.1       ubuntu-2 192.168.10.14   ubuntu-1

do live migration again.

root@ubuntu-1:~# virsh migrate --live ubuntu-vm2 qemu+ssh://ubuntu-2/system Are you sure you want to continue connecting (yes/no)? yes root@ubuntu-2's password: root@ubuntu-1:~# echo $? 0

Live migration was compeleted :)

ubuntu-vm2 moved to ubuntu-2 KVM machine n’ it’s running !

on ubuntu-1

root@ubuntu-1:~# virsh list --all Id 名前               状態 ----------------------------------  - ubuntu-vm1           シャットオフ  - ubuntu-vm2           シャットオフ

on ubuntu-2

root@ubuntu-2:~# virsh list --all Id 名前               状態 ----------------------------------  7 ubuntu-vm2           実行中  - SL6-vm1              シャットオフ root@ubuntu-2:~# virsh console ubuntu-vm2 Connected to domain ubuntu-vm2 エスケープ文字は  ^] です 64 bytes from 74.125.235.82: icmp_seq=2125 ttl=53 time=4.99 ms 64 bytes from 74.125.235.82: icmp_seq=2126 ttl=53 time=6.91 ms 64 bytes from 74.125.235.82: icmp_seq=2127 ttl=53 time=5.30 ms 64 bytes from 74.125.235.82: icmp_seq=2128 ttl=53 time=7.97 ms