lost and found ( for me ? )

Ubuntu10.10 Server 64bit ver : KVM Live Migration w/ virsh


Here’s an example how to do Live migration on KVM

[ Network Topology ]

ubuntu-1  : Ubuntu 10.10 server    ---- L2SW ---- shared storage ( NFS : CentOS5.5 )
ubuntu-2  : Ubuntu 10.10 server    -------|

NFS server : 192.168.10.13
ubuntu-1 :192.168.10.14
ubuntu-2  : 192.168.10.15
VM for live migration ( ubuntu-vm2 ) : 192.168.10.101
both servers (ubuntu-1 n’ ubuntu-2 ) can resolve hostname each other not IP address

[ To do KVM live migration ]

you need to do:

Both KVM machines have same type of CPU
prepare shared storage such as NFS , iSCSI , GFS2.
prepare a VM which is stored in shared storage
VM must run w/ same KVM version
VM must have same network configuration ( I configure VM’s network as bridge )

[ NFS Server ( CentOS) ]

1. set up NFS server on CentOS5.5 for live migration

Usually NFS server is installed.
If you haven’t installed NFS , just type “yum install nfs-utils”
# yum install nfs-utils


create a folder for KVM machines
# mkdir /shared_storage


edit /etc/exports file
# cat /etc/exports
/shared_storage 192.168.10.0/255.255.255.0(rw,no_root_squash,async)


start NFS
# /etc/init.d/nfs start
NFS サービスを起動中:                                      [  OK  ]
NFS クォータを起動中:                                      [  OK  ]
NFS デーモンを起動中:                                      [  OK  ]
NFS mountd を起動中:                                       [  OK  ]


check exported directories through NSF
# exportfs
/shared_storage
               192.168.10.0/255.255.255.0


[ NFS Client ( Ubuntu ) ]

install NFS client
# apt-get install nfs-common


see the list of all shared directories
# showmount -e 192.168.10.13
Export list for 192.168.10.13:
/shared_storage 192.168.10.0/255.255.255.0


confirm NFS client can mount NFS server ( OK )
# mount.nfs 192.168.10.13:/shared_storage /root/tmp

# cat /proc/mounts | grep shared
192.168.10.13:/shared_storage /root/tmp nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0


unmount
# umount.nfs /root/tmp/


how to have NFS client mount NFS server automatically.
edit /etc/fstab like below.
# mount NFS server
192.168.10.13:/shared_storage /root/tmp nfs rw,soft 0 0


Now I could confirm NFS client mount NFS server.
Next , I’ll set up the shared storage settings for live migration.

[ shared storage settings for KVM ]

assume shared storage path like below.

NFS server’s shared storage path : /var/lib/libvirt/shared_storage
KVM’s storage path : /var/lib/libvirt/shared_storage

on NFS server ( CentOS )

make a directory for shared_storage n’ export that directory
# mkdir -p /var/lib/libvirt/shared_storage


edit /etc/exports
# cat /etc/exports
/var/lib/libvirt/shared_storage 192.168.10.0/255.255.255.0(rw,no_root_squash,async)


do exportfs -ra to reflect on configuration
# exportfs -ra


show the list of all directories exported
# showmount -e
Export list for centos-5-3.localdomain:
/var/lib/libvirt/shared_storage 192.168.10.0/255.255.255.0


on KVM machines ( Ubuntu )

make a directory to mount NFS server on two KVM machines
root@ubuntu-1:~# mkdir -p /var/lib/libvirt/shared_storage
root@ubuntu-2:~# mkdir -p /var/lib/libvirt/shared_storage


mount
root@ubuntu-1:~# mount.nfs 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage
root@ubuntu-2:~# mount.nfs 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage


check KVM machines could mount
root@ubuntu-1:~# egrep shared /proc/mounts
192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0

root@ubuntu-2:~# egrep shared /proc/mounts
192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0


registre /var/lib/libvirt/shared_storage as KVM storage

edit xml file

on ubuntu-1
root@ubuntu-1:/etc/libvirt/storage# pwd
/etc/libvirt/storage

root@ubuntu-1:/etc/libvirt/storage# cat shared_storage.xml
<pool type='dir'>
 <name>shared_storage</name>
 <capacity>0</capacity>
 <allocation>0</allocation>
 <available>0</available>
 <source>
 </source>
 <target>
   <path>/var/lib/libvirt/shared_storage</path>
   <permissions>
     <mode>0700</mode>
     <owner>-1</owner>
     <group>-1</group>
   </permissions>
 </target>
</pool>
root@ubuntu-1:/etc/libvirt/storage#


on ubuntu-2
root@ubuntu-2:/etc/libvirt/storage# pwd
/etc/libvirt/storage

root@ubuntu-2:/etc/libvirt/storage# cat shared_storage.xml
<pool type='dir'>
 <name>shared_storage</name>
 <capacity>0</capacity>
 <allocation>0</allocation>
 <available>0</available>
 <source>
 </source>
 <target>
   <path>/var/lib/libvirt/shared_storage</path>
   <permissions>
     <mode>0700</mode>
     <owner>-1</owner>
     <group>-1</group>
   </permissions>
 </target>
</pool>


register /var/lib/libvirt/shared_storage as KVM’s pool

on ubuntu-1
root@ubuntu-1:~# virsh

virsh # pool-define /etc/libvirt/storage/shared_storage.xml
プール shared_storage が /etc/libvirt/storage/shared_storage.xml から定義されま した

virsh # pool-list --all
名前               状態     自動起動
-----------------------------------------
default              動作中  yes
disk1                動作中  yes
shared_storage       停止状態 no

virsh #


do same operation on ubuntu-2
root@ubuntu-2:~# virsh

virsh # pool-define /etc/libvirt/storage/shared_storage.xml
プール shared_storage が /etc/libvirt/storage/shared_storage.xml から定義されま した

virsh # pool-list --all
名前               状態     自動起動
-----------------------------------------
default              動作中  yes
disk1                動作中  yes
shared_storage       停止状態 no

virsh #


start shared_storage pool

ubuntu-1
virsh # pool-start shared_storage
プール shared_storage が起動されました

virsh # pool-list --all
名前               状態     自動起動
-----------------------------------------
default              動作中  yes
disk1                動作中  yes
shared_storage       動作中  no

virsh #


ubuntu-2
virsh # pool-list --all
名前               状態     自動起動
-----------------------------------------
default              動作中  yes
disk1                動作中  yes
shared_storage       停止状態 no

virsh # pool-start shared_storage
プール shared_storage が起動されました

virsh # pool-list --all
名前               状態     自動起動
-----------------------------------------
default              動作中  yes
disk1                動作中  yes
shared_storage       動作中  no

virsh #


prepare VM for live migration n’ store it in shared storage.
I created a clone VM of ubuntu-vm1 and named it ubuntu-vm2 which is stored in shared_storage ( NFS )  
root@ubuntu-1:~# virt-clone --connect qemu:///system --original ubuntu-vm1 --name ubuntu-vm2 -f /var/lib/libvirt/shared_storage/ubuntu-vm2.img
Cloning tmpEf1a8J.qcow2   100% |=========================|  107 B    00:05

Clone 'ubuntu-vm2' created successfully.
root@ubuntu-1:~#

root@ubuntu-1:~# virsh list --all
Id 名前               状態
----------------------------------
 - ubuntu-vm1           シャットオフ
 - ubuntu-vm2           シャットオフ

root@ubuntu-1:~# virsh dumpxml ubuntu-vm2 | grep shared
     <source file='/var/lib/libvirt/shared_storage/ubuntu-vm2.img'/>


There’s a VM image I copied on NFS server storage. ( on NFS Server )
[root@centos-5-3 ~]# ls /var/lib/libvirt/shared_storage/*
/var/lib/libvirt/shared_storage/ubuntu-vm2.img


confirm I can start ubuntu-vm2 from shared storage.
root@ubuntu-1:~# virsh
virsh にようこそ、仮想化対話式ターミナルです。

入力方法: 'help' コマンドに関するヘルプ
         'quit' 終了します

virsh # start ubuntu-vm2
ドメイン ubuntu-vm2 が起動されました


[ confitm ubuntu-1 can connect to ubuntu-2 n’ vice versa ]

confirm ubuntu-1 connect to ubuntu-2

on ubuntu-1
root@ubuntu-1:~# virsh -c qemu+ssh://192.168.10.15/system
root@192.168.10.15's password:
virsh にようこそ、仮想化対話式ターミナルです。

入力方法: 'help' コマンドに関するヘルプ
         'quit' 終了します

virsh # list --all
Id 名前               状態
----------------------------------
 - SL6-vm1              シャットオフ

virsh # quit


on ubuntu-2
root@ubuntu-2:~# virsh -c qemu+ssh://192.168.10.14/system
root@192.168.10.14's password:
virsh にようこそ、仮想化対話式ターミナルです。

入力方法: 'help' コマンドに関するヘルプ
         'quit' 終了します

virsh # list --all
Id 名前               状態
----------------------------------
 - ubuntu-vm1           シャットオフ
 - ubuntu-vm2           シャットオフ
virsh # quit

root@ubuntu-2:~#


[ do live migration ]

I’ll move ubuntu-vm2 VM that is running from KVM1(ubuntu-1) to KVM2(ubuntu-2)

fly ubuntu-vm2 on KVM1
root@ubuntu-1:~# virsh start ubuntu-vm2
ドメイン ubuntu-vm2 が起動されました

root@ubuntu-1:~# virsh list --all
Id 名前               状態
----------------------------------
 7 ubuntu-vm2           実行中
 - ubuntu-vm1           シャットオフ


ping to www.google.co.jp on ubuntu-vm2
root@ubuntu-1:~# virsh console ubuntu-vm2
Connected to domain ubuntu-vm2
エスケープ文字は  ^] です

Ubuntu 10.04.2 LTS ubuntu-vm2 ttyS0

ubuntu-vm2 login: root
Password:
Last login: Tue Mar 29 09:25:16 UTC 2011 on ttyS0
Linux ubuntu-vm2 2.6.32-30-generic-pae #59-Ubuntu SMP Tue Mar 1 23:01:33 UTC 2011 i686 GNU/Linux
Ubuntu 10.04.2 LTS

Welcome to Ubuntu!
* Documentation:  https://help.ubuntu.com/
root@ubuntu-vm2:~# ping www.google.co.jp
PING www.l.google.com (74.125.235.82) 56(84) bytes of data.
64 bytes from 74.125.235.82: icmp_seq=1 ttl=53 time=6.84 ms
64 bytes from 74.125.235.82: icmp_seq=2 ttl=53 time=9.05 ms


do live migration ( migrate ubunbu-vm2 from ubuntu-1 to ubuntu-2 )

failed ..
root@ubuntu-1:~# virsh migrate --live ubuntu-vm2 qemu+ssh://192.168.10.15/system
root@192.168.10.15's password:
エラー: operation failed: migration to 'tcp:ubuntu-2:49152' failed: migration failed


After registering hostname in /etc/hosts file , Live migration suceeded.

on ubuntu-1
root@ubuntu-1:~# egrep ubuntu-* /etc/hosts
127.0.0.1       ubuntu-1        localhost.localdomain   localhost
::1     ubuntu-1        localhost6.localdomain6 localhost6
127.0.1.1       ubuntu-1
192.168.10.15   ubuntu-2


on ubuntu-2
root@ubuntu-2:~# egrep ubuntu-* /etc/hosts
127.0.0.1       ubuntu-2        localhost.localdomain   localhost
::1     ubuntu-2        localhost6.localdomain6 localhost6
127.0.1.1       ubuntu-2
192.168.10.14   ubuntu-1


do live migration again.
root@ubuntu-1:~# virsh migrate --live ubuntu-vm2 qemu+ssh://ubuntu-2/system
Are you sure you want to continue connecting (yes/no)? yes
root@ubuntu-2's password:

root@ubuntu-1:~# echo $?
0


Live migration was compeleted :)

ubuntu-vm2 moved to ubuntu-2 KVM machine n’ it’s running !

on ubuntu-1
root@ubuntu-1:~# virsh list --all
Id 名前               状態
----------------------------------
 - ubuntu-vm1           シャットオフ
 - ubuntu-vm2           シャットオフ


on ubuntu-2
root@ubuntu-2:~# virsh list --all
Id 名前               状態
----------------------------------
 7 ubuntu-vm2           実行中
 - SL6-vm1              シャットオフ

root@ubuntu-2:~# virsh console ubuntu-vm2
Connected to domain ubuntu-vm2
エスケープ文字は  ^] です
64 bytes from 74.125.235.82: icmp_seq=2125 ttl=53 time=4.99 ms
64 bytes from 74.125.235.82: icmp_seq=2126 ttl=53 time=6.91 ms
64 bytes from 74.125.235.82: icmp_seq=2127 ttl=53 time=5.30 ms
64 bytes from 74.125.235.82: icmp_seq=2128 ttl=53 time=7.97 ms