Here’s an example how to do Live migration on KVM
[ Network Topology ]
ubuntu-1 : Ubuntu 10.10 server ---- L2SW ---- shared storage ( NFS : CentOS5.5 )
ubuntu-2 : Ubuntu 10.10 server -------|
NFS server : 192.168.10.13
ubuntu-1 :192.168.10.14
ubuntu-2 : 192.168.10.15
VM for live migration ( ubuntu-vm2 ) : 192.168.10.101
both servers (ubuntu-1 n’ ubuntu-2 ) can resolve hostname each other not IP address
[ To do KVM live migration ]
you need to do:
Both KVM machines have same type of CPU
prepare shared storage such as NFS , iSCSI , GFS2.
prepare a VM which is stored in shared storage
VM must run w/ same KVM version
VM must have same network configuration ( I configure VM’s network as bridge )
[ NFS Server ( CentOS) ]
1. set up NFS server on CentOS5.5 for live migration
Usually NFS server is installed.
If you haven’t installed NFS , just type “yum install nfs-utils”
create a folder for KVM machines
edit /etc/exports file
start NFS
check exported directories through NSF
[ NFS Client ( Ubuntu ) ]
install NFS client
see the list of all shared directories
confirm NFS client can mount NFS server ( OK )
unmount
how to have NFS client mount NFS server automatically.
edit /etc/fstab like below.
Now I could confirm NFS client mount NFS server.
Next , I’ll set up the shared storage settings for live migration.
[ shared storage settings for KVM ]
assume shared storage path like below.
NFS server’s shared storage path : /var/lib/libvirt/shared_storage
KVM’s storage path : /var/lib/libvirt/shared_storage
on NFS server ( CentOS )
make a directory for shared_storage n’ export that directory
edit /etc/exports
do exportfs -ra to reflect on configuration
show the list of all directories exported
on KVM machines ( Ubuntu )
make a directory to mount NFS server on two KVM machines
mount
check KVM machines could mount
registre /var/lib/libvirt/shared_storage as KVM storage
edit xml file
on ubuntu-1
on ubuntu-2
register /var/lib/libvirt/shared_storage as KVM’s pool
on ubuntu-1
do same operation on ubuntu-2
start shared_storage pool
ubuntu-1
ubuntu-2
prepare VM for live migration n’ store it in shared storage.
I created a clone VM of ubuntu-vm1 and named it ubuntu-vm2 which is stored in shared_storage ( NFS )
There’s a VM image I copied on NFS server storage. ( on NFS Server )
confirm I can start ubuntu-vm2 from shared storage.
[ confitm ubuntu-1 can connect to ubuntu-2 n’ vice versa ]
confirm ubuntu-1 connect to ubuntu-2
on ubuntu-1
on ubuntu-2
[ do live migration ]
I’ll move ubuntu-vm2 VM that is running from KVM1(ubuntu-1) to KVM2(ubuntu-2)
fly ubuntu-vm2 on KVM1
ping to www.google.co.jp on ubuntu-vm2
do live migration ( migrate ubunbu-vm2 from ubuntu-1 to ubuntu-2 )
failed ..
After registering hostname in /etc/hosts file , Live migration suceeded.
on ubuntu-1
on ubuntu-2
do live migration again.
Live migration was compeleted :)
ubuntu-vm2 moved to ubuntu-2 KVM machine n’ it’s running !
on ubuntu-1
on ubuntu-2
[ Network Topology ]
ubuntu-1 : Ubuntu 10.10 server ---- L2SW ---- shared storage ( NFS : CentOS5.5 )
ubuntu-2 : Ubuntu 10.10 server -------|
NFS server : 192.168.10.13
ubuntu-1 :192.168.10.14
ubuntu-2 : 192.168.10.15
VM for live migration ( ubuntu-vm2 ) : 192.168.10.101
both servers (ubuntu-1 n’ ubuntu-2 ) can resolve hostname each other not IP address
[ To do KVM live migration ]
you need to do:
Both KVM machines have same type of CPU
prepare shared storage such as NFS , iSCSI , GFS2.
prepare a VM which is stored in shared storage
VM must run w/ same KVM version
VM must have same network configuration ( I configure VM’s network as bridge )
[ NFS Server ( CentOS) ]
1. set up NFS server on CentOS5.5 for live migration
Usually NFS server is installed.
If you haven’t installed NFS , just type “yum install nfs-utils”
# yum install nfs-utils |
create a folder for KVM machines
# mkdir /shared_storage |
edit /etc/exports file
# cat /etc/exports /shared_storage 192.168.10.0/255.255.255.0(rw,no_root_squash,async) |
start NFS
# /etc/init.d/nfs start NFS サービスを起動中: [ OK ] NFS クォータを起動中: [ OK ] NFS デーモンを起動中: [ OK ] NFS mountd を起動中: [ OK ] |
check exported directories through NSF
# exportfs /shared_storage 192.168.10.0/255.255.255.0 |
[ NFS Client ( Ubuntu ) ]
install NFS client
# apt-get install nfs-common |
see the list of all shared directories
# showmount -e 192.168.10.13 Export list for 192.168.10.13: /shared_storage 192.168.10.0/255.255.255.0 |
confirm NFS client can mount NFS server ( OK )
# mount.nfs 192.168.10.13:/shared_storage /root/tmp |
# cat /proc/mounts | grep shared 192.168.10.13:/shared_storage /root/tmp nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0 |
unmount
# umount.nfs /root/tmp/ |
how to have NFS client mount NFS server automatically.
edit /etc/fstab like below.
# mount NFS server 192.168.10.13:/shared_storage /root/tmp nfs rw,soft 0 0 |
Now I could confirm NFS client mount NFS server.
Next , I’ll set up the shared storage settings for live migration.
[ shared storage settings for KVM ]
assume shared storage path like below.
NFS server’s shared storage path : /var/lib/libvirt/shared_storage
KVM’s storage path : /var/lib/libvirt/shared_storage
on NFS server ( CentOS )
make a directory for shared_storage n’ export that directory
# mkdir -p /var/lib/libvirt/shared_storage |
edit /etc/exports
# cat /etc/exports /var/lib/libvirt/shared_storage 192.168.10.0/255.255.255.0(rw,no_root_squash,async) |
do exportfs -ra to reflect on configuration
# exportfs -ra |
show the list of all directories exported
# showmount -e Export list for centos-5-3.localdomain: /var/lib/libvirt/shared_storage 192.168.10.0/255.255.255.0 |
on KVM machines ( Ubuntu )
make a directory to mount NFS server on two KVM machines
root@ubuntu-1:~# mkdir -p /var/lib/libvirt/shared_storage root@ubuntu-2:~# mkdir -p /var/lib/libvirt/shared_storage |
mount
root@ubuntu-1:~# mount.nfs 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage root@ubuntu-2:~# mount.nfs 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage |
check KVM machines could mount
root@ubuntu-1:~# egrep shared /proc/mounts 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0 |
root@ubuntu-2:~# egrep shared /proc/mounts 192.168.10.13:/var/lib/libvirt/shared_storage /var/lib/libvirt/shared_storage nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.13,mountvers=3,mountport=968,mountproto=udp,addr=192.168.10.13 0 0 |
registre /var/lib/libvirt/shared_storage as KVM storage
edit xml file
on ubuntu-1
root@ubuntu-1:/etc/libvirt/storage# pwd /etc/libvirt/storage root@ubuntu-1:/etc/libvirt/storage# cat shared_storage.xml <pool type='dir'> <name>shared_storage</name> <capacity>0</capacity> <allocation>0</allocation> <available>0</available> <source> </source> <target> <path>/var/lib/libvirt/shared_storage</path> <permissions> <mode>0700</mode> <owner>-1</owner> <group>-1</group> </permissions> </target> </pool> root@ubuntu-1:/etc/libvirt/storage# |
on ubuntu-2
root@ubuntu-2:/etc/libvirt/storage# pwd /etc/libvirt/storage root@ubuntu-2:/etc/libvirt/storage# cat shared_storage.xml <pool type='dir'> <name>shared_storage</name> <capacity>0</capacity> <allocation>0</allocation> <available>0</available> <source> </source> <target> <path>/var/lib/libvirt/shared_storage</path> <permissions> <mode>0700</mode> <owner>-1</owner> <group>-1</group> </permissions> </target> </pool> |
register /var/lib/libvirt/shared_storage as KVM’s pool
on ubuntu-1
root@ubuntu-1:~# virsh virsh # pool-define /etc/libvirt/storage/shared_storage.xml プール shared_storage が /etc/libvirt/storage/shared_storage.xml から定義されま した virsh # pool-list --all 名前 状態 自動起動 ----------------------------------------- default 動作中 yes disk1 動作中 yes shared_storage 停止状態 no virsh # |
do same operation on ubuntu-2
root@ubuntu-2:~# virsh virsh # pool-define /etc/libvirt/storage/shared_storage.xml プール shared_storage が /etc/libvirt/storage/shared_storage.xml から定義されま した virsh # pool-list --all 名前 状態 自動起動 ----------------------------------------- default 動作中 yes disk1 動作中 yes shared_storage 停止状態 no virsh # |
start shared_storage pool
ubuntu-1
virsh # pool-start shared_storage プール shared_storage が起動されました virsh # pool-list --all 名前 状態 自動起動 ----------------------------------------- default 動作中 yes disk1 動作中 yes shared_storage 動作中 no virsh # |
ubuntu-2
virsh # pool-list --all 名前 状態 自動起動 ----------------------------------------- default 動作中 yes disk1 動作中 yes shared_storage 停止状態 no virsh # pool-start shared_storage プール shared_storage が起動されました virsh # pool-list --all 名前 状態 自動起動 ----------------------------------------- default 動作中 yes disk1 動作中 yes shared_storage 動作中 no virsh # |
prepare VM for live migration n’ store it in shared storage.
I created a clone VM of ubuntu-vm1 and named it ubuntu-vm2 which is stored in shared_storage ( NFS )
root@ubuntu-1:~# virt-clone --connect qemu:///system --original ubuntu-vm1 --name ubuntu-vm2 -f /var/lib/libvirt/shared_storage/ubuntu-vm2.img Cloning tmpEf1a8J.qcow2 100% |=========================| 107 B 00:05 Clone 'ubuntu-vm2' created successfully. root@ubuntu-1:~# |
root@ubuntu-1:~# virsh list --all Id 名前 状態 ---------------------------------- - ubuntu-vm1 シャットオフ - ubuntu-vm2 シャットオフ root@ubuntu-1:~# virsh dumpxml ubuntu-vm2 | grep shared <source file='/var/lib/libvirt/shared_storage/ubuntu-vm2.img'/> |
There’s a VM image I copied on NFS server storage. ( on NFS Server )
[root@centos-5-3 ~]# ls /var/lib/libvirt/shared_storage/* /var/lib/libvirt/shared_storage/ubuntu-vm2.img |
confirm I can start ubuntu-vm2 from shared storage.
root@ubuntu-1:~# virsh virsh にようこそ、仮想化対話式ターミナルです。 入力方法: 'help' コマンドに関するヘルプ 'quit' 終了します virsh # start ubuntu-vm2 ドメイン ubuntu-vm2 が起動されました |
[ confitm ubuntu-1 can connect to ubuntu-2 n’ vice versa ]
confirm ubuntu-1 connect to ubuntu-2
on ubuntu-1
root@ubuntu-1:~# virsh -c qemu+ssh://192.168.10.15/system root@192.168.10.15's password: virsh にようこそ、仮想化対話式ターミナルです。 入力方法: 'help' コマンドに関するヘルプ 'quit' 終了します virsh # list --all Id 名前 状態 ---------------------------------- - SL6-vm1 シャットオフ virsh # quit |
on ubuntu-2
root@ubuntu-2:~# virsh -c qemu+ssh://192.168.10.14/system root@192.168.10.14's password: virsh にようこそ、仮想化対話式ターミナルです。 入力方法: 'help' コマンドに関するヘルプ 'quit' 終了します virsh # list --all Id 名前 状態 ---------------------------------- - ubuntu-vm1 シャットオフ - ubuntu-vm2 シャットオフ virsh # quit root@ubuntu-2:~# |
[ do live migration ]
I’ll move ubuntu-vm2 VM that is running from KVM1(ubuntu-1) to KVM2(ubuntu-2)
fly ubuntu-vm2 on KVM1
root@ubuntu-1:~# virsh start ubuntu-vm2 ドメイン ubuntu-vm2 が起動されました root@ubuntu-1:~# virsh list --all Id 名前 状態 ---------------------------------- 7 ubuntu-vm2 実行中 - ubuntu-vm1 シャットオフ |
ping to www.google.co.jp on ubuntu-vm2
root@ubuntu-1:~# virsh console ubuntu-vm2 Connected to domain ubuntu-vm2 エスケープ文字は ^] です Ubuntu 10.04.2 LTS ubuntu-vm2 ttyS0 ubuntu-vm2 login: root Password: Last login: Tue Mar 29 09:25:16 UTC 2011 on ttyS0 Linux ubuntu-vm2 2.6.32-30-generic-pae #59-Ubuntu SMP Tue Mar 1 23:01:33 UTC 2011 i686 GNU/Linux Ubuntu 10.04.2 LTS Welcome to Ubuntu! * Documentation: https://help.ubuntu.com/ root@ubuntu-vm2:~# ping www.google.co.jp PING www.l.google.com (74.125.235.82) 56(84) bytes of data. 64 bytes from 74.125.235.82: icmp_seq=1 ttl=53 time=6.84 ms 64 bytes from 74.125.235.82: icmp_seq=2 ttl=53 time=9.05 ms |
do live migration ( migrate ubunbu-vm2 from ubuntu-1 to ubuntu-2 )
failed ..
root@ubuntu-1:~# virsh migrate --live ubuntu-vm2 qemu+ssh://192.168.10.15/system root@192.168.10.15's password: エラー: operation failed: migration to 'tcp:ubuntu-2:49152' failed: migration failed |
After registering hostname in /etc/hosts file , Live migration suceeded.
on ubuntu-1
root@ubuntu-1:~# egrep ubuntu-* /etc/hosts 127.0.0.1 ubuntu-1 localhost.localdomain localhost ::1 ubuntu-1 localhost6.localdomain6 localhost6 127.0.1.1 ubuntu-1 192.168.10.15 ubuntu-2 |
on ubuntu-2
root@ubuntu-2:~# egrep ubuntu-* /etc/hosts 127.0.0.1 ubuntu-2 localhost.localdomain localhost ::1 ubuntu-2 localhost6.localdomain6 localhost6 127.0.1.1 ubuntu-2 192.168.10.14 ubuntu-1 |
do live migration again.
root@ubuntu-1:~# virsh migrate --live ubuntu-vm2 qemu+ssh://ubuntu-2/system Are you sure you want to continue connecting (yes/no)? yes root@ubuntu-2's password: root@ubuntu-1:~# echo $? 0 |
Live migration was compeleted :)
ubuntu-vm2 moved to ubuntu-2 KVM machine n’ it’s running !
on ubuntu-1
root@ubuntu-1:~# virsh list --all Id 名前 状態 ---------------------------------- - ubuntu-vm1 シャットオフ - ubuntu-vm2 シャットオフ |
on ubuntu-2
root@ubuntu-2:~# virsh list --all Id 名前 状態 ---------------------------------- 7 ubuntu-vm2 実行中 - SL6-vm1 シャットオフ root@ubuntu-2:~# virsh console ubuntu-vm2 Connected to domain ubuntu-vm2 エスケープ文字は ^] です 64 bytes from 74.125.235.82: icmp_seq=2125 ttl=53 time=4.99 ms 64 bytes from 74.125.235.82: icmp_seq=2126 ttl=53 time=6.91 ms 64 bytes from 74.125.235.82: icmp_seq=2127 ttl=53 time=5.30 ms 64 bytes from 74.125.235.82: icmp_seq=2128 ttl=53 time=7.97 ms |