I sometimes wonder my nested KVM environment works correctly or not.
Here is my trial and error logs when checking nested VMs.
[ success case ]
KVM host: Fedora19 ( AMD Phenom(tm) II X4 925 Processor )
VM: Ubuntu13.04 ( disk path IDE, storage format qcow2 )
[ failed case ]
KVM host : Fedora 19 64bit (AMD Phenom(tm) II X4 925 Processor)
VM : CentOS 6.4 64bit
- on the KVM host
confirm nested option is enabled ( in case of Intel chip, option name is “kvm_intel” instad of “kvm_amd” )
KVM host CPU info
[root@localhost ~]# cat /etc/fedora-release;uname -ri;qemu-system-x86_64 -version;cat /proc/cpuinfo
Fedora release 19 (Schrödinger’s Cat)
3.10.7-200.fc19.x86_64 x86_64
QEMU emulator version 1.6.0, Copyright (c) 2003-2008 Fabrice Bellard
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Phenom(tm) II X4 925 Processor
stepping : 2
microcode : 0x10000db
cpu MHz : 1600.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate npt lbrv svm_lock nrip_save
bogomips : 5599.72
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
|
enable nested
[root@localhost ~]# cat /sys/module/kvm_amd/parameters/nested
1
|
- on the VM, check CPU flags
[root@localhost ~]# egrep '(svm|vmx)' /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm 3dnowext 3dnow up rep_good extd_apicid unfair_spinlock pni cx16 popcnt hypervisor cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw
|
- install kvm on the VM
[root@localhost ~]# yum groupinstall Virtualization
[root@localhost ~]# yum groupinstall "Virtualization Client"
[root@localhost ~]# yum install virt-install libvirt libvirt-python libvirt-client
|
reboot the VM
[root@localhost ~]# reboot
|
after booting, confirm kvm and kvm_intel or kvm_amd module has been loaded
on the VM
nnn, not enable nested.
on the VM
[root@localhost ~]# lsmod | grep kvm
kvm 317376 0
kvm-clock: Using msrs 4b564d01 and 4b564d00
kvm-clock: cpu 0, msr 0:1c25681, boot clock
Booting paravirtualized kernel on KVM
kvm-clock: cpu 0, msr 0:2216681, primary cpu clock
kvm-stealtime: cpu 0, msr 220e840
Switching to clocksource kvm-clock
kvm: no hardware support
[root@localhost ~]# virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization)
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS
|
on the KVM host ( Fedora 19 )
[root@localhost hattori]# virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : PASS
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS
|
shutdown the VM to change vCPU configuration.
on the VM
[root@localhost ~]# init 0
|
edit VM’s xml file
on the KVM host
# virsh edit “vm domain”
<pae/>
</features>
<cpu mode='host-passthrough'>
</cpu>
<clock offset='utc'/>
|
start the VM
on the KVM host
virsh # start centos6-vm1-ganeti01
|
on the VM
okay.
[root@localhost ~]# egrep -i '(svm|vmx)' /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm 3dnowext 3dnow up rep_good extd_apicid unfair_spinlock pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw npt nrip_save
[root@localhost ~]# virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : PASS
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS
[root@localhost ~]# dmesg | grep -i kvm
kvm-clock: Using msrs 4b564d01 and 4b564d00
kvm-clock: cpu 0, msr 0:1c25681, boot clock
Booting paravirtualized kernel on KVM
kvm-clock: cpu 0, msr 0:2216681, primary cpu clock
kvm-stealtime: cpu 0, msr 220e840
Switching to clocksource kvm-clock
kvm: Nested Virtualization enabled
kvm: Nested Paging enabled
[root@localhost ~]#
[root@localhost ~]# lsmod | grep kvm
kvm_amd 41702 0
kvm 317376 1 kvm_amd
|
run tiny linux image to confirm
login: root, password: password
on the VM, start the image.
[root@localhost ~]# /usr/libexec/qemu-kvm -hda ttylinux.img -m 128
|
error.
dmesg on the VM
handle_exit: unexpected exit_ini_info 0x80000008 exit_code 0x60
handle_exit: unexpected exit_ini_info 0x80000008 exit_code 0x60
handle_exit: unexpected exit_ini_info 0x80000008 exit_code 0x60
|
on the KVM host
kvm [7175]: vcpu0 unhandled rdmsr: 0xc0010001
|
Googling this error, this seems to be a bug..
I tried to solve this by changing disk type ( Virtio/IDE/SCSI with raw/qcow2 ), but I was not able to find solutions…
[ success case ]
I changed a VM to Ubuntu13.04 VMs from CentOS VMs, I was able to do nested VM without errors.
KVM host: Fedora19 ( AMD Phenom(tm) II X4 925 Processor )
VM: Ubuntu13.04 ( disk path IDE, storage format qcow2 )
When using virtio as disk path, ubuntu vm does not start up due to kernel panic..)
build a VM on Ubuntu 13.04 VM running within Fedora 19 KVM
on the Ubuntu 13.04 VM
root@ubuntu:~# vmbuilder kvm ubuntu
2013-08-21 01:19:02,794 INFO : Unmounting target filesystem
2013-08-21 01:19:06,103 INFO : Calling hook: convert
2013-08-21 01:19:06,103 INFO : Converting /tmp/tmpcYqKtA to qcow2, format ubuntu-kvm/tmpcYqKtA.qcow2
2013-08-21 01:19:25,803 INFO : Calling hook: fix_ownership
2013-08-21 01:19:25,812 INFO : Calling hook: deploy
2013-08-21 01:19:25,813 INFO : Calling hook: fix_ownership
root@ubuntu:~# echo $?
0
|
start that VM ( Ubuntu VM )
root@ubuntu:~# cd ubuntu-kvm/
root@ubuntu:~/ubuntu-kvm# ls
run.sh tmpcYqKtA.qcow2
root@ubuntu:~/ubuntu-kvm# cat run.sh
#!/bin/sh
exec kvm -m 128 -smp 1 -drive file=tmpcYqKtA.qcow2 "$@"
root@ubuntu:~/ubuntu-kvm# ./run.sh
|
Ubuntu 10.04 VM is running within Ubuntu 13.04 VM which is running Fedora 19 KVM.
on the KVM host ( Fedora 19 )
virsh # dumpxml 21
<domain type='kvm' id='21'>
<name>ubuntu1304-vm3</name>
<uuid>c4cefa40-6dcd-194e-5e87-8675024a5029</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-1.6'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>phenom</model>
<vendor>AMD</vendor>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/ubuntu1304-vm3.qcow2'/>
<target dev='hda' bus='ide'/>
<alias name='ide0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw' cache='none'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:a3:70:c7'/>
<source network='network1'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/2'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'/>
<graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='qxl' ram='65536' vram='65536' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c89,c482</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c89,c482</imagelabel>
</seclabel>
</domain>
|
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.