KVM : hot-add PCI devices ( NIC )

[ KVM host info ]

# tail -1 /etc/lsb-release DISTRIB_DESCRIPTION="Ubuntu 11.04" # uname -r 2.6.38-11-server # kvm --version QEMU emulator version 0.14.0 (qemu-kvm-0.14.0), Copyright (c) 2003-2008 Fabrice Bellard

[ Requirements ]

in my testing environment :

KVM host needs kvm-pxe. ( you can install it via apt-get )
VM : Scientfic Linux 6.1 needs load
VM : Ubuntu 10.04 needs to load acpiphp module.

Here are trial and error logs I did.

[ Preparation on KVM host ]

I need to install kvm-pxe on KVM host.
When I haven’t installed kvm-pxe on KVM host , I see the following error when adding a NIC and I can’t add a NIC to a VM.

virsh # qemu-monitor-command --hmp ubuntu-vm1 'pci_add auto nic model=virtio,macaddr=11:22:33:44:55:66' pci_add_option_rom: failed to find romfile "pxe-virtio.bin" OK domain 0, bus 0, slot 5, function 0

install kvm-pxe on KVM host.

# apt-get install kvm-pxe

[ add/delete PCI devices ( NIC ) to a guest VM running Scientific linux 6.1 ]

Seen from http://www.linux-kvm.org/page/Hotadd_pci_devices , this describes that a VM needs to load acpiphp and pci_hotplug modules , however I could add PCI devices without loading the above modules on a VM running SL 6.1.

Let’s add a NIC to SL6.1.
Here’s an output of “lsmod” on SL6.1.

# lsmod Module                  Size  Used by autofs4                27683  3 sunrpc                242213  1 ipt_REJECT              2349  2 nf_conntrack_ipv4       9440  2 nf_defrag_ipv4          1449  1 nf_conntrack_ipv4 iptable_filter          2759  1 ip_tables              17765  1 iptable_filter ip6t_REJECT             4562  2 nf_conntrack_ipv6       8650  2 nf_defrag_ipv6         12148  1 nf_conntrack_ipv6 xt_state                1458  4 nf_conntrack           79611  3 nf_conntrack_ipv4,nf_conntrack_ipv6,xt_state ip6table_filter         2855  1 ip6_tables             19424  1 ip6table_filter ipv6                  322291  27 ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6 dm_mirror              14067  0 dm_region_hash         12136  1 dm_mirror dm_log                 10120  2 dm_mirror,dm_region_hash uinput                  8182  0 sg                     30090  0 microcode             112781  0 virtio_balloon          4281  0 virtio_net             15741  0 i2c_piix4              12574  0 i2c_core               31274  1 i2c_piix4 ext4                  359671  1 mbcache                 7918  1 ext4 jbd2                   88768  1 ext4 sr_mod                 16194  0 cdrom                  39769  1 sr_mod virtio_blk              5692  3 ata_generic             3611  0 pata_acpi               3667  0 ata_piix               22652  0 virtio_pci              6653  0 virtio_ring             7169  4 virtio_balloon,virtio_net,virtio_blk,virtio_pci virtio                  4824  4 virtio_balloon,virtio_net,virtio_blk,virtio_pci dm_mod                 75539  2 dm_mirror,dm_log

before adding a NIC.
SL6.1 has one NIC. ( qemu monitor )

virsh # qemu-monitor-command --hmp SL6-vm1 'info network' Devices not on any VLAN:  hostnet0: fd=18 peer=net0  net0: model=virtio-net-pci,macaddr=52:54:00:11:18:f1 peer=hostnet0

add a NIC ( qemu monitor )

virsh # qemu-monitor-command --hmp SL6-vm1 'pci_add auto nic model=virtio' OK domain 0, bus 0, slot 4, function 0

qemu monitor

virsh # qemu-monitor-command --hmp SL6-vm1 'info network' VLAN 0 devices:  virtio-net-pci.0: model=virtio-net-pci,macaddr=52:54:00:12:34:56 Devices not on any VLAN:  hostnet0: fd=18 peer=net0  net0: model=virtio-net-pci,macaddr=52:54:00:11:18:f1 peer=hostnet0

on VM

# ifconfig -a | grep eth | awk '{print $1}' eth0 eth1  <- added

dmesg on VM

virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11  alloc irq_desc for 29 on node -1  alloc kstat_irqs on node -1 virtio-pci 0000:00:04.0: irq 29 for MSI/MSI-X  alloc irq_desc for 30 on node -1  alloc kstat_irqs on node -1 virtio-pci 0000:00:04.0: irq 30 for MSI/MSI-X  alloc irq_desc for 31 on node -1  alloc kstat_irqs on node -1 virtio-pci 0000:00:04.0: irq 31 for MSI/MSI-X

delete a NIC eth1 ( qemu monitor )

virsh # qemu-monitor-command --hmp SL6-vm1 'pci_del pci_addr=4 virsh # qemu-monitor-command --hmp SL6-vm1 'info network' VLAN 0 devices: Devices not on any VLAN:  hostnet0: fd=18 peer=net0  net0: model=virtio-net-pci,macaddr=52:54:00:11:18:f1 peer=hostnet0

on VM

# ifconfig -a | grep eth | awk '{print $1}' eth0

[ add/delete PCI devices ( NIC ) to a guest VM running Ubuntu 10.04 ]

I built a Ubuntu 10.04 VM with vmbuilder.

An output of “lsmod” on VM.

# lsmod Module                  Size  Used by fbcon                  35102  71 tileblit                1999  1 fbcon font                    7557  1 fbcon bitblit                 4707  1 fbcon serio_raw               3978  0 softcursor              1189  1 bitblit psmouse                63245  0 vga16fb                11385  1 virtio_balloon          2835  0 vgastate                8961  1 vga16fb floppy                 53080  0

VM has one NIC. ( qemu-monitor )

virsh # qemu-monitor-command --hmp ubuntu-vm1 'info network' Devices not on any VLAN:  hostnet0: fd=18 peer=net0  net0: model=virtio-net-pci,macaddr=52:54:00:0a:7e:0a peer=hostnet0

add a NIC ( qemu-monitor )

virsh # qemu-monitor-command --hmp ubuntu-vm1 'pci_add auto nic model=virtio,mac addr=11:22:33:44:55:66' OK domain 0, bus 0, slot 5, function 0 virsh # qemu-monitor-command --hmp ubuntu-vm1 'info network' VLAN 0 devices:  virtio-net-pci.0: model=virtio-net-pci,macaddr=11:22:33:44:55:66 Devices not on any VLAN:  hostnet0: fd=18 peer=net0  net0: model=virtio-net-pci,macaddr=52:54:00:0a:7e:0a peer=hostnet0

nnn , there isn’t an additional NIC on Ubuntu VM.

# ifconfig –a | grep eth | awk '{print $1}' eth0

load acpiphp and pci_hotplug modules and try it again.
It seems that there’s no pci_hotplug module in Ubuntu.
so load only acpipho module.

# modprobe acpiphp # modprobe pci_hotplug FATAL: Module pci_hotplug not found. # lsmod Module                  Size  Used by acpiphp                18316  0 fbcon                  35102  71 tileblit                1999  1 fbcon font                    7557  1 fbcon bitblit                 4707  1 fbcon softcursor              1189  1 bitblit serio_raw               3978  0 vga16fb                11385  1 psmouse                63245  0 vgastate                8961  1 vga16fb virtio_balloon          2835  0 floppy                 53080  0

try again.

virsh # qemu-monitor-command --hmp ubuntu-vm1 'pci_add auto nic model=virtio,macaddr=11:22:33:44:55:77' OK domain 0, bus 0, slot 6, function 0

NICs has been added.

# ifconfig -a | grep eth | awk '{print $1}' eth0 eth1 eth3

delete NICs ( eth1 , eth3 )

virsh # qemu-monitor-command --hmp ubuntu-vm1 'pci_del pci_addr=5' virsh # qemu-monitor-command --hmp ubuntu-vm1 'pci_del pci_addr=6'

NICs have been removed.

# ifconfig -a | grep eth | awk '{print $1}' eth0