stoney cloud: Nested virtualization

Abstract

This document describes how to set up nested virtualization on a default installation (until it gets enabled by default).

Nested virtualization permits to start another hypervisor (in our case KVM) inside a virtual machine.

With the current version of Qemu/KVM you can expect a performance of about 25% of the host performance (concerning CPU and Memory access) in a guest within a guest (L2). In the future Qemu/KVM should support Intels Virtual EPT which should give 80% of the host performance for L2.

See:

• http://events.linuxfoundation.org/sites/events/files/cojp13_nakajima.pdf
• https://docs.google.com/file/d/0BzyAwvVlQckedmpobUY1Sm0zNWc/edit

For Intel CPUs we are going to enable Shadow VMCS which requires a sufficiently new CPU, otherwise it gets ignored. This should give some performance increase, disable it if it causes problems.

Setup

Intel

Login as root, then execute the following:

cat > /etc/modprobe.d/nested-virtualization.conf << EOF
options kvm-intel nested=Y enable_shadow_vmcs=Y
EOF

You either have to reboot or rmmod/modprobe the corresponding module if no VM is running. For example:

rmmod kvm_intel
modprobe kvm_intel

AMD

Login as root, then execute the following:

cat > /etc/modprobe.d/nested-virtualization.conf << EOF
options kvm-amd nested=Y
EOF

You either have to reboot or rmmod/modprobe the corresponding module if no VM is running. For example:

rmmod kvm_amd
modprobe kvm_amd

VM configuration

The following is largely untested, therefore this is more an instruction on how to figure out what the correct setting is. (hint: remove this line and update the configuration below if you have figured it out)

First try to start any VM and execute the following inside a Linux guest to see whether virtualization is available:

grep vmx /proc/cpuinfo

If there is support, the output should be something like this:

flags		: [...] vmx [...]

If it does not get enabled automatically by libvirt as soon as the hypervisor supports nested virtualization (it should, though), you have to change to XML description of the VM. To that end, use virsh edit on the node after having it defined it and replace the lines:

  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>

with

  <cpu mode='host-model'>
    <model fallback='allow'/>
    <feature policy='require' name='vmx'/>
  </cpu>