Xen 4.1 and Ubuntu 11.04 Natty Narwhal
Ubuntu is my current favorite Linux distribution and the newest release Natty Narwhal has some spiffy stuff, but not Xen 4.1, nor updated versions of libvirt or virt-manager. As Canonical seems to have preferred KVM (kernel virtual machines) over the Xen hypervisor, there isn't a convenient package to upgrade to Xen 4.1. I'm also being brave and installing the latest versions of libvirt (0.9.1) and virt-manager (0.8.7).
Why Xen?
I need hardware virtual machines (HVM) and I need a free hypervisor to run them on. Since I'm not forking out for VMware ESXi, Citrix XenServer, or Microsoft Hyper-V, and the lack of true HVM support in KVM, QEMU, VirtualBox, or <your favorite here> eliminate them as well. Since I'm also doing this for $work I can't use the free or community version either, with their not-for-commercial licenses.
Install Ubuntu
Ubuntu installation is straight forward here. I used Ubuntu 11.04 Natty Narwhal x86_64, downloaded the ISO and burned it to a CD. Boot, install, update. I tweaked my install slightly here, adding apt-fast, changing to a local Ubuntu mirror, and installing firmware and drivers for any special devices (a LSI RAID controller and QLogic Fibre Channel Controller). Installing the 3rd party drivers is fairly important, especially any source or kernel patches. Since I intend on using virt-manager I also installed the Ubuntu GUI desktop.
$ apt-fast install ubuntu-desktop
Compile and Install Xen 4.1
This bit doesn't get too tricky until you get to grub2 or creating the required network bridges.
Install dependencies
Some of the dependencies need to be updated.
$ apt-fast install libcurl3-gnutls-dev xserver-xorg-dev mercurial gitk build-essential libncurses5-dev uuid-dev gawk gettext texinfo bcc dpkg-dev debhelper kernel-package fakeroot iasl bridge-utils bison flex ocaml-findlib libparted-dev gparted
$ apt-fast build-dep xen-3.3
Create and install a Xen kernel
Use git to clone and checkout the latest xen branch of the Linux kernel.
$ cd /usr/src
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
$ cd linux-2.6-xen
$ git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
Compile the kernel and install it the Debian way:
$ cd /usr/src/linux-2.6-xen
$ export CONCURRENCY_LEVEL=<cores + 1>
$ make menuconfig
$ make-kpkg clean
$ fakeroot make-kpkg --initrd --append-to-version=-xen kernel-image kernel-headers
$ dpkg -i ../linux-image*2.6.32*-xen*.deb
$ update-initramfs -c -k 2.6.32.*-xen
Note: The file globs will need to be expanded, I use the tab key.
Kernel menuconfig
Most of the important configuration settings for the Xen kernel have been already configured by the git checkout, but the following settings need to be checked.
<*> Xen /dev/xen/evtchn device
<*> userspace grant access device driver
Compile and install Xen 4.1
Another fairly straight forward get source, compile and install. Don't worry, it gets tricky later.
$ cd /usr/src
$ hg clone http://xenbits.xensource.com/xen-4.1-testing.hg
$ cd xen-4.2-testing
$ make xen
$ make tools
$ make stubdom
$ make install-xen
$ make install-tools PYTHON_PREFIX_ARG=
$ make install-stubdom
Set up Xen service daemons:
$ update-rc.d xencommons defaults 19 18
$ update-rc.d xend defaults 20 21
$ update-rc.d xendomains defaults 21 20
$ update-rc.d xen-watchdog defaults 22 23
Note: All you need to do to get libvirt to switch between Xen 3.0.3 API to the new Xenlight API is not have xend autostart at boot.
This far we should have the Xen hypervisor and Xen kernel in place, but not active, we'll leave them like this while libvirt and virt-manager are installed.
Compile and install libvirt, and other tools
Install dependencies:
$ apt-fast install libgtk-vnc-1.0-dev libgtk2.0-dev libglade2-dev libgnutls-dev libdevmapper-dev intltool libnlj-dev libparted-dev liblum2-dev libute-dev python-gconf python-dbus python-lxml python-urlgrabber python-gtk-vnc
Download the source tarballs for libvirt, virt-manager and unpackaged dependencies.
mkdir ~/download
$ cd ~/download
$ wget http://libvirt.org/sources/libvirt-0.9.1.tar.gz
$ wget http://virt-manager.et.redhat.com/download/sources/virt-manager/virt-manager-0.8.7.tar.gz
$ wget http://virt-manager.et.redhat.com/download/sources/virtinst/virtinst-0.500.6.tar.gz
$ wget http://virt-manager.et.redhat.com/download/sources/virt-viewer/virt-viewer-0.2.0.tar.gz
Uncompress them somewhere:
$ cd /usr/src
$ tar xzvf ~/download/libvirt-0.9.1.tar.gz
$ tar xzvf ~/download/virt-manager-0.8.7.tar.gz
$ tar xzvf ~/download/virtinst-0.500.6.tar.gz
$ tar xzvf ~/download/virt-viewer-0.2.0.tar.gz
Compile and install them as per the instructions but with ./configure --prefix=/usr where appropriate. Install them in the following order
- libvirt
- virtinst
- virt-viewer
- virt-manager
All straight forward so far, the tricky bit is getting libvirtd to start up correctly. After much wailing and gnashing at Google, it turns out there's a perfectly acceptable upstart script that comes with libvirt, so copy that where it will be useful and work with it from there.
$ cp /usr/src/libvirt-0.9.1/daemon/libvirtd.upstart /etc/init/libvirtd.conf
$ initctl reload-configuration
$ initctl start libvirtd
Now to make libvirtd start at the appropriate time, it needs to watch out for the Xen Ethernet bridges to come up, so add the following startup condition to /etc/init/libvirtd.conf:
startup on (runlevel [2345] and net-device-up IFACE=peth0)
Note: This waits for the first interface only, you might want to add additional and net-device-up IFACE=pethX clauses if you want to wait for them too.
Activate Xen
There are two components to activating Xen, the first is setting up your network bridging, and the second is beating grub2 until it complies with your demands.
Network bridging
Setting up your network bridges for Xen can get confusing, there is much advice. I have found that it's best to let Xen do it for you, but it needs help. I'm going to do an example for a setting up two network interfaces, each on separate networks. I've used this method to do four interfaces.
Static interfaces
Bridging interfaces does not get along well with DHCP, so make all your interfaces static. So your /etc/network/interfaces should resemble this:
auto eth0
iface eth0 inet static
address 192.168.1.1
netmask 255.255.255.0
broadcast 192.168.1.255
gateway 192.168.1.254
auto eth1
iface eth1 inet static
address 10.1.1.1
netmask 255.255.255.0
broadcast 10.1.1.255
# gateway 10.1.1.254
Note: The gateway on the second interface is commented out, as in this instance traffic is expected to go out via eth0 by default. If traffic is expected to go to different interfaces for different sub-nets, you need to work with iptables to do your routing.
Note: Do not worry about any special routing for the guest VMs, they can do that themselves. This is just the networking for the domU host.
Customising Xen networking
There are two parts to this step, creating a bridging script, and modifying the Xen configuration.
Create the script file /etc/xen/scripts/network-multi-bridge with the content:
#!/bin/sh
dir=$(dirname "$0")
# Add a new bridge for each interface, where vifnum is set to the interface number
"$dir/network-bridge" "$@" vifnum=0
"$dir/network-bridge" "$@" vifnum=1
Make the script executable by root:
$ chmod 755 /etc/xen/scripts/network-multi-bridge
Edit /etc/xen/xend-config.sxp and comment out (network-script network-bridge) and add your multi-bridge script, so the lines look like this:
#(network-script network-bridge)
(network-script network-multi-bridge)
That should do it.
Note: I not quite satisfied that this is working right yet...
Making grub2 boot with Xen by default
Mostly I like grub2, but I really dislike what update-grub2 does, in particular, how it likes to clobber your carefully crafted /boot/grub/grub.cfg file. Since this happens every time apt-fast dist-upgrade upgrades a kernel, it's better to make changes that will be preserved when update-grub2 is run.
The good news is update-grub2 will actually generate an appropriate menu entry for us, we just have to find it and make it the default entry.
First update your grub configuration:
$ update-grub2
This will create a whole bunch of menu entries in /boot/grub/grub.cfg the one you want is actually in there, it should look something like:
menuentry 'Ubuntu GNU/Linux, with Xen 4.1 and Linux 2.6.32.39-xen' --class ubuntu --class
gnu-linux --class gnu --class os --class xen {
insmod part_msdos
insmod ext2
set root='(/dev/sda,msdos1)'
search --no-floppy --fs-uuid --set=root <a UUID>
echo 'Loading Xen 4.1 ...'
multiboot /boot/xen-4.1.gz placeholder
echo 'Loading Linux 2.6.32.39-xen ...'
module /boot/vmlinuz-2.6.32.39-xen placeholder root=UUID=<a UUID> ro splash quiet
echo 'Loading initial ramdisk ...'
module /boot/initrd.img-2.6.32.39-xen
}
Restart your server and checks that this entry actually works, it might be hidden in a sub-menu.
Once you're sure you've found the entry you want, copy the whole section and paste it in at the end of /etc/grub.d/40_custom and change the entry name to something like "Xen 4.1 on Ubuntu 11.04 kernel 2.6.39-xen", this has to not match any other entry in /boot/grub/grub.cfg.
Now edit /etc/default/grub and change the default boot entry to:
GRUB_DEFAULT='Xen 4.1 on Ubuntu 11.04 kernel 2.6.39-xen'
Now rerun update-grub2 and restart, it should now boot to your carefully crafted Xen setup.
