Xen 4.1 and Ubuntu 11.04

Xen 4.1 and Ubuntu 11.04 Natty Narwhal

Ubuntu is my current favorite Linux distribution and the newest release Natty Narwhal has some spiffy stuff, but not Xen 4.1, nor updated versions of libvirt or virt-manager. As Canonical seems to have preferred KVM (kernel virtual machines) over the Xen hypervisor, there isn't a convenient package to upgrade to Xen 4.1. I'm also being brave and installing the latest versions of libvirt (0.9.1) and virt-manager (0.8.7).

Why Xen?

I need hardware virtual machines (HVM) and I need a free hypervisor to run them on. Since I'm not forking out for VMware ESXi, Citrix XenServer, or Microsoft Hyper-V, and the lack of true HVM support in KVM, QEMU, VirtualBox, or <your favorite here> eliminate them as well. Since I'm also doing this for $work I can't use the free or community version either, with their not-for-commercial licenses.

Install Ubuntu

Ubuntu installation is straight forward here. I used Ubuntu 11.04 Natty Narwhal x86_64, downloaded the ISO and burned it to a CD. Boot, install, update. I tweaked my install slightly here, adding apt-fast, changing to a local Ubuntu mirror, and installing firmware and drivers for any special devices (a LSI RAID controller and QLogic Fibre Channel Controller). Installing the 3^rd party drivers is fairly important, especially any source or kernel patches. Since I intend on using virt-manager I also installed the Ubuntu GUI desktop.

$ apt-fast install ubuntu-desktop

Compile and Install Xen 4.1

This bit doesn't get too tricky until you get to grub2 or creating the required network bridges.

Install dependencies

Some of the dependencies need to be updated.

$ apt-fast install libcurl3-gnutls-dev xserver-xorg-dev mercurial gitk build-essential libncurses5-dev uuid-dev gawk gettext texinfo bcc dpkg-dev debhelper kernel-package fakeroot iasl bridge-utils bison flex ocaml-findlib libparted-dev gparted

$ apt-fast build-dep xen-3.3

Create and install a Xen kernel

Use git to clone and checkout the latest xen branch of the Linux kernel.

$ cd /usr/src

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen

$ cd linux-2.6-xen

$ git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x

Compile the kernel and install it the Debian way:

$ cd /usr/src/linux-2.6-xen

$ export CONCURRENCY_LEVEL=<cores + 1>

$ make menuconfig

$ make-kpkg clean

$ fakeroot make-kpkg --initrd --append-to-version=-xen kernel-image kernel-headers

$ dpkg -i ../linux-image*2.6.32*-xen*.deb

$ update-initramfs -c -k 2.6.32.*-xen

Note: The file globs will need to be expanded, I use the tab key.

Kernel menuconfig

Most of the important configuration settings for the Xen kernel have been already configured by the git checkout, but the following settings need to be checked.

<*> Xen /dev/xen/evtchn device

<*> userspace grant access device driver

Compile and install Xen 4.1

Another fairly straight forward get source, compile and install. Don't worry, it gets tricky later.

$ cd /usr/src

$ hg clone http://xenbits.xensource.com/xen-4.1-testing.hg

$ cd xen-4.2-testing

$ make xen

$ make tools

$ make stubdom

$ make install-xen

$ make install-tools PYTHON_PREFIX_ARG=

$ make install-stubdom

Set up Xen service daemons:

$ update-rc.d xencommons defaults 19 18

$ update-rc.d xend defaults 20 21

$ update-rc.d xendomains defaults 21 20

$ update-rc.d xen-watchdog defaults 22 23

Note: All you need to do to get libvirt to switch between Xen 3.0.3 API to the new Xenlight API is not have xend autostart at boot.

This far we should have the Xen hypervisor and Xen kernel in place, but not active, we'll leave them like this while libvirt and virt-manager are installed.

Compile and install libvirt, and other tools

Install dependencies:

$ apt-fast install libgtk-vnc-1.0-dev libgtk2.0-dev libglade2-dev libgnutls-dev libdevmapper-dev intltool libnlj-dev libparted-dev liblum2-dev libute-dev python-gconf python-dbus python-lxml python-urlgrabber python-gtk-vnc

Download the source tarballs for libvirt, virt-manager and unpackaged dependencies.

mkdir ~/download

$ cd ~/download

$ wget http://libvirt.org/sources/libvirt-0.9.1.tar.gz

$ wget http://virt-manager.et.redhat.com/download/sources/virt-manager/virt-manager-0.8.7.tar.gz

$ wget http://virt-manager.et.redhat.com/download/sources/virtinst/virtinst-0.500.6.tar.gz

$ wget http://virt-manager.et.redhat.com/download/sources/virt-viewer/virt-viewer-0.2.0.tar.gz

Uncompress them somewhere:

$ cd /usr/src

$ tar xzvf ~/download/libvirt-0.9.1.tar.gz

$ tar xzvf ~/download/virt-manager-0.8.7.tar.gz

$ tar xzvf ~/download/virtinst-0.500.6.tar.gz

$ tar xzvf ~/download/virt-viewer-0.2.0.tar.gz

Compile and install them as per the instructions but with ./configure --prefix=/usr where appropriate. Install them in the following order

1. libvirt


2. virtinst


3. virt-viewer


4. virt-manager

All straight forward so far, the tricky bit is getting libvirtd to start up correctly. After much wailing and gnashing at Google, it turns out there's a perfectly acceptable upstart script that comes with libvirt, so copy that where it will be useful and work with it from there.

$ cp /usr/src/libvirt-0.9.1/daemon/libvirtd.upstart /etc/init/libvirtd.conf

$ initctl reload-configuration

$ initctl start libvirtd

Now to make libvirtd start at the appropriate time, it needs to watch out for the Xen Ethernet bridges to come up, so add the following startup condition to /etc/init/libvirtd.conf:

startup on (runlevel [2345] and net-device-up IFACE=peth0)

Note: This waits for the first interface only, you might want to add additional and net-device-up IFACE=pethX clauses if you want to wait for them too.

Activate Xen

There are two components to activating Xen, the first is setting up your network bridging, and the second is beating grub2 until it complies with your demands.

Network bridging

Setting up your network bridges for Xen can get confusing, there is much advice. I have found that it's best to let Xen do it for you, but it needs help. I'm going to do an example for a setting up two network interfaces, each on separate networks. I've used this method to do four interfaces.

Static interfaces

Bridging interfaces does not get along well with DHCP, so make all your interfaces static. So your /etc/network/interfaces should resemble this:

auto eth0

iface eth0 inet static

  address    192.168.1.1

  netmask    255.255.255.0

  broadcast  192.168.1.255

  gateway    192.168.1.254



auto eth1

iface eth1 inet static

  address    10.1.1.1

  netmask    255.255.255.0

  broadcast  10.1.1.255

# gateway    10.1.1.254

Note: The gateway on the second interface is commented out, as in this instance traffic is expected to go out via eth0 by default. If traffic is expected to go to different interfaces for different sub-nets, you need to work with iptables to do your routing.
Note: Do not worry about any special routing for the guest VMs, they can do that themselves. This is just the networking for the domU host.

Customising Xen networking

There are two parts to this step, creating a bridging script, and modifying the Xen configuration.

Create the script file /etc/xen/scripts/network-multi-bridge with the content:

#!/bin/sh

dir=$(dirname "$0")



# Add a new bridge for each interface, where vifnum is set to the interface number

"$dir/network-bridge" "$@" vifnum=0

"$dir/network-bridge" "$@" vifnum=1

Make the script executable by root:

$ chmod 755 /etc/xen/scripts/network-multi-bridge

Edit /etc/xen/xend-config.sxp and comment out (network-script network-bridge) and add your multi-bridge script, so the lines look like this:

#(network-script network-bridge)

(network-script network-multi-bridge)

That should do it.

Note: I not quite satisfied that this is working right yet...

Making grub2 boot with Xen by default

Mostly I like grub2, but I really dislike what update-grub2 does, in particular, how it likes to clobber your carefully crafted /boot/grub/grub.cfg file. Since this happens every time apt-fast dist-upgrade upgrades a kernel, it's better to make changes that will be preserved when update-grub2 is run.

The good news is update-grub2 will actually generate an appropriate menu entry for us, we just have to find it and make it the default entry.

First update your grub configuration:

$ update-grub2

This will create a whole bunch of menu entries in /boot/grub/grub.cfg the one you want is actually in there, it should look something like:

menuentry 'Ubuntu GNU/Linux, with Xen 4.1 and Linux 2.6.32.39-xen' --class ubuntu --class

gnu-linux --class gnu --class os --class xen {

        insmod part_msdos

        insmod ext2

        set root='(/dev/sda,msdos1)'

        search --no-floppy --fs-uuid --set=root <a UUID>

        echo    'Loading Xen 4.1 ...'

        multiboot       /boot/xen-4.1.gz placeholder

        echo    'Loading Linux 2.6.32.39-xen ...'

        module  /boot/vmlinuz-2.6.32.39-xen placeholder root=UUID=<a UUID> ro  splash quiet

        echo    'Loading initial ramdisk ...'

        module  /boot/initrd.img-2.6.32.39-xen

}

Restart your server and checks that this entry actually works, it might be hidden in a sub-menu.

Once you're sure you've found the entry you want, copy the whole section and paste it in at the end of /etc/grub.d/40_custom and change the entry name to something like "Xen 4.1 on Ubuntu 11.04 kernel 2.6.39-xen", this has to not match any other entry in /boot/grub/grub.cfg.

Now edit /etc/default/grub and change the default boot entry to:

GRUB_DEFAULT='Xen 4.1 on Ubuntu 11.04 kernel 2.6.39-xen'

Now rerun update-grub2 and restart, it should now boot to your carefully crafted Xen setup.

References

• Xen Virtualisation on Linux and Solaris

Resizing swap partition with LVM from the command line

I've been tweaking swap partitions a lot recently as a template virtual machine I created only reserved 512MB of space for a swap partition, where this should really have been at least 2GB for efficient swap usage.

It would seem that mostly other Linux users just boot to a LiveCD and use gparted, but using a GUI interface on a Virtual Machine's remote console is like playing pin-the-tail-on-the-donkey, with a real donkey, who's rather pissed off about you having his tail and all. So, doing it from a command line is a requisite here.

I've used this method on CentOS 5.5, and it should also work with other distributions, you could probably just use the CentOS DVD as well, but I suspect the rescue mode on any distribution's install disc will do the job.

The following method repartitions a 40GB drive (a virtual drive in this case, should work just fine on real ones) from 39.5GB ext3 file system and 512MB swap to 35GB ext3 file system and 2.5GB swap. Note that there is some unallocated space, that's because I'm being paranoid and I do not want the partitions to cross streams and rip a breach in the space time fabric and all my bits to leak out. If you want to tweak the numbers and risk your file system's integrity, you do that, but remember I didn't, I did something safer.

Just to keep things exciting, these drives use LVM (Logical Volume Management), which is awkward, until you get used to it, but gives you a lot more flexibility when it comes to mangling your file systems into something completely unmanageable.

Before you start, clearly identify your partitions, in this case /dev/VolGroup00/Logvol00 is my ext3 partion with the server's file system, and /dev/VolGroup00/LogVol01 is the swap space. Decide how much you want to shrink one, and grow the other. Be sure you're using ext2, ext3, or ext4. Make sure you have 20min to an hour to sit in front of your workstation while the file system is resized, and if you do find that you've been interrupted and CentOS has rebooted, you should be able to restart, repeat the lvm vgchange -a y command after booting to rescue mode and continue from where you left off.

• Boot from the CentOS 5.5 installation DVD 1, instead of selecting install type in linux rescue and press enter.
• When prompted, don't load your network drivers, you won't need them.
• Skip mounting your local drives, we want them for ourselves.
• Once you've got to a command prompt you're ready to go:

$ lvm vgchange -a y
$ e2fschk -f /dev/VolGroup00/LogVol00
$ resize2fs /dev/VolGroup00/LogVol00 35G
$ lvm lvreduce /dev/VolGroup00/LogVol00 -L 36G
$ lvm lvresize /dev/VolGroup00/LogVol01 -L +2G
$ mkswap /dev/VolGroup00/LogVol01

• Reboot, and all should be well and you should now have 2GB more swap space.

Can haz syntax!

According to the instructions here, I should now have syntax highlighting.

It looks like the Bash/Shell brush isn't included (so I added it):

$ sudo rm -rf /

Some of my horriblest JavaScript from a 5k challenge

var r=true;function st(sz){sz--;
var sc=[[0,0],[1,0],[1,2],[3,1],[4,0],[5,0],[6,0]];//edit this for a new pattern
sc=cp(sc,sz);document.getElementById('tt').innerHTML=mt(sc,sz,sz,sz,sz);rl(sc,(sz),0);}
function h(){if(r){r=false;document.getElementById('tb').disabled=true;document.getElementById('tb').value="Reload page to resart";}}
function cp(p,sz){sz=sz/2;var np=[];var lx=0;var bx=0;var ly=0;var by=0;for(i=0;i<p.length;i++){var px=p[i][0];var py=p[i][1];lx=Math.min(lx,px);bx=Math.max(bx,px);ly=Math.min(ly,py);by=Math.max(by,py);}var dx=Math.round(sz-(bx-lx+1)/2);var dy=Math.round(sz-(by-ly+1)/2);for(i=0;i<p.length;i++){np.push([p[i][0]+dx,p[i][1]+dy]);}return np;}
function rl(cs,sz,it){document.getElementById('it').innerHTML="Iteration:"+it+" Population:"+cs.length;it++;t=setTimeout("this.rc()",0);this.rc=function(){if(r){rl(uc(cs,sz),sz,it);}}}
function mt(cs,sz,x,y){var c=cc(cs,sz,x,y);var sr='<tr>';var er='</tr>';if(x==sz&&y==sz){return "<table>"+sr+c+mt(cs,sz,x-1,y);}else{if(x==sz){return sr+c+mt(cs,sz,x-1,y);}else if(x>0){return c+mt(cs,sz,x-1,y);}else{if(y > 0){return c+er+mt(cs,sz,sz,y-1);}else{return c+er+"</table>";}}}}
function cc(cs,sz,x,y){var sd='<td id="X'+x+'Y'+y+'" class="';var ed='"></td>';if(ii(x,y,cs)){return sd+'l'+ed;}else{return sd+'d'+ed;}}
function ii(x,y,cs){var r=false;if(cs.length >0){for(k=0;k<cs.length;k++){if(cs[k][0]==x&&cs[k][1]==y){r=true;}}}return r;}
function cl(ls,cs){var n=0;while(ls.length >0){var co=ls.pop();if(ii(co[0],co[1],cs)){n++;}}return n;}
function nb(x,y,sz){var ls=[];for(i=-1;i<2;i++){for(j=-1;j<2;j++){var nx=ib(x+i,sz);var ny=ib(y+j,sz);if((x!=nx)||(y!=ny)){ls.push([nx,ny]);}}}return ls;}
function ib(r,sz){if(r<0){r=sz;}if(r>sz){r=0;}return r;}
function uc(cs,sz){var nc=[];var ac=[];for(l=0;l<cs.length;l++){var ls=nb(cs[l][0],cs[l][1],sz);ls.push(cs[l]);while(ls.length >0){var co=ls.pop();if(!ii(co[0],co[1],ac)){ac.push(co);}}}while(ac.length >0){var co=ac.pop(0);var m=cl(nb(co[0],co[1],sz),cs);var c='d';if(m==2&&ii(co[0],co[1],cs)){nc.push(co);c='l';}else if(m==3){nc.push(co);c='l';}document.getElementById('X'+co[0]+'Y'+co[1]).className=c;}return nc;}

Relocated

I have blogs, more than one. More and more there is increasing hate for LiveJournal and the awesome invasiveness of Facebook. More and more, I like blogger, though I think I'm going to have to figure out how to do without, or replicate, the fancy formatting tools I use for mediawiki or Confluence.