Wednesday, May 25, 2011

Xen 4.1 and Ubuntu 11.04

Xen 4.1 and Ubuntu 11.04 Natty Narwhal


Ubuntu is my current favorite Linux distribution and the newest release Natty Narwhal has some spiffy stuff, but not Xen 4.1, nor updated versions of libvirt or virt-manager. As Canonical seems to have preferred KVM (kernel virtual machines) over the Xen hypervisor, there isn't a convenient package to upgrade to Xen 4.1. I'm also being brave and installing the latest versions of libvirt (0.9.1) and virt-manager (0.8.7).

Why Xen?


I need hardware virtual machines (HVM) and I need a free hypervisor to run them on. Since I'm not forking out for VMware ESXi, Citrix XenServer, or Microsoft Hyper-V, and the lack of true HVM support in KVM, QEMU, VirtualBox, or <your favorite here> eliminate them as well. Since I'm also doing this for $work I can't use the free or community version either, with their not-for-commercial licenses.

Install Ubuntu


Ubuntu installation is straight forward here. I used Ubuntu 11.04 Natty Narwhal x86_64, downloaded the ISO and burned it to a CD. Boot, install, update. I tweaked my install slightly here, adding apt-fast, changing to a local Ubuntu mirror, and installing firmware and drivers for any special devices (a LSI RAID controller and QLogic Fibre Channel Controller). Installing the 3rd party drivers is fairly important, especially any source or kernel patches. Since I intend on using virt-manager I also installed the Ubuntu GUI desktop.
$ apt-fast install ubuntu-desktop

Compile and Install Xen 4.1


This bit doesn't get too tricky until you get to grub2 or creating the required network bridges.

Install dependencies


Some of the dependencies need to be updated.
$ apt-fast install libcurl3-gnutls-dev xserver-xorg-dev mercurial gitk build-essential libncurses5-dev uuid-dev gawk gettext texinfo bcc dpkg-dev debhelper kernel-package fakeroot iasl bridge-utils bison flex ocaml-findlib libparted-dev gparted

$ apt-fast build-dep xen-3.3


Create and install a Xen kernel


Use git to clone and checkout the latest xen branch of the Linux kernel.

$ cd /usr/src
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
$ cd linux-2.6-xen
$ git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x


Compile the kernel and install it the Debian way:

$ cd /usr/src/linux-2.6-xen
$ export CONCURRENCY_LEVEL=<cores + 1>
$ make menuconfig
$ make-kpkg clean
$ fakeroot make-kpkg --initrd --append-to-version=-xen kernel-image kernel-headers
$ dpkg -i ../linux-image*2.6.32*-xen*.deb
$ update-initramfs -c -k 2.6.32.*-xen

Note: The file globs will need to be expanded, I use the tab key.

Kernel menuconfig


Most of the important configuration settings for the Xen kernel have been already configured by the git checkout, but the following settings need to be checked.

<*> Xen /dev/xen/evtchn device
<*> userspace grant access device driver


Compile and install Xen 4.1


Another fairly straight forward get source, compile and install. Don't worry, it gets tricky later.
$ cd /usr/src

$ hg clone http://xenbits.xensource.com/xen-4.1-testing.hg
$ cd xen-4.2-testing
$ make xen
$ make tools
$ make stubdom
$ make install-xen
$ make install-tools PYTHON_PREFIX_ARG=
$ make install-stubdom


Set up Xen service daemons:

$ update-rc.d xencommons defaults 19 18
$ update-rc.d xend defaults 20 21
$ update-rc.d xendomains defaults 21 20
$ update-rc.d xen-watchdog defaults 22 23

Note: All you need to do to get libvirt to switch between Xen 3.0.3 API to the new Xenlight API is not have xend autostart at boot.

This far we should have the Xen hypervisor and Xen kernel in place, but not active, we'll leave them like this while libvirt and virt-manager are installed.

Compile and install libvirt, and other tools



Install dependencies:
$ apt-fast install libgtk-vnc-1.0-dev libgtk2.0-dev libglade2-dev libgnutls-dev libdevmapper-dev intltool libnlj-dev libparted-dev liblum2-dev libute-dev python-gconf python-dbus python-lxml python-urlgrabber python-gtk-vnc


Download the source tarballs for libvirt, virt-manager and unpackaged dependencies.
mkdir ~/download

$ cd ~/download
$ wget http://libvirt.org/sources/libvirt-0.9.1.tar.gz
$ wget http://virt-manager.et.redhat.com/download/sources/virt-manager/virt-manager-0.8.7.tar.gz
$ wget http://virt-manager.et.redhat.com/download/sources/virtinst/virtinst-0.500.6.tar.gz
$ wget http://virt-manager.et.redhat.com/download/sources/virt-viewer/virt-viewer-0.2.0.tar.gz


Uncompress them somewhere:
$ cd /usr/src

$ tar xzvf ~/download/libvirt-0.9.1.tar.gz
$ tar xzvf ~/download/virt-manager-0.8.7.tar.gz
$ tar xzvf ~/download/virtinst-0.500.6.tar.gz
$ tar xzvf ~/download/virt-viewer-0.2.0.tar.gz


Compile and install them as per the instructions but with ./configure --prefix=/usr where appropriate. Install them in the following order
  1. libvirt

  2. virtinst

  3. virt-viewer

  4. virt-manager


All straight forward so far, the tricky bit is getting libvirtd to start up correctly. After much wailing and gnashing at Google, it turns out there's a perfectly acceptable upstart script that comes with libvirt, so copy that where it will be useful and work with it from there.
$ cp /usr/src/libvirt-0.9.1/daemon/libvirtd.upstart /etc/init/libvirtd.conf

$ initctl reload-configuration
$ initctl start libvirtd


Now to make libvirtd start at the appropriate time, it needs to watch out for the Xen Ethernet bridges to come up, so add the following startup condition to /etc/init/libvirtd.conf:
startup on (runlevel [2345] and net-device-up IFACE=peth0)

Note: This waits for the first interface only, you might want to add additional and net-device-up IFACE=pethX clauses if you want to wait for them too.


Activate Xen


There are two components to activating Xen, the first is setting up your network bridging, and the second is beating grub2 until it complies with your demands.

Network bridging


Setting up your network bridges for Xen can get confusing, there is much advice. I have found that it's best to let Xen do it for you, but it needs help. I'm going to do an example for a setting up two network interfaces, each on separate networks. I've used this method to do four interfaces.

Static interfaces


Bridging interfaces does not get along well with DHCP, so make all your interfaces static. So your /etc/network/interfaces should resemble this:
auto eth0

iface eth0 inet static
address 192.168.1.1
netmask 255.255.255.0
broadcast 192.168.1.255
gateway 192.168.1.254

auto eth1
iface eth1 inet static
address 10.1.1.1
netmask 255.255.255.0
broadcast 10.1.1.255
# gateway 10.1.1.254


Note: The gateway on the second interface is commented out, as in this instance traffic is expected to go out via eth0 by default. If traffic is expected to go to different interfaces for different sub-nets, you need to work with iptables to do your routing.
Note: Do not worry about any special routing for the guest VMs, they can do that themselves. This is just the networking for the domU host.

Customising Xen networking


There are two parts to this step, creating a bridging script, and modifying the Xen configuration.

Create the script file /etc/xen/scripts/network-multi-bridge with the content:
#!/bin/sh

dir=$(dirname "$0")

# Add a new bridge for each interface, where vifnum is set to the interface number
"$dir/network-bridge" "$@" vifnum=0
"$dir/network-bridge" "$@" vifnum=1


Make the script executable by root:
$ chmod 755 /etc/xen/scripts/network-multi-bridge


Edit /etc/xen/xend-config.sxp and comment out (network-script network-bridge) and add your multi-bridge script, so the lines look like this:
#(network-script network-bridge)

(network-script network-multi-bridge)


That should do it.

Note: I not quite satisfied that this is working right yet...

Making grub2 boot with Xen by default


Mostly I like grub2, but I really dislike what update-grub2 does, in particular, how it likes to clobber your carefully crafted /boot/grub/grub.cfg file. Since this happens every time apt-fast dist-upgrade upgrades a kernel, it's better to make changes that will be preserved when update-grub2 is run.

The good news is update-grub2 will actually generate an appropriate menu entry for us, we just have to find it and make it the default entry.

First update your grub configuration:
$ update-grub2


This will create a whole bunch of menu entries in /boot/grub/grub.cfg the one you want is actually in there, it should look something like:
menuentry 'Ubuntu GNU/Linux, with Xen 4.1 and Linux 2.6.32.39-xen' --class ubuntu --class

gnu-linux --class gnu --class os --class xen {
insmod part_msdos
insmod ext2
set root='(/dev/sda,msdos1)'
search --no-floppy --fs-uuid --set=root <a UUID>
echo 'Loading Xen 4.1 ...'
multiboot /boot/xen-4.1.gz placeholder
echo 'Loading Linux 2.6.32.39-xen ...'
module /boot/vmlinuz-2.6.32.39-xen placeholder root=UUID=<a UUID> ro splash quiet
echo 'Loading initial ramdisk ...'
module /boot/initrd.img-2.6.32.39-xen
}


Restart your server and checks that this entry actually works, it might be hidden in a sub-menu.

Once you're sure you've found the entry you want, copy the whole section and paste it in at the end of /etc/grub.d/40_custom and change the entry name to something like "Xen 4.1 on Ubuntu 11.04 kernel 2.6.39-xen", this has to not match any other entry in /boot/grub/grub.cfg.

Now edit /etc/default/grub and change the default boot entry to:
GRUB_DEFAULT='Xen 4.1 on Ubuntu 11.04 kernel 2.6.39-xen'


Now rerun update-grub2 and restart, it should now boot to your carefully crafted Xen setup.

References


24 comments:

  1. Build Xen 4.1.1 on Ubuntu 11.10 and set up HVM via virt-manager

    http://bderzhavets.wordpress.com/2011/07/23/build-xen-4-1-1-on-ubuntu-11-10/

    ReplyDelete
  2. Hello, I follow your installation steps, everything is going fine but after I reboot my system trying to enter Linux-Xen kernel in grub menu.
    I can't enter Xen kernel, it shows:
    PANIC: early exception Oe rip 10:0 error 10 c r2 0

    any suggestions?

    regards

    ReplyDelete
  3. Under "Compile and Install Xen 4.1", under "Install dependencies", the apt-fast line should read "xserver-xorg-dev" vice "xserver-org-dev".

    ReplyDelete
  4. Chao, sorry, I don't have any suggestions apart from rechecking your kernel compiler options. I think I've seen similar behavior with Xen and non-Xen kernels. You might also want to recheck the typo I've just corrected for Tim.

    ReplyDelete
  5. My build worked nicely, once I figured out:
    - xorg vice org
    - fixed the path for gcrypt (I suspect there's a configure option for that)
    - system complaint, about xencommons and xend not running due to evtchn device not found, is fixed by running "modprobe xen-evtchn", then starting xencommons and then xend

    After that, the system boots and "xm list" reports normal stuff.

    A few questions about the libraries used in the tools build (libute-dev, libnlj-dev, and liblum2-dev):
    - Are they actually needed? (having trouble finding sources)
    - If so, which tool(s) require them?
    - Is there a site which hosts the sources? (Unified Trace Environment is difficult to find)

    ReplyDelete
  6. Hi Tim,

    I trimmed the libraries down from the install instructions I'd found on other sites (which were even more bloated), when I do another install I'll give it a go without them and see what happens.

    Thanks for the xencommons fix, and I think I'd done the gcrypt path as well, but haven't included it in the write up.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Hello, Bloodrage
    is any missing things on last command:
    update-initramfs -c -k 2.6.32.*-xen"
    in "Compile the kernel and install it the Debian way" ?
    because I got below error:

    update-initramfs: Generating /boot/initrd.img-2.6.32.*-xen
    dpkg: error: version '2.6.32.*-xen' has bad syntax: invalid character in version number
    WARNING: missing /lib/modules/2.6.32.*-xen
    Device driver support needs thus be built-in linux image!
    WARNING: Couldn't open directory /lib/modules/2.6.32.*-xen: No such file or directory

    if missing anything, please tell me what it is. thanks!

    because the error, I change as to
    $sudo update-initramfs -c -k 2.6.32.43-xen

    and finish all steps your mentioned on here, but
    when I reboot my pc I trying to entry Xen kernel via grub.
    it just a complete black screen without GUI interface!
    so please tell me why this happened?

    again, thank you!

    ReplyDelete
  9. another part of error about
    update-initramfs: Generating /boot/initrd.img-2.6.32.*-xen
    ========================================
    WARNING: Couldn't open directory /tmp/mkinitramfs_1Sd6lk/lib/modules/2.6.32.*-xen: No such file or directory
    FATAL: Could not open /tmp/mkinitramfs_1Sd6lk/lib/modules/2.6.32.*-xen/modules.dep.temp for writing: No such file or directory

    ReplyDelete
  10. I only can entry the kernel without xen kernel

    as you said, "similar behavior with Xen and non-Xen kernels."

    but it seems like I still can't start xend

    $ sudo xm list
    Error: Unable to connect to xend: No such file or directory. Is xend running?

    PS: sorry I am a little confuse about Tim and you said about xserver and server.
    should I just change that commands "server" as "xsever" ?
    anyway, I tried both way....

    apt-fast install libcurl3-gnutls-dev xserver-xorg-dev mercurial gitk build-essential libncurses5-dev uuid-dev gawk gettext texinfo bcc dpkg-dev debhelper kernel-package fakeroot iasl bridge-utils bison flex ocaml-findlib libparted-dev gparted

    ReplyDelete
  11. Chao, those steps will need to be done as root or with sudo.

    Tim pointed out that I had xserver-org-dev which is incorrect, it is xserver-xorg-dev, but apt should fail when it encounters a package that doesn't exist.

    When you're unable to connect with xend, it's most likely xend is not running, and the most likely cause is you're not running a xen kernel, so until you resolve the other issues, you're not going to progress with that one.

    ReplyDelete
  12. I only can entry the kernel without xen kernel

    as you said, "similar behavior with Xen and non-Xen kernels."

    but it seems like I still can't start xend

    $ sudo xm list
    Error: Unable to connect to xend: No such file or directory. Is xend running?

    PS: sorry I am a little confuse about Tim and you said about xserver and server.
    should I just change that commands "server" as "xsever" ?
    anyway, I tried both way....

    apt-fast install libcurl3-gnutls-dev xserver-xorg-dev mercurial gitk build-essential libncurses5-dev uuid-dev gawk gettext texinfo bcc dpkg-dev debhelper kernel-package fakeroot iasl bridge-utils bison flex ocaml-findlib libparted-dev gparted

    Hello, have you noticed this one?
    I stuck on here many days, do you know why this happened?

    regards

    ReplyDelete
  13. Chao, when I said "similar behavior with Xen and non-Xen kernels." I meant when trying to run the Xen hypervisor with a kernel that does not have the xen options compiled into it.

    I think you need to check your Grub2 setup to be sure that it's correctly preloading the Xen hypervisor with multiboot before loading your specifically compiled Xen kernel.

    ReplyDelete
  14. Chao, I had the same issue during my first install. If you _are_ running the proper kernel, try:

    1) modprobe evtchn
    2) service xencommons start
    3) service xend start

    ReplyDelete
  15. I've done something horrible to my build. It now reboots every 15 minutes or so.

    Note to self: stop mixing local builds with pre-packaged software. (bleh!)

    ReplyDelete
  16. Unsolicited rebooting turned out to be a hardware issue. New MB, new video card. Retrying build. Going to try including SMB feature in kernel (quad core!).

    ReplyDelete
  17. Works fine once you clean up:
    1. apt-fast install libgtk-vne-1.0-dev should be
    apt-fast install libgtk-vnc-1.0-dev

    ReplyDelete
  18. Hi ,

    I have installed ubuntu 11.04 on my external hard disk(500gb).Afterwards i installed Xen on it.When command( make && make modules_install && make install)finishes execution it gives me the following display

    Found linux image: /boot/vmlinuz-3.0.4
    Found initrd image: /boot/initrd.img-3.0.4
    dpkg: error: version 'syms-4.1.1' has bad syntax: version number does not start with digit
    dpkg: error: version '/boot/xen.gz' has bad syntax: version number does not start with digit
    Found linux image: /boot/vmlinuz-3.0.4
    Found initrd image: /boot/initrd.img-3.0.4
    dpkg: error: version 'syms-4.1.1' has bad syntax: version number does not start with digit
    dpkg: error: version '/boot/xen.gz' has bad syntax: version number does not start with digit
    Found linux image: /boot/vmlinuz-3.0.4
    Found initrd image: /boot/initrd.img-3.0.4
    dpkg: error: version '/boot/xen.gz' has bad syntax: version number does not start with digit
    Found linux image: /boot/vmlinuz-3.0.4
    Found initrd image: /boot/initrd.img-3.0.4
    Found linux image: /boot/vmlinuz-3.0.4
    Found initrd image: /boot/initrd.img-3.0.4
    Found memtest86+ image: /memtest86+.bin
    done
    root@khalid-Inspiron-1525:/usr/src/linux-source-3.0.0#

    I don't know from where these(dpkg) errors are coming from and how to fix it.I am beginner in Ubuntu and Xen so sorry if its too basic.

    ReplyDelete
  19. git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen

    Fatal: unable to lookup git.kernel.org port9418 name or service not known.

    What does it mean?

    ReplyDelete
    Replies
    1. It looks like a network issue.

      It means that either you can't access the internet, the name is not resolving correctly (check you've got the correct nameservers in /etc/resolv.conf) or your firewall is blocking port 9418.

      If you can do an update with apt-get, then it's most likely the firewall.

      Delete
  20. Hi,
    After git clone if I do git checkout as mentioned in the website I'm getting the below error.

    /usr/src/linux-2.6-xen# git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
    fatal: git checkout: updating paths is incompatible with switching branches.
    Did you intend to checkout 'origin/xen/stable-2.6.32.x' which can not be resolved as commit?

    ReplyDelete
  21. hi
    i installed ubuntu inside window as program.Now i am trying to install xen.When i run the command 'apt-get build-dep xen-3.3' it displays 'E:Unable to find a source package for xen-3.3'..Then i left this step and move to next but for command 'git clone
    git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen' it displays 'connection timeout'..

    ReplyDelete