Building a Dom0

Various distributions deliver the base to create a Xen domain-zero (Dom0). Below a few step-by-step CLI guides on getting from bare metal to a Dom0 for PV or HVM.

Install a minimal OpenSuse system, choose “other” graphical system, and choose text based server).

Now the system will come up with a graphical-screen (ROTFL). With ctrl-alt-F1 change to text-mode and log in with the created non-root user during install. Use:

sudo su - root
passwd root

to go to a more comfortable area. Use yast:

• system → System Services (expert): to the default runlevel to 3
• security → firewall: to disable the firewall
• system → System Services: to enable sshd in runlevel 3 (and optionally 4 and 5)
• network → to change the hostname and make the IP static

Update the repository cache and install generic usefull stuff:

zypper up
zypper install  mc iputils man bind-util vim-enhanced tcpdump rsync wget iftop

De-install conflicting patterns-openSUSE-minimal_base because it conflicts with python-base (which is a requirement for xen-tools). As a replacment for the package, some other packages need to be installed:

zypper remove  patterns-openSUSE-minimal_base
zypper install  python-base

Install the Xen hypervisor and a Xen-enabled Linux dom0 kernel:

zypper install  libvirt  xen-libs    xen  kernel-xen  xen-tools

Extra stuff:

zypper install  libvirt-python  vm-install  xen-tools-ioemu

The required linux-bridging stuff has come as dependency.

Change the default boot-option to the Xen hypervisor in /boot/grub/menu.lst

default 0

As an alternative for cmdline zypper; use Yast: Virtualization → Install Hypervisor and Tools → Xen to install the Xen hypervisor and dom0 patched kernel. Resolve the python dependency also.

Configure in /etc/xen/xend-config.sxp the following network-script line (and comment out all other network-script lines):

(network-script hannibal-xen-network-script)

Leave the line as is:

(vif-script vif-bridge)

And create a file (chmod +x) /etc/xen/scripts/hannibal-xen-network-script (an example for a machine with four physical NIC's:

#!/bin/sh
BASEPATH=/etc/xen/scripts
${BASEPATH}/network-bridge "$@" vifnum=0 netdev=eth0 bridge=xenbr0
${BASEPATH}/network-bridge "$@" vifnum=1 netdev=eth1 bridge=xenbr1
${BASEPATH}/network-bridge "$@" vifnum=2 netdev=eth2 bridge=xenbr2
${BASEPATH}/network-bridge "$@" vifnum=3 netdev=eth3 bridge=xenbr3

Further reading:

• http://www.scribd.com/doc/43734889/Xen3-Zero-to-Xen-From-the-CLI-in-OpenSuSE-10-3

Install a minimal Debian-6 (Squeeze) system in 64 bit. Preferrably configured with a static IP-configuration in /etc/network/interfaces (ie. DHCP-client not tested).

Prepare the system installing your usual administration tools of choice and some required for our Xen networking. For example:

apt-get install ssh dnsutils tcpdump bzip2 bridge-utils (etc.)

Install the Xen hypervisor and a Xen-enabled Linux kernel. This package will also install some dependencies as Python and iproute.

apt-get install  xen-hypervisor-4.0-amd64  linux-image-xen-amd64

To get Xen HVM support

apt-get install xen-qemu-dm-4.0

IMPORTANT! Make the Xen-hypervisor the first boot entry in grub:

mv /etc/grub.d/20_linux_xen /etc/grub.d/08_linux_xen

Append the stanza below to /etc/default/grub to prevent the prober to list al VM's:

# Disable OS prober to prevent virtual machines on logical volumes from appearing in the boot menu.
GRUB_DISABLE_OS_PROBER=true

To fix the Dom0 assigned memory to a static value, add to /etc/default/grub

# Xen4 dom0 and Grub2:
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=256M"

Finally, update the grub2 config:

update-grub2

When the physical machine (ie the dom0) has more than 1 physical NIC, configure in /etc/xen/xend-config.sxp the following network-script line (and comment out all other network-script lines):

(network-script hannibal-xen-network-script)

and comment out the line:

#(vif-script vif-bridge)

And create a file /etc/xen/scripts/hannibal-xen-network-script (an example for a machine with four physical NIC's:

#!/bin/sh
BASEPATH=/etc/xen/scripts
${BASEPATH}/network-bridge "$@" vifnum=0 bridge=xenbr0 netdev=eth0
${BASEPATH}/network-bridge "$@" vifnum=1 bridge=xenbr1 netdev=eth1
${BASEPATH}/network-bridge "$@" vifnum=2 bridge=xenbr2 netdev=eth2
${BASEPATH}/network-bridge "$@" vifnum=3 bridge=xenbr3 netdev=eth3

And make the script executable:

chmod 755 /etc/xen/scripts/hannibal-xen-network-script

Now there is a working setup, with multiple bridges. However, only the bridge which is configured with an IP-adress with a working gateway is activated fast. It takes a few minutes for the other bridges to come up. This is a bug in /etc/xen/scripts/network-bridge (squeeze/jan 2011)

The fix is to subsititute the line 118 with lines 119-121:

118         #[ -n "$gateway" ] && ip route add default via ${gateway}
119         if [ -n "$gateway" ]; then
120             ip route add default via ${gateway}
121         fi

See also: http://lists.xensource.com/archives/html/xen-users/2010-06/msg00420.html

If your hardware has a firewire device, udev will automatically create network devices (i.e. ethN) for them. Great, is'n it If you do not want this, edit /etc/udev/rules.d/70-persistent-net.rules and give them a proper name like firewN or completely disable them.

Add contrib and non-free to /etc/apt/sources and install the Realtec binary-firmware, and update your initrd image:

apt-get install firmware-realtek
apt-get install firmware-linux-nonfree
update-initramfs -u

See also:

• http://wiki.debian.org/Xen
• http://wiki.xensource.com/xenwiki/Xen4.0

See Debian 4.0

With the availability of 64-bit hardware in mind, we only use the AMD64 port of Debian Etch in order to create Xen enabled servers. There are several reasons for this decision, one of them is to avoid the TLS-issue's with the 32-bit Xen hypervisor on x86 CPU's.

Configure /etc/apt/sources.list just for the regular Debian repositories.

deb http://ftp.nl.debian.org/debian/ etch main contrib
deb-src http://ftp.nl.debian.org/debian/ etch main contrib
deb http://security.debian.org/ etch/updates main contrib
deb-src http://security.debian.org/ etch/updates main contrib

Then run:

apt-get update && apt-get dist-upgrade -fu

Prepare the system installing your usual administration tools of choice. For example:

apt-get install ssh dnsutils tcpdump bzip2 (etc.)

The following package is required for our Xen networking configuration:

apt-get install bridge-utils

Install the Xen hypervisor and a Xen-enabled Linux kernel. This package will also install some dependencies as Python and iproute.

apt-get install xen-linux-system-2.6.xx-x-xen-amd64              (2.6.xx-x is 2.6.18-5 January 2008)

or

apt-get install xen-linux-system-2.6.xx-x-xen-vserver-amd64      (incl. vserver support)

The required initial ramdisk is automatically rebuilt and the GRUB-config is adjusted for the new hypervisor/kernel combination. By default we add a Xen boot option to the hypervisor line in /boot/grub/menu.lst so that the hypervisor won't consume all memory. Although Xen automagically reallocates memory to the domU's, we prefer this boot option. Leave the line itself commented as this is required by grub.

# xenhopt=dom0_mem=256M

Now you can run:

update-grub

On 32-bit dom0 systems you'll want to install libc6 libraries that are optimized for the Xen hypervisor (to avoid the TLS issues).

apt-get install libc6-xen 

A reboot will complete the installation.

In /etc/default/xendomains, the following option can be changed to:

XENDOMAINS_SAVE=""

Networking is not trivial with Xen. All traffic goes through bridged NIC's (see: http://wiki.xensource.com/xenwiki/XenNetworking). The default configuration works out-of-the-box for systems with one NIC. Just leave the configuration in /etc/network/interfaces for the desired IP-configuration of the dom0

When the physical machine (ie the dom0) has more than 1 NIC, configure in /etc/xen/xend-config.sxp the following network-script line (and comment out all other network-script lines):

(network-script hannibal-xen-network-script)

And create a file /etc/xen/scripts/hannibal-xen-network-script using this content (an example for a machine with three physical NIC's:

#!/bin/sh
BASEPATH=/etc/xen/scripts
"$BASEPATH/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0
"$BASEPATH/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1
"$BASEPATH/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=xenbr2

And make executable:

 chmod 755 /etc/xen/scripts/hannibal-xen-network-script

(See also: http://www.debian-administration.org/articles/470).

In /etc/network/interfaces the regular eth0 has to be swapped now with for example:

auto xenbr0
iface xenbr0 inet static
  address       10.64.100.239
  network       10.64.100.0
  netmask       255.255.255.0
  broadcast     10.64.100.255
 gateway       10.64.100.1

In /etc/rc.local the following has to be added to make the bridges which have no IP-number working for the domU's:

/sbin/ifconfig xenbr0 up
/sbin/ifconfig xenbr1 up
/sbin/ifconfig xenbr2 up

If you have troubles with udev creating firewire network devices that mess up your intended network configuration, rename them to a higher ethX number in:

/etc/udev/rules.d/z25_persistent-net.rules

To enable the http-based admin tool, place the following in /etc/xen/xend-config.sxp

(xend-unix-server yes)
(xend-port        8000)

Not all linux default kernel parameters are optimal for the dom0 job. Add the part below to /etc/sysctl.conf for some network optimizations:

# ## some Xen dom0 optimizations:
net.core.rmem_max=4194304
net.core.wmem_max=4194304
net.core.netdev_max_backlog=2048
net.core.somaxconn=512

At this moment backports contains already the 3.2 version of the hypervisor with the credit-schedular. To use this, add the line below to /etc/apt/sources.list

deb http://www.backports.org/debian etch-backports main contrib non-free

and install with:

apt-get install xen-hypervisor-3.2-1-amd64 xen-utils-3.2-1 xen-utils-common xen-linux-system-2.6.18-6-xen-amd64

This installs the new hypervisor and utils, but does not make any changes to /boot/grub/menu.list so make these manually.

Start with the install of a minimal working Debian-sarge on your bare metal. Download a debian-31r2-i386-netinst.iso for this. At the end of the install procedure, answer no to the options task-select and dselect. As the result of this, you end up wit a minimal system. Use apt-get softwarepackagename to install stuff you need. For example ssh could be useful.

Use /dev/sda1 or /dev/hda1 for your root filesystem. This is a Xen requirement! The use of a 64bit is highly recommended. Only when you have 32bits hardware use the plain i386 version.

append the following line to /etc/apt/sources.list

## ################# Xen-stuff:
## (replace <DISTRO> with either sid, sarge or etch and <ARCH> with i386 or amd64.)
deb     http://packages.debianbase.de/<DISTRO>/<ARCH>/xen3 ./
deb-src http://packages.debianbase.de/sources/xen3 ./
#

and use the command:

apt-get update
apt-get install xen linux-xen0-2.6      (or a newer version, if available :-)

to install the debian-base Xen packages. Read and follow the instructions mentioned during the install of this package carefully. You have to make an initial-ramdisk for your situation and an addition to menu.list (grub).

and use the command:

apt-get install bridge-utils

for some extra stuff.

In case of an i386 architecture do:

mv /lib/tls /lib/tls.disabled

Install the kernel as a package:

dpkg -i /usr/src/kernel-xen.version-here-xxxxxxxxxxxxxxxx.deb

Make a initial-ramdisk:

mkinitrd -o /boot/xen-modules-2.6.12.6-xen 2.6.12.6-xen

Add a section to the grub menu.list:

title     Xen 3.0.1 - Linux 2.6.12.6-xen
           kernel      /boot/xen-3.0.1.gz dom0_mem=131072
           module      /boot/xen-linux-2.6.12.6-xen root=/dev/sda1 ro
           module      /boot/xen-modules-2.6.12.6-xen

When using more than 1 physical xen host on an ethernet subnet, place a # before the commands in the script: /etc/xen/scripts/network-bridge on the follow lines:

#       ip link set ${pdev} addr fe:ff:ff:ff:ff:ff
#       ip link set ${netdev} addr fe:ff:ff:ff:ff:ff

Create an /etc/init.d/xennetworking and place an /etc/rc2.d/S20 symlink

#!/bin/bash
#
# This scripts sets up the Xen networking environment.
# It uses the existing Xen scripts to create bridges on the host machine and to add interfaces to them.
#
# eth0 = realtec 8169       = OOB1
# eth1 = via rhine          = internet
#
# Use Xen-scripts to create bridges
/etc/xen/scripts/network-bridge start bridge=xenbr0 netdev=eth0 vifnum=0
/etc/xen/scripts/network-bridge start bridge=xenbr1 netdev=eth1 vifnum=1
/etc/xen/scripts/network-bridge start bridge=xenbr2 netdev=eth2 vifnum=2
#
# Patience is good
sleep 2
#
####################
# Set IP-configuration on bridges
#
# OOB1 = 10.66.1.121/24
# OOB2 = -
# Internet = nvt.
/sbin/ip addr add 172.17.1.5/24 dev xenbr0
#
# Set default route
/sbin/ip route add default via 172.17.1.1

RHEL-6 does not ship Xen hypervisor or tools and does not ship Xen dom0 capable kernel.

It can be build, see: http://wiki.xen.org/xenwiki/RHEL6Xen4Tutorial

We prefer to run our Xen guests from logical volumes instead of running them from loop mounted files. By default we choose to format the file systems of the Xen guests to xfs. Install the necessary LVM and XFS packages:

apt-get install lvm2 dmsetup xfsprogs xfsdump

Create an LVM (8E-type) partition using fdisk and prepare a logical volume for the domU configuration files. We'll mount this volume on /opt/xen.

pvcreate /dev/sda5
vgcreate vg00 /dev/sda5
lvcreate -n optxen -L 20G vg00
mkdir /opt/xen
mkfs.xfs /dev/vg00/optxen

Add this mountpoint to /etc/fstab so that the volume will be automagically remounted on system reboots.

/dev/mapper/vg00-optxen   /opt/xen   xfs   defaults   0   2

Mount the partition we just created.

mount -a

Now we can create logical volumes for our Xen guests. Don't forget to add a volume per domU for swap!

If you want you might continue reading the section creation_of_a_domu.

For APC equipment, there is software ready in the Debian repository. Good reason to buy this if you need uninterrupted power. After connecting the equipment (power and USB-cabling), install the ups-daemon software:

apt-get install  apcupsd apcupsd-doc apcupsd-cgi

Edit the configfile /etc/apcupsd/apcupsd.conf

## apcupsd.conf v1.1 ##
#
#  for apcupsd release 3.14.8 (16 January 2010) - debian
#
# "apcupsd" POSIX config file
UPSNAME ups01
UPSCABLE usb
UPSTYPE usb
DEVICE
LOCKFILE /var/lock
ONBATTERYDELAY 6
BATTERYLEVEL 90
MINUTES 10
TIMEOUT 0
ANNOY 300
#
UPSCLASS standalone
UPSMODE disable
#
STATFILE /var/log/apcupsd.status
EVENTSFILE /var/log/apcupsd.events

And flag the parameter ISCONFIGURED in /etc/default/apcupsd to yes.