======Building a Dom0====== Various distributions deliver the base to create a Xen domain-zero (Dom0). Below a few step-by-step CLI guides on getting from bare metal to a Dom0 for PV or HVM. =====Suse===== ====OpenSuse 11.4==== Install a minimal OpenSuse system, choose "other" graphical system, and choose text based server). Now the system will come up with a graphical-screen (ROTFL). With ctrl-alt-F1 change to text-mode and log in with the created non-root user during install. Use: sudo su - root passwd root to go to a more comfortable area. Use yast: * system -> System Services (expert): to the default runlevel to 3 * security -> firewall: to disable the firewall * system -> System Services: to enable sshd in runlevel 3 (and optionally 4 and 5) * network -> to change the hostname and make the IP static Update the repository cache and install generic usefull stuff: zypper up zypper install mc iputils man bind-util vim-enhanced tcpdump rsync wget iftop De-install conflicting patterns-openSUSE-minimal_base because it conflicts with python-base (which is a requirement for xen-tools). As a replacment for the package, some other packages need to be installed: zypper remove patterns-openSUSE-minimal_base zypper install python-base Install the Xen hypervisor and a Xen-enabled Linux dom0 kernel: zypper install libvirt xen-libs xen kernel-xen xen-tools Extra stuff: zypper install libvirt-python vm-install xen-tools-ioemu The required linux-bridging stuff has come as dependency. Change the default boot-option to the Xen hypervisor in /boot/grub/menu.lst default 0 As an alternative for cmdline zypper; use Yast: Virtualization -> Install Hypervisor and Tools -> Xen to install the Xen hypervisor and dom0 patched kernel. Resolve the python dependency also. ===bridged networking=== Configure in /etc/xen/xend-config.sxp the following network-script line (and comment out all other network-script lines): (network-script hannibal-xen-network-script) Leave the line as is: (vif-script vif-bridge) And create a file (chmod +x) /etc/xen/scripts/hannibal-xen-network-script (an example for a machine with four physical NIC's: #!/bin/sh BASEPATH=/etc/xen/scripts ${BASEPATH}/network-bridge "$@" vifnum=0 netdev=eth0 bridge=xenbr0 ${BASEPATH}/network-bridge "$@" vifnum=1 netdev=eth1 bridge=xenbr1 ${BASEPATH}/network-bridge "$@" vifnum=2 netdev=eth2 bridge=xenbr2 ${BASEPATH}/network-bridge "$@" vifnum=3 netdev=eth3 bridge=xenbr3 ===References=== Further reading: * [[http://www.scribd.com/doc/43734889/Xen3-Zero-to-Xen-From-the-CLI-in-OpenSuSE-10-3]] * =====Debian===== ====Debian 6.0 (Squeeze)==== Install a minimal Debian-6 (Squeeze) system in 64 bit. Preferrably configured with a static IP-configuration in /etc/network/interfaces (ie. DHCP-client not tested). Prepare the system installing your usual administration tools of choice and some required for our Xen networking. For example: apt-get install ssh dnsutils tcpdump bzip2 bridge-utils (etc.) Install the Xen hypervisor and a Xen-enabled Linux kernel. This package will also install some dependencies as Python and iproute. apt-get install xen-hypervisor-4.0-amd64 linux-image-xen-amd64 To get Xen HVM support apt-get install xen-qemu-dm-4.0 ===grub2 corrections and tuning=== IMPORTANT! Make the Xen-hypervisor the first boot entry in grub: mv /etc/grub.d/20_linux_xen /etc/grub.d/08_linux_xen Append the stanza below to /etc/default/grub to prevent the prober to list al VM's: # Disable OS prober to prevent virtual machines on logical volumes from appearing in the boot menu. GRUB_DISABLE_OS_PROBER=true To fix the Dom0 assigned memory to a static value, add to /etc/default/grub # Xen4 dom0 and Grub2: GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=256M" Finally, update the grub2 config: update-grub2 ===bridged networking=== When the physical machine (ie the dom0) has more than 1 physical NIC, configure in /etc/xen/xend-config.sxp the following network-script line (and comment out all other network-script lines): (network-script hannibal-xen-network-script) and comment out the line: #(vif-script vif-bridge) And create a file /etc/xen/scripts/hannibal-xen-network-script (an example for a machine with four physical NIC's: #!/bin/sh BASEPATH=/etc/xen/scripts ${BASEPATH}/network-bridge "$@" vifnum=0 bridge=xenbr0 netdev=eth0 ${BASEPATH}/network-bridge "$@" vifnum=1 bridge=xenbr1 netdev=eth1 ${BASEPATH}/network-bridge "$@" vifnum=2 bridge=xenbr2 netdev=eth2 ${BASEPATH}/network-bridge "$@" vifnum=3 bridge=xenbr3 netdev=eth3 And make the script executable: chmod 755 /etc/xen/scripts/hannibal-xen-network-script Now there is a working setup, with multiple bridges. However, only the bridge which is configured with an IP-adress with a working gateway is activated fast. It takes a few minutes for the other bridges to come up. This is a bug in /etc/xen/scripts/network-bridge (squeeze/jan 2011) The fix is to subsititute the line 118 with lines 119-121: 118 #[ -n "$gateway" ] && ip route add default via ${gateway} 119 if [ -n "$gateway" ]; then 120 ip route add default via ${gateway} 121 fi See also: [[http://lists.xensource.com/archives/html/xen-users/2010-06/msg00420.html]] ==network devices== If your hardware has a firewire device, udev will automatically create network devices (i.e. ethN) for them. Great, is'n it :-) If you do not want this, edit /etc/udev/rules.d/70-persistent-net.rules and give them a proper name like firewN or completely disable them. Add contrib and non-free to /etc/apt/sources and install the Realtec binary-firmware, and update your initrd image: apt-get install firmware-realtek apt-get install firmware-linux-nonfree update-initramfs -u ===Refs=== See also: * [[http://wiki.debian.org/Xen]] * [[http://wiki.xensource.com/xenwiki/Xen4.0]] ====Debian 5.0 (Lenny)==== See Debian 4.0 ====Debian 4.0 (Etch)==== ===Core OS and hypervisor=== ==create a bootable system== With the availability of 64-bit hardware in mind, we only use the AMD64 port of Debian Etch in order to create Xen enabled servers. There are several reasons for this decision, one of them is to avoid the TLS-issue's with the 32-bit Xen hypervisor on x86 CPU's. Configure /etc/apt/sources.list just for the regular Debian repositories. deb http://ftp.nl.debian.org/debian/ etch main contrib deb-src http://ftp.nl.debian.org/debian/ etch main contrib deb http://security.debian.org/ etch/updates main contrib deb-src http://security.debian.org/ etch/updates main contrib Then run: apt-get update && apt-get dist-upgrade -fu Prepare the system installing your usual administration tools of choice. For example: apt-get install ssh dnsutils tcpdump bzip2 (etc.) The following package is required for our Xen networking configuration: apt-get install bridge-utils Install the Xen hypervisor and a Xen-enabled Linux kernel. This package will also install some dependencies as Python and iproute. apt-get install xen-linux-system-2.6.xx-x-xen-amd64 (2.6.xx-x is 2.6.18-5 January 2008) or apt-get install xen-linux-system-2.6.xx-x-xen-vserver-amd64 (incl. vserver support) The required initial ramdisk is automatically rebuilt and the GRUB-config is adjusted for the new hypervisor/kernel combination. By default we add a Xen boot option to the hypervisor line in /boot/grub/menu.lst so that the hypervisor won't consume all memory. Although Xen automagically reallocates memory to the domU's, we prefer this boot option. Leave the line itself commented as this is required by grub. # xenhopt=dom0_mem=256M Now you can run: update-grub On 32-bit dom0 systems you'll want to install libc6 libraries that are optimized for the Xen hypervisor (to avoid the TLS issues). apt-get install libc6-xen A reboot will complete the installation. ==Xen config== In /etc/default/xendomains, the following option can be changed to: XENDOMAINS_SAVE="" ==networking with bridges== Networking is not trivial with Xen. All traffic goes through bridged NIC's (see: http://wiki.xensource.com/xenwiki/XenNetworking). The default configuration works out-of-the-box for systems with one NIC. Just leave the configuration in /etc/network/interfaces for the desired IP-configuration of the dom0 When the physical machine (ie the dom0) has more than 1 NIC, configure in /etc/xen/xend-config.sxp the following ''network-script'' line (and comment out all other ''network-script'' lines): (network-script hannibal-xen-network-script) And create a file /etc/xen/scripts/hannibal-xen-network-script using this content (an example for a machine with three physical NIC's: #!/bin/sh BASEPATH=/etc/xen/scripts "$BASEPATH/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0 "$BASEPATH/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1 "$BASEPATH/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=xenbr2 And make executable: chmod 755 /etc/xen/scripts/hannibal-xen-network-script (See also: http://www.debian-administration.org/articles/470). In /etc/network/interfaces the regular eth0 has to be swapped now with for example: auto xenbr0 iface xenbr0 inet static address 10.64.100.239 network 10.64.100.0 netmask 255.255.255.0 broadcast 10.64.100.255 gateway 10.64.100.1 In /etc/rc.local the following has to be added to make the bridges which have no IP-number working for the domU's: /sbin/ifconfig xenbr0 up /sbin/ifconfig xenbr1 up /sbin/ifconfig xenbr2 up ==udev== If you have troubles with udev creating firewire network devices that mess up your intended network configuration, rename them to a higher ''ethX'' number in: /etc/udev/rules.d/z25_persistent-net.rules ==virtual machine administration== To enable the http-based admin tool, place the following in /etc/xen/xend-config.sxp (xend-unix-server yes) (xend-port 8000) ===Performance tuning=== Not all linux default kernel parameters are optimal for the dom0 job. Add the part below to /etc/sysctl.conf for some network optimizations: # ## some Xen dom0 optimizations: net.core.rmem_max=4194304 net.core.wmem_max=4194304 net.core.netdev_max_backlog=2048 net.core.somaxconn=512 ===Hypervisor updates from backports=== At this moment backports contains already the 3.2 version of the hypervisor with the credit-schedular. To use this, add the line below to /etc/apt/sources.list deb http://www.backports.org/debian etch-backports main contrib non-free and install with: apt-get install xen-hypervisor-3.2-1-amd64 xen-utils-3.2-1 xen-utils-common xen-linux-system-2.6.18-6-xen-amd64 This installs the new hypervisor and utils, but does not make any changes to /boot/grub/menu.list so make these manually. ====Debian 3.0 (Sarge)==== ===Core operating system and hypervisor=== ==create a bootable system== Start with the install of a minimal working Debian-sarge on your bare metal. Download a debian-31r2-i386-netinst.iso for this. At the end of the install procedure, answer no to the options task-select and dselect. As the result of this, you end up wit a minimal system. Use apt-get softwarepackagename to install stuff you need. For example ssh could be useful. Use /dev/sda1 or /dev/hda1 for your root filesystem. This is a Xen requirement! The use of a 64bit is highly recommended. Only when you have 32bits hardware use the plain i386 version. ==repository== append the following line to /etc/apt/sources.list ## ################# Xen-stuff: ## (replace with either sid, sarge or etch and with i386 or amd64.) deb http://packages.debianbase.de///xen3 ./ deb-src http://packages.debianbase.de/sources/xen3 ./ # and use the command: apt-get update apt-get install xen linux-xen0-2.6 (or a newer version, if available :-) to install the debian-base Xen packages. Read and follow the instructions mentioned during the install of this package carefully. You have to make an initial-ramdisk for your situation and an addition to menu.list (grub). and use the command: apt-get install bridge-utils for some extra stuff. ==post aptget== In case of an i386 architecture do: mv /lib/tls /lib/tls.disabled Install the kernel as a package: dpkg -i /usr/src/kernel-xen.version-here-xxxxxxxxxxxxxxxx.deb Make a initial-ramdisk: mkinitrd -o /boot/xen-modules-2.6.12.6-xen 2.6.12.6-xen Add a section to the grub menu.list: title Xen 3.0.1 - Linux 2.6.12.6-xen kernel /boot/xen-3.0.1.gz dom0_mem=131072 module /boot/xen-linux-2.6.12.6-xen root=/dev/sda1 ro module /boot/xen-modules-2.6.12.6-xen ==networking with bridges== When using more than 1 physical xen host on an ethernet subnet, place a # before the commands in the script: /etc/xen/scripts/network-bridge on the follow lines: # ip link set ${pdev} addr fe:ff:ff:ff:ff:ff # ip link set ${netdev} addr fe:ff:ff:ff:ff:ff Create an /etc/init.d/xennetworking and place an /etc/rc2.d/S20 symlink #!/bin/bash # # This scripts sets up the Xen networking environment. # It uses the existing Xen scripts to create bridges on the host machine and to add interfaces to them. # # eth0 = realtec 8169 = OOB1 # eth1 = via rhine = internet # # Use Xen-scripts to create bridges /etc/xen/scripts/network-bridge start bridge=xenbr0 netdev=eth0 vifnum=0 /etc/xen/scripts/network-bridge start bridge=xenbr1 netdev=eth1 vifnum=1 /etc/xen/scripts/network-bridge start bridge=xenbr2 netdev=eth2 vifnum=2 # # Patience is good sleep 2 # #################### # Set IP-configuration on bridges # # OOB1 = 10.66.1.121/24 # OOB2 = - # Internet = nvt. /sbin/ip addr add 172.17.1.5/24 dev xenbr0 # # Set default route /sbin/ip route add default via 172.17.1.1 =====Redhat===== ====Redhat 6.x ==== RHEL-6 does not ship Xen hypervisor or tools and does not ship Xen dom0 capable kernel. It can be build, see: http://wiki.xen.org/xenwiki/RHEL6Xen4Tutorial =====Dom0 additions===== ====Debian==== ===Volume manager=== We prefer to run our Xen guests from logical volumes instead of running them from loop mounted files. By default we choose to format the file systems of the Xen guests to xfs. Install the necessary LVM and XFS packages: apt-get install lvm2 dmsetup xfsprogs xfsdump Create an LVM (8E-type) partition using fdisk and prepare a logical volume for the domU configuration files. We'll mount this volume on /opt/xen. pvcreate /dev/sda5 vgcreate vg00 /dev/sda5 lvcreate -n optxen -L 20G vg00 mkdir /opt/xen mkfs.xfs /dev/vg00/optxen Add this mountpoint to /etc/fstab so that the volume will be automagically remounted on system reboots. /dev/mapper/vg00-optxen /opt/xen xfs defaults 0 2 Mount the partition we just created. mount -a Now we can create logical volumes for our Xen guests. Don't forget to add a volume per domU for swap! If you want you might continue reading the section [["xen:creation_of_a_domu"|creation_of_a_domu]]. ====Power==== For APC equipment, there is software ready in the Debian repository. Good reason to buy this if you need uninterrupted power. After connecting the equipment (power and USB-cabling), install the ups-daemon software: apt-get install apcupsd apcupsd-doc apcupsd-cgi Edit the configfile /etc/apcupsd/apcupsd.conf ## apcupsd.conf v1.1 ## # # for apcupsd release 3.14.8 (16 January 2010) - debian # # "apcupsd" POSIX config file UPSNAME ups01 UPSCABLE usb UPSTYPE usb DEVICE LOCKFILE /var/lock ONBATTERYDELAY 6 BATTERYLEVEL 90 MINUTES 10 TIMEOUT 0 ANNOY 300 # UPSCLASS standalone UPSMODE disable # STATFILE /var/log/apcupsd.status EVENTSFILE /var/log/apcupsd.events And flag the parameter ISCONFIGURED in /etc/default/apcupsd to yes.