Advogato blog for penguin42

23 May 2010

Sun, 23 May 2010 17:45:31 GMT

Fun with Ubuntu Lucid, KVM and iSCSI

(Note: I wrote this after doing this install and haven't walked through it and checked it, so there maybe thinkos)

I run Ubuntu Lucid on my desktop and wanted to play about with server stuff, and I've already got the KVM setup running with other guests running in it.

Here I show how I've set up a pair of Ubuntu Lucid server KVM guests, where one iSCSI serves the root to the other, with iSCSI booting using gPXE.

I won't claim this is efficient, but it's been interesting to setup - bits of this setup are probably useful in other scenarious, e.g. I think you should be able to iSCSI boot a KVM guest off an external SAN. There may even be more sane ways of doing this insane combination!

1) The base

For my base system I have Ubuntu Lucid (64bit) in my case on an i7-860 with 8GB of RAM. On it I have the kvm, qemu-common, qemu- kvm and kvm-pxe packages installed, together with libvirt- bin, libvirt0 and virt-manager.

I've got some of my disc space in a LVM volume group.

I configured the dhcp range for the default network in KVM downto only part of the range and setup a netboot like this ( /var/lib/libvirt/network/default.xml )



 <network>
  <name>default</name>
  <uuid>e72880a1-3159-cb84-4314-ee938ebb954b</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/libvirt/tftp' />
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.127' />
      <host mac='52:54:00:41:73:fb' name='client1' 
ip='192.168.122.130' />
      <bootp file='netboot' />
    </dhcp>
  </ip>
</network>

NOTE: To restart the libvirt dhcp server to use the config file you'll need to:


   stop libvirt-bin
   killall dnsmasq
   ifconfig virbr0 down
   brctl delbr
   start libvirt-bin

Create /var/lib/libvirt/tftp

2) The server guest

I used lvm to create an logical volume called server1a (which is now /dev/main/server1a on the host), and then created a guest using virt-manager and gave it that volume as a virtio disk. (The ability to create a new lv within virt-manager doesn't seem to work for me).

I then booted this off a Lucid server CD and installed it, creating it's own LVM hierarchy on it's disk and giving it a fixed IP (192.168.122.129). For convenience I then added an entry to the host machines /etc/hosts to point to that.

Within the server guest I created a new logical volume to be the root disk of the client, so from within the server this is /dev/server1/export.

We now have:


   Host physical Disk (/dev/sda on the host)
   fdisk partition for LVM (/dev/sda3 on the host)
   LVM volume group (main on the host)
   LVM logical volume for the server guest 
(/dev/main/server1a)

    Guest virtual Disk (/dev/vda on the server1 guest, 
/dev/main/server1a on host)
   fdisk partition for LVM (/dev/vda5 on server1 guest)
   LVM volume group (server1 on the server1 guest)
   LVM logical volume for the client guest 
(/dev/server1/export)

Now we can get iscsi serving going. On the server1 guest I installed the iscsitarget package, made sure that /etc/default/iscsitarget enabled it and wrote an /etc/ietd.conf with just the lines:


Target iqn.2010-05.org.treblig:server1.export.a.b
       Lun 0 Path=/dev/mapper/server1-export,Type=blockio

I then made sure this would run at boot by using

update-rc.d iscsitarget defaults

3) The client

I created another guest with virt-manager with no disk and booted it using the Lucid server disk. Note that I have a fixed IP specified in the DHCP config for libvirt for the MAC address of the client.

When it realises it hasn't got a disk it should give you the option of using iscsi, and giving it the IP of the server it should then list the iscsi target name from the ietd.conf.

I then did a normal lucid server install, and then shut it down.

4) Booting the client

This is somewhat trickier than it should be; the PXE netboot ROM in Qemu/KVM on Lucid doesn't have iSCSI booting in it. There are various ways to fix this, you could PXE boot Grub2 and then the kernel/initrd copied out of the client, but that wouldn't get updated by any kernel updates in the client. You can also replace the ROM in the Qemu, but I didn't want to change any of the packaged stuff.

My solution was to chain boot a gPXE with iSCSI support; I went to :

rom-o-matic

and selected .kpxe format, virtio-net for the NIC type, TFTP download protocol, SANBOOT_PROTO_ISCSI, AUTOBOOT_CMD, SANBOOT_CMD and most of the other defaults and then added the following embedded script:


ifopen net0
dhcp net0
chain /${ip}

rom-o-matic gives back an image file that I copied into /var/lib/libvirt/tftp on the host as a file called netboot. When the client netboots it will load this new instance of gPXE that will then try and load a file based on it's IP, in my case I've got a file called 192.168.122.130 in /var/lib/libvirt/tftp on the host with the following contents:


#!gpxe
sanboot iscsi:192.168.122.129::::iqn.2010-
05.org.treblig:server1.export.a.b

where the first part is the IP of the iSCSI server partition, and the second part is the identifier from the ietd.conf on the server.

Remember to set the Boot Options on the client to Network.

Now when I boot the client VM I boot all the way into Lucid client!

The boot process of the client is:


   DHCP, PXE boot, tftp fetch the new gPXE
   DHCP, fetch the /192.168.122.130 script
   iSCSI boot off the remote disk

Notes: On the client there is a lot of iscsi config in /etc, and /etc/iscsi in particular, if you move the server after installation you'll have to fix up all of that and then fix up the initramfs by regenerating it - it'll pick up the /etc/iscsi/iscsi.initramfs

18 Feb 2001

Sun, 18 Feb 2001 20:12:49 GMT

Ho hum; fancied posting a diary.

Top one of the weekend; don't try and build a cross compiler from Alpha to 68k using an old host compiler; egcs-1.1.2 screws up some floating point stuff - that took me ages to find.

Now I'm just stuck with binutils fighting when it tries to link.

Fighting autoconf and its mates trying to conditionally remove the win32 support in xine so I cna build it on Alpha.

6 Aug 2000

Sun, 6 Aug 2000 20:22:23 GMT

Hacked up one of he LCD panels fully off an old VGA card - working quite nicely; although the inverter circuit we cobbled together is a bit touchy. (Anyone with ideas for a nice, safe, stable, cheap inverter for cold cathode backlights please tell me!).

A kludge with SVGAtextmode gave me something almost right - although I face the option of loosing about 2 pixels off either edge (I can choose, but I always loose a couple of pixels...).

Fighting Linux/ARM on the old platforms; these machines have unreasonably large 32K pages (and not many of them) - there are some very odd things happening in some places. (Like a divide by 0 I got last week in fork_init).

16 Jul 2000

Sun, 16 Jul 2000 16:00:27 GMT

Interesting weekend; Thursday saw the arrival of 20 640x480 10.4" LCD panels (sans light inverters and VGA interface and case).

Before the ManLUG talk on Saturday I managed to get cabling from Maplin and other bits off a friend. Saturday evening/Sunday morning was crimping duty - without a crimp tool. Hellishly small crimps.

Early on Sunday I managed to get the panel running off a VGA feature connector (with just one colour wired) - looks promising.

Now if only I could array them.....

Spent the rest of time fighting trying to get a modern ARM cross compiler going,

11 Jun 2000

Sun, 11 Jun 2000 19:41:31 GMT

Spent Saturday fighting with my Sparc/IPX to try and install RedHat 6.2 on it. It already had 5.2 on another disc but since that disc was getting iffy I wanted a clean install.

Unfortunatly Disk Druid managed to create a partition with duff entries which seemed to indicate that it was around 1000 times larger than it really was. It took me some hours to discover this, and thus explain why mke2fs was taking many hours and consuming about 180M of RAM.

I selected 'server install' - and then spent a while fighting with RPM to get all the GNOME stuff in afterwards. Hmm GNOME is not that responsive on an IPX with 32MB of RAM.

I left it building XFree 4 over night; 14 hours later and it had built and I managed to get it not to start; ho hum - another X bug reported.

The interesting revelation of the day was bought to me by Owen Taylor. I had found that large selections just didn't work for me on Gnome on my Alpha - this turned out to be a brokedness in Gtk which was hitting a rather nasty 64 bit brokedness in X. Yuch!

14 May 2000

Sun, 14 May 2000 13:03:12 GMT

Downloaded KDE 1.90 over Friday/Saturday night (wonders of free net calls - even if it does drop the line every 2 hours, something which a retry on wget doesn't mind about).

Built; hit problems. It works fine if you let KDE take over your entire system by letting it start its window manager etc, but not if I just want to run some apps under Gnome - no keyboard input.

Having said that, running it in KDE domination mode seems to work fairly well. Koffice is promising - I wrote a set of slides in kpresenter and it only crashed a few times. (The fact that it stores documents as gzip'd tars of XML files is great when you have to go out and fix something it screwed up).

You can see the result at: www.treblig.org/talks/xfree4

19 Mar 2000

Sun, 19 Mar 2000 21:29:49 GMT

Got about midday.

Found the two X builds nor the EGCS that I had set building over night (over morning?) had built. And I hadn't slept well either. Bummer. EGCS seems to complain about a problem with the definition of RUSAGE something - and this seems to be kernel version dependent.

Got two different failure modes for X with DRI; either X segs internally, or I get a kernel oops in the drm_release code. Great.

Found a way to hang 2.3.99-pre2-5 - open the Gnome window manager configuration. Really odd - it seems pretty stable other than that......

After those hangs, it came to me that I should be using a journalling fs; waiting for a 34GB drive to fsck is just painful.

18 Mar 2000

Sat, 18 Mar 2000 23:23:54 GMT

Went to ManLUG; had a good chat and listened/watched a talk on the GIMP and a rather interesting talk from a nice man at SuSE who was handing out lots of goodies (got a Tux/suse tie/pin - now I just have to work out where to wear it since I don't often wear ties).

Built 2.3.99-pre2-5 on my Alpha and built the sourceforge CVS'd DRI version of X. X ain't happy running with DRI on my Alpha. Hmm.

Vaguely started thinking about an XFree 4 talk I'm going to give in a couple of months - hmm.

12 Mar 2000

Sun, 12 Mar 2000 20:03:17 GMT

Got up at about 12.30 - yawn.....

Built XFree 4.0; works a treat on my Alpha with my Voodoo 3 card; although I know some people are having problems with MGA and Permedia cards. Built GLUT and GLU to go with it and tried one or two examples; uploaded the builds to my ftp site.

Carried on my mods to Elvis for a Gnome front end; managed to get page up/down/home/end working and found a way to get a signal before a menu appears (thus hopefully enabling me to do a buffers menu). Added a 'goto line' dialog box.

Failed to figure out how to trap the closure of dialog boxes when they are closed by the window manager - hmm.

4 Mar 2000

Sat, 4 Mar 2000 20:28:45 GMT

Spent an hour battering pppd into auth'ing itself with BT internet - finally got it to work. Seems fast, but it has the annoying 2 hour disconnect problem.

Built 2.3.49 on my Alpha; this seems good - its fast and hasn't screwed up my disc!

Built Mozilla M14 - well it got a window up; but some parts were unrendered leaving bits of the lower windows showing through, and after a minute or two it fell to bits. Ah Mozilla.