Heartbreak Ridge

Action movie from 1986 starring Clint Eastwood as hard boiled gunnery sergeant Tom Highway, trying to make real soldiers out of a spoiled platoon.

While entering for the first time he makes this very clear. After the guys refuse to turn off the music, he throws the radio to the other side of the room, smashes a pool stick on the table, and kicks the bin. Then these famous words are spoken:

Your ass is mine!
I’m here to tell you all, that live as you know it has ended. You might as well go all to town tonight, you might as well laugh and make fools of yourselves, rub your pathetic little peckers up against your honey, or stick it in a knothole in the fence. But whatever it is, get rid of it – because at 0600 tomorrow… your ass is mine!”

Check http://www.youtube.com/watch?v=QqM_tPumTZI.

Too bad we can’t hire this guy to teach some discipline 😉

mount: special device /var/run does not exist

While P2V-ing an Ubuntu 6.06 server with my warm-cloning P2V method, I ran into a strange problem when booting:

Sceendump - click to enlarge

The exact text of the error is:

mount: special device /var/run does not exist
mount: special device /var/lock does not exist
mount: wrong fs type, bad option, bad superblock on /dev/shm/var.run,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so

mount: wrong fs type, bad option, bad superblock on /dev/shm/var.lock,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so

Everything seems to work, untill I found out that the extra IP addresses on secondary network interfaces were not able to carry any network traffic…
I had to cancel the virtualization and revert to the physical machine again.
The system in question has a separate /var partition (see booting picture, /dev/sda6 in my case).
I first mount the root filesystem, which holds on empty “var” directory, and then mount /dev/sda6 on that “var” directory. I had created empty “run” and “lock” directories in there to be able to mount /var/run and /var/lock. Wrong!

Turns out that the root filesystem needs to contain /var/run and /var/lock, even though the system has a separate /var partition.

:bonk: :bonk: :bonk:

Thanks to Chris Siebenmann for pointing this out on his wiki.

The solution thus is to boot from Ubuntu Live CD, mount ONLY the root filesystem, and create the /var/run and /var/lock directories.
They are only needed for mounting the tempfs partitions, and will be hidden by the real /var partition, which is mounted over it once the system has finished booting.

Manual P2V of Debian Sarge

Now that we at TERENA have a new and shiny setup of VMware VI3, I had to migrate several of our Debian 3.1 Sarge servers. Some of them had custom kernels, because of specific hardware.

This virtualisation process (P2V, or Physical to Virtual) is properly supported for the Windows platform using the VMware Converter software. This works very nice, and supports hot cloning. However when your Physical Machine (PM) is running Linux, hot cloning is not possible. The way to go is to use the VMWare Converter Boot CD. This requires rebooting the PM with a bootable CDROM, and from the PE-boot environment on that CDROM the dead corps is cloned to a VM.

The downside is of course that the machine has to be brought down for a substantial amount of time. Also, if your PM uses specific I/O controllers and/or network cards, the boot CDROM will need to be customized to hold the right drivers. This has to be tested too, so it might even take several times of downtime.

By doing things manually, you can avoid almost all of the downtime. I P2V-ed 3 systems, all over 100Gb, each with less than 20 minutes downtime.

Also, because you ‘warm’ clone a live system, you don’t need to worry about disk and network drivers. Another benefit compared to cold cloning is that you can test things first on a dummy VM without any downtime at all.
This article documents all the steps needed. It assumes your old PM is running Debian Sarge with one of the 2.6.8 kernels, uses GRUB as bootloader, has rsync installed, and can be reached by the root user via SSH.

The procedure basically comes down to cloning a live system to a dead VM, stopping all services, do a final syncronisation, and revive the dead VM.

Step-by-step guide:

  • Create a VM with at least as much diskspace as the PM.
  • Configure the VM to boot an ISO image of Ubuntu 6.06 LTS Desktop Edition
  • Open up a shell, su to root, and partition the disk. If you stick to exactly the same partition scheme, you don’t have to change the fstab file. You can change the size without any problem too. If you decide to change the partition scheme, be sure to not split directories that contain hard links. For example, if your PM has just one big /-partition, and you decide that the new VM will have separate / and /usr partitions, this will not work because hard links cannot be created across partitions.
  • Once all partitions are created, make filesystems on them (don’t forget swap), and mount them in the correct order under a temporary directory, let’s say /root/oldbox . Create root’s dir /root/oldbox/root and in there create a file /root/oldbox/root/excluded that contains:


    If you changed the partition scheme, you should put /etc/fstab here too, and manually put the correct one in place.
  • cd into /root/oldbox/root and rsync everything from the PM into it:
    rsync -avH --numeric-ids --delete \
    --exclude-from=root/excluded IP_of_PM:/ .

    This will take a while, depending on the amount of data your PM has.
  • Once everything is copied over, the time has come to shutdown all data-writing services on your PM (mail, databases, etc). Ideally only the SSH daemon should run. This means that most of your services will be offline from here. The good thing is that this period can be really small.
  • Once you made sure that nothing runs except SSH on your PM, rerun the rsync command. This time it will be quick, as only the diffs will need to be transferred. This usually involves open files from databases, logfiles, etc.
  • Now create the initial device nodes needed for the kernel:
    cd /root/oldbox
    mknod -m 660 /root/oldbox/dev/console c 5 1
    mknod -m 660 /root/oldbox/dev/null c 1 3
  • mount the proc and dev filesystem and chroot into the /root/oldbox dir:

    mount -t proc none /root/oldbox/proc
    mount -o bind /dev /root/oldbox/dev
    chroot /root/oldbox
  • If we now would reboot, the old initrd image would not recognize the proper modules to (unless your PM accidentally had a LSI controller). We need to add the drivers. To do this, add this to the file /etc/mkinitrd/modules (assuming your PM runs one of the 2.6.8 debian kernels):


    And regenerate the initrd image (depending on your specific kernel version):
    mkinitrd -o /boot/initrd.img-2.6.8-4-686-smp 2.6.8-4-686-smp
    (Since my PM had an older, custom kernel (2.6.8-3-686-smp), I installed a newer one, plus udevd. During installation a new initrd image is generated automatically:
    apt-get install udev kernel-image-2.6.8-4-686-smp)
  • Now we need to regenerate the bootblock. Run the grub command to enter the grub shell and see where it finds the stage1 file:
    find /boot/grub/stage1

    It should come up with something like (hd0,1). Use this as argument for the next command:root (hd0,1)
    Then use the hd part only for the next command:
    setup (hd0)
    Then issue quit to leave the grub shell.

By now your system is ready to boot. Leave the chroot environment (exit), unmount the dev and proc filesystem, then unmount all filesystems under /root/oldbox, issue sync, and then halt.
To avoid network clash, unplug the network cable of the PM, or shut it down.
Now you can power on your VM, it should boot a virtualized copy of your Debian system :-)

TODO – Some things to do afterwards (vmtools, etc)

Configuring a static IPv6 address in Windows 2003 with netsh

For each network interface, Windows 2003 Server offers a GUI to configure IPv4 parameters (Internet Protocol (TCP/IP)).

After installing IPv6 there is a new item Microsoft TCP/IP Version 6. When selecting that, the “Properties” box becomes grey-out.

This means that there is no GUI to configure IPv6 parameters. You will have to use the netsh command.

This is quite simple, but it takes some effort to find out the right combination of switches to make it happen.

As an example my employer’s WindowsMedia streaming server. This box has native IPv6 connectivity.

I set the address and gateway as follows:

netsh interface ipv6 set privacy disabled
netsh interface ipv6 add address interface="Local Area Connection" \
    address=2001:610:148:dead:202:b3ff:fe9a:d264 store=persistent
netsh interface ipv6 add route ::/0 interface="Local Area Connection" \
    2001:610:148:dead::1 store=persistent