TIENHUIS Networking

While P2V-ing an Ubuntu 6.06 server with my warm-cloning P2V method, I ran into a strange problem when booting:

The exact text of the error is:

mount: special device /var/run does not exist
mount: special device /var/lock does not exist
mount: wrong fs type, bad option, bad superblock on /dev/shm/var.run,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so

mount: wrong fs type, bad option, bad superblock on /dev/shm/var.lock,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so

Everything seems to work, untill I found out that the extra IP addresses on secondary network interfaces were not able to carry any network traffic…
I had to cancel the virtualization and revert to the physical machine again.
The system in question has a separate /var partition (see booting picture, /dev/sda6 in my case).
I first mount the root filesystem, which holds on empty “var” directory, and then mount /dev/sda6 on that “var” directory. I had created empty “run” and “lock” directories in there to be able to mount /var/run and /var/lock. Wrong!

Turns out that the root filesystem needs to contain /var/run and /var/lock, even though the system has a separate /var partition.

:bonk: :bonk: :bonk:

Thanks to Chris Siebenmann for pointing this out on his wiki.

The solution thus is to boot from Ubuntu Live CD, mount ONLY the root filesystem, and create the /var/run and /var/lock directories.
They are only needed for mounting the tempfs partitions, and will be hidden by the real /var partition, which is mounted over it once the system has finished booting.

Now that we at TERENA have a new and shiny setup of VMware VI3, I had to migrate several of our Debian 3.1 Sarge servers. Some of them had custom kernels, because of specific hardware.

This virtualisation process (P2V, or Physical to Virtual) is properly supported for the Windows platform using the VMware Converter software. This works very nice, and supports hot cloning. However when your Physical Machine (PM) is running Linux, hot cloning is not possible. The way to go is to use the VMWare Converter Boot CD. This requires rebooting the PM with a bootable CDROM, and from the PE-boot environment on that CDROM the dead corps is cloned to a VM.

The downside is of course that the machine has to be brought down for a substantial amount of time. Also, if your PM uses specific I/O controllers and/or network cards, the boot CDROM will need to be customized to hold the right drivers. This has to be tested too, so it might even take several times of downtime.

By doing things manually, you can avoid almost all of the downtime. I P2V-ed 3 systems, all over 100Gb, each with less than 20 minutes downtime.

Also, because you ‘warm’ clone a live system, you don’t need to worry about disk and network drivers. Another benefit compared to cold cloning is that you can test things first on a dummy VM without any downtime at all.
This article documents all the steps needed. It assumes your old PM is running Debian Sarge with one of the 2.6.8 kernels, uses GRUB as bootloader, has rsync installed, and can be reached by the root user via SSH.

The procedure basically comes down to cloning a live system to a dead VM, stopping all services, do a final syncronisation, and revive the dead VM.

Step-by-step guide:

• Create a VM with at least as much diskspace as the PM.
• Configure the VM to boot an ISO image of Ubuntu 6.06 LTS Desktop Edition
• Open up a shell, su to root, and partition the disk. If you stick to exactly the same partition scheme, you don’t have to change the fstab file. You can change the size without any problem too. If you decide to change the partition scheme, be sure to not split directories that contain hard links. For example, if your PM has just one big /-partition, and you decide that the new VM will have separate / and /usr partitions, this will not work because hard links cannot be created across partitions.
• Once all partitions are created, make filesystems on them (don’t forget swap), and mount them in the correct order under a temporary directory, let’s say /root/oldbox . Create root’s dir /root/oldbox/root and in there create a file /root/oldbox/root/excluded that contains:
  
/proc/*
/sys/*
/dev/*
/mnt/*
/tmp/*
/root/excluded

  If you changed the partition scheme, you should put /etc/fstab here too, and manually put the correct one in place.
• cd into /root/oldbox/root and rsync everything from the PM into it:
  rsync -avH --numeric-ids --delete \
--exclude-from=root/excluded IP_of_PM:/ .
  This will take a while, depending on the amount of data your PM has.
• Once everything is copied over, the time has come to shutdown all data-writing services on your PM (mail, databases, etc). Ideally only the SSH daemon should run. This means that most of your services will be offline from here. The good thing is that this period can be really small.
• Once you made sure that nothing runs except SSH on your PM, rerun the rsync command. This time it will be quick, as only the diffs will need to be transferred. This usually involves open files from databases, logfiles, etc.
• Now create the initial device nodes needed for the kernel:
  cd /root/oldbox
mknod -m 660 /root/oldbox/dev/console c 5 1
mknod -m 660 /root/oldbox/dev/null c 1 3

• mount the proc and dev filesystem and chroot into the /root/oldbox dir:
  
mount -t proc none /root/oldbox/proc
mount -o bind /dev /root/oldbox/dev
chroot /root/oldbox

• If we now would reboot, the old initrd image would not recognize the proper modules to (unless your PM accidentally had a LSI controller). We need to add the drivers. To do this, add this to the file /etc/mkinitrd/modules (assuming your PM runs one of the 2.6.8 debian kernels):
  
mptscsih
mptbase
  And regenerate the initrd image (depending on your specific kernel version):
  mkinitrd -o /boot/initrd.img-2.6.8-4-686-smp 2.6.8-4-686-smp
  (Since my PM had an older, custom kernel (2.6.8-3-686-smp), I installed a newer one, plus udevd. During installation a new initrd image is generated automatically:
  apt-get install udev kernel-image-2.6.8-4-686-smp)
• Now we need to regenerate the bootblock. Run the grub command to enter the grub shell and see where it finds the stage1 file:
  find /boot/grub/stage1
  
  It should come up with something like (hd0,1). Use this as argument for the next command:root (hd0,1)
  Then use the hd part only for the next command:
  setup (hd0)
  Then issue quit to leave the grub shell.

By now your system is ready to boot. Leave the chroot environment (exit), unmount the dev and proc filesystem, then unmount all filesystems under /root/oldbox, issue sync, and then halt.
To avoid network clash, unplug the network cable of the PM, or shut it down.
Now you can power on your VM, it should boot a virtualized copy of your Debian system

TODO – Some things to do afterwards (vmtools, etc)