Linux Upgrades

Once the web server was back up and running, I started on upgrading the two remaining servers to Linux 2.6. (The web server has been on 2.6 for years). This involved first upgrading all of the other software to the latest levels. That isnt so hard under Gentoo, but the mail server is complex and was a pain to reconfigure, and the other machine is a bit older and took a while to compile everything.

The old machine doesnt need much in the way of speed, and it has been caringly maintained over the years. It runs an old 233Mhz Pentium-MMX CPU, but it did have its motherboard upgraded a few months back so that it could be added to the rack, instead of sitting next to it on the floor. I was rather surprised when I found an ATX style board for an old Socket7 CPU on ebay, so I snapped it up and moved the machine to a new 2U Spin Server rack case.

While I was doing the mail server, I was trying to clean out a bunch of old dead symlinks in /usr/lib and put together a "find" command to list them all. After I had tested it and it had located all the right files, I thought I would be clever and get it to delete them at the same time...

It was taking too long so I killed it, and found that it had somehow (I still dont know how) wiped out half of /usr, which is an important part of the filesystem hierachy on a linux system.

I copied a few key files over from another system to make the machine usable again, and then set about rebuilding the GCC compiler, without having a working compiler to start with. Once that was back up, I then basically had to reinstall every library in /usr/lib and rebuild most of the binaries on the /usr partition. It took several days to get it all back again. Every now and then, while I was rebuilding something, the compile would crash as it discovered another missing library, and I would have to stop, reinstall the library, and then continue. I wont be adding -delete to my find commands in the future.

Then I had to rebuild all the missing modules under /usr/perl5 which took a while because I was getting version mis-matches with bits that had been left behind. I eventually deleted the whole /usr/perl5 tree and started over from scratch.

As part of the upgrade, I had to migrate both systems from devfs to udev, which was fairly painless. I run raid on my root partitions, so I decided to upgrade my initial ramdisk (initrd) to support the latest version of EVMS. The Pentium-MMX loads the new initrd fine, but doesnt seem to execute linuxrc, which causes a kernel panic because it cant find the root filesystem. It works fine with the old version of the ramdisk, and the other machines have no problem with the new one. Go figure.

Eventually the Linux upgrades went smoothly, and im just cleaning up a few loose ends, such as ntp, which refused to run on any of the machines. It turned out that the problem was due to some missing compile options (Gentoo USE flags) which had been introduced since the last time ntp had been compiled here, so suddenly bits stopped compiling until I added the new flags.

Everything is now up to date, and I can get back to other things such as 3D modelling, creating web sites, and building the HTPC. I'll be buying the next lot of HTPC components on Monday.