This week and the weekend before that I've had the chance to experience
the joy of migrating servers. BeWelcome has
opted for a new production server, as a) the new contract is cheaper and
b) the new server has better hardware. Based on that experience, there
are a number of conclusions to draw:
- Before you do anything that looks like migrating, make sure that
everything on the server is documented.
- Detail everything that needs to be done before starting.
- After that, make sure that everything is backed up.
- Start moving things to the new server in good time. Last minute
means errors.
- Make sure you go through all the phases: preparation, migration,
cleanup.
Getting ready for the server migration, I tried to get down details of
everything that needed to be moved. This proved VERY hard, because we
had no documentation on what was running on the server. Sure, I know
that we're running a website using an Apache webserver and utilizing PHP
and MySQL. However, what do we use to send out emails? How are we
handling normal backups? Which users are needed, which are outdated and
should be removed? Most of these things weren't documented anywhere, so
there was a whole lot of detective work to do. And while this can be
interesting, you don't want to lose data because of something as simple
as this.
There are obviously also further consequences to the lack of
documentation: which scripts should be running automatically? Those that
run now or less? Should they be running with the privileges they have
now or should they actually be restrained? While you don't necessarily
want to deal with this during a server migration, this is actually the
time when you're most likely to look closer at what's running and how -
unless you're doing an actual integrity check of the server (and how
often are you doing those? As often as you should or less?).
Another of the points that didn't exactly to plan for our move was
backing things up. I didn't give it too much thought, as the data would
be present on the old server during the migration and afterwards too.
However, we were handing the old server back to the provider, so the
data would obviously not be available for long - in fact, the data was
available for one day only after the migration was done. We didn't
manage to lose anything critical, but there were a few things we didn't
manage to get set up properly before the old settings were gone. This
relates to the point on documentation: there were some things that were
not documented properly and so we didn't know they were needed on the
new server. Hence, when we found out, it was too late to go back and
check.
Now, one way of handling backup would of course be to just dd the entire
disk and be done with it. Which is a good idea if you have the space for
it and can transfer it off your server in good time. Even easier, if
you're just migrating servers and will be mirroring the old setup, then
you can just copy the old server and stick it on the new. In our case,
though, we were moving from Debian 4 to 5. Doing a copy of the system
wasn't an option, and to make sure that apps were working properly, we
needed to get them set up first. Exactly what do you backup then? On a
Linux setup, probably /etc, /home, some of /var ... and again, you see
the problem of lacking documentation. Guesswork isn't really good enough
here.
Because we weren't done with the migration till a day before handing the
old server back, there wasn't a lot of time for cleaning up either. This
wasn't too big a problem with regards to the old server: files were
deleted, the system was reinstalled a couple of times, nothing should
remain unless you're either reading the sectors manually or going for
the magnetic trace of things - neither of which are too likely. With
regard to the new server, there is more cleaning up to do: because of
the lacking documentation, some things were setup in less than perfect
ways and some data was copied over without ending in the right places.
In other words, there's quite a bit of cleanup to do. And because the
migration took place too late, there's no original setup to look at for
comparison.
For next time
One thing is for sure - I've definitely learned my lesson from this
experience. I'm currently busy writing up documentation on what we're
running, where everything is, how to go about setting it up, etc. It
should result in two things:
- Proper documentation of what we use and what we have
- Proper documentation on how to move it
Together, the two should allow for a fairly easy time when handling the
next migration. Not sure I want to be part of that, but at the very
least I should make sure anyone taking up the job after me has an easier
time. Chances are, though, that I'll be needing the documentation myself
before another migration given the daily maintenance tasks ...