Skip to main content

Offsite backups for KDE servers

Most of KDE's servers are hosted in Hetzner in Germany. We also have a storage box there where we store backups for most servers (including those outside Hetzner).

When the OVH datacenter fire happened, everyone who managed servers but didn't use OVH still got worried about their own data and backups. Including the KDE Sysadmin team; we wondered if KDE's data would be protected if a similar disaster happened at Hetzner.

We can know in which specific datacenter a server is, and even choose the DC when ordering a new server. However, it turns out we can't know or choose the datacenter for storage boxes. I asked Hetzner support and they said they don't give that information, and "that may even change" (I guess they reserve the right to transfer the storage to another DC according to internal needs).

This means it's possible that one of our servers is in the same datacenter as the backup storage, and if there is a catastrophic accident in the datacenter, as unlikely as that may be, we could lose that server and its backups.

Well, last week our storage box ran out of disk space. We could have quickly upgraded to a larger storage "in place", but we decided to take the opportunity to finally fix this point of failure.

During the weekend, we ordered a new (larger) storage box in Hetzner's separate region in Finland, transferred all 2TB(!) of data to it, and changed the scripts on the servers to store new backups there. Our backups are now ~1500km away from the production servers, and our data should be safe even if a major incident affects a whole datacenter in Germany \o/

Even if we have backups of the data, we would still have extended downtime as we set up the new servers. Reinstalling a server still involves an annoying number of manual and vaguely documented steps. Improving that is a slow ongoing task...