The Cold, Hard Truth About Your Cloud DR Strategy

Disaster recovery / business continuity / “backups” are always an interesting subject for very large scale cloud environments. Many of the old data-center strategies that grumpy old sysadmins (that’s me!) relied upon don’t hold water anymore. I mentioned a couple of years ago that S3 isn’t a backup, and that’s true in isolation. AWS’s vaunted “11 9’s of durability” solely apply to disk durability math; disasters, human error, and the earth crashing into the sun aren’t accounted for in that math.

Cloud providers love to talk about their redundancy, their availability zones, and their durability numbers. But here’s what they don’t emphasize enough: most “disasters” aren’t actually externally triggered disasters – they’re mundane mistakes made by sleep-deprived humans who thought they were in the staging environment, or well-intentioned folks making a small configuration mistake that compounds when something else intersects with that error.

Let’s be honest: the chance that you’ll fat-finger something into oblivion is orders of magnitude more likely than the odds of a simultaneous failure across multiple AWS availability zones. You’ll delete the wrong object from a bucket, run that terrifying production script in the wrong terminal window, or (my personal favorite) discover that your production environment credentials somehow made it into your staging configuration. This is why privilege separation isn’t just a nice-to-have – it’s a must-have. The folks who can access your backups shouldn’t have access to production data, and vice versa. Why? Because when (not if) credentials get compromised or someone goes rogue, you don’t want them to have the keys to both your castle and your backup fortress. “Steven is trustworthy” may be well and good, but the person who exploits Steven’s laptop and steals credentials absolutely is not.

Leave a Comment