Tech Kettle: Experience the aftermath will makes you tougher, wiser and ready
That day was a very important day, as the client was expecting his own client to turn up. The client wanted to make a demo on the application and show how they manage some business processes.
During that event, I got a phone call claiming client users not able to access the web application, I took the request as usual and started checking the filed issue, few seconds later I got another call about other users not able to access their mailbox, it is then I realized that a very nasty thing is happening and I'm in serious trouble.
That day I was performing usual maintenance tasks on the server, freeing some disk space here and there. However, at some moment I deleted critical files that belong to different services like Postgres, Mysql, mail server...etc.
We lost three months of data as backups resided in a single place were I launched the deletion operation, there were no other copies of backups. Many services were surviving with what was left in RAM (I guess) and any respawned process was lethal for the corresponding service.
