(names have been garbled to protect the names of the innocent and not-so-innocent)
A couple of days ago my buddy, Ken Pa… ummm.. Ben Kangpurn, IM’d me and said “You’re not going to believe this…” or something similar - you get the point. Anyway, to make a long story short.
Ben and his partner Mark Clarx have a client named ummm… LifespanOccurrencesPhotography.com. They’ve built, maintained, and added to the site over the last few years. It’s built on ASP, ASP.NET and SQL Server. A few months ago they were having problems where the hosting provider was “losing” files or they were getting corrupted, etc. Ben suggested that the client move the site to a local hosting provider who Ben had some interactions with and seemed to be on the up and up, WeAreProfitable.net. Ben knows some other people who host with the company and they are happy with the service. The client did move the site and everything was fine until a few days ago.
Apparently the hosting provider had some type of catastrophic failure on the server and lost everything. They were able to get everything reinstalled from backup and all seemed to be good. They sent out an email to their customers saying something about all files being up and in place, but they had lost all the SQL Server databases and anyone using one would have to re-install their database and import data. Ben was a little ticked off about the news, but thought, okay not too big of a deal we’ll just restore everything from the database backups, it will be fine. He calls WeAreProfitable.net and the support guy proceeds to tell him that they don’t have any backups of the databases. Ben couldn’t believe what he heard - all of his client’s data was gone. Not just gone as in a lost dog and hopefully we can find it one day, but gone as in never to be seen again. The “tech“ proceeds to tell him that they shouldn’t be relied on as the primary data source and that if he wanted the data backed up Ben should have done it himself. I’ve been told some crap in my life, but that one may take the cake.
As most of us know, most file-system based backup utilities will not backup the active SQL Server database file because it is in use. Apparently the hosting provider doesn’t know this, doesn’t care, or is just some big bunch of morons. In their situation at a minimum they should have setup a maintenance schedule to backup the databases at least once weekly (I prefer twice weekly) and to backup the transaction logs daily (some people prefer more). The simple file-system based backup utility that runs nightly (or at least should run nightly) would then grab those backed up databases and transaction logs. This is a very simplified scenario that really shouldn’t be adopted for business critical applications, but for most of the everyday-type stuff that I work with it is fine. With this scenario at most you can lose one days worth of data if there is a catastrophic disk failure.
So, what did old Ben do. Well, he grabbed his gat and went in and shot the place up. No, not really - he figured out that an older copy of the database was still on the previous providers server, so he grabbed that. I would assume that everything that occurred since the move will have to be re-built by hand. I haven’t talked to Ben since he initially told me the story, I assume he’s been a busy boy :).
What do we learn from this story? Find a good and reputable hosting provider. Make sure you know their backup policies, especially with relation to your database. Backup the data yourself - trust no one. Don’t talk to Ben or Mark for the next few days, they might be in a nasty mood.