Reply to comment:

Daniel Karrenberg
One response to all comments so far:Of course back-ups should have been there. We are still investigating exactly why they were not. My suspicion is that in practice this happens much more often than is generally admitted. No information was lost, no updates were lost. We consider ourselves very professional but not infallible. If anyone with appropriate credentials and experience were to offer a free review of our processes under appropriate conditions, I would seriously consider it. What good could come from "suspending" any staff in connection with incidents like this? We would end up with less people to help investigate what happened, get less information from the people who know the most and having to do all the other work with less people too. Not to speak about what it would do to morale. That is not professional. Professional is to analyse, draw conclusions and then take appropriate action. Such action may very well include consequences for those responsible for design and operation of systems, but it is only very rarely professional to discuss those in public. There was no need to work in the weekend. We were back in business by 07:30 Friday and for all practical purposes the evening before. P ost-mortems are not weekend work. And rest assured that we were monitoring closely during the weekend. Of course we do have our processes and software documented. Sharing the detailed documentation here would neither be useful or appropriate. My choice was either to use the hand-drawn figure or to delay publication and waste resources on a "professional" version that would contain the same information. I opted for timely publication and saving resources. And you get an original Karrenberg to boot. Frame it! Personally I strongly believe in openness, learning lessons and sharing them. This culture helps to improve systems and to avoid repeating mistakes. Therefore I welcome constructive comments, even more so once the analysis is complete and we have reported our conclusions. Personally I can also deal with the not-so-constructive comments. However, should they appear to remain the majority, it will become difficult to defend and nurture a culture of openness. So thank you Nick for your remarks.