A dev ER
This Monday I came back from a long vacation, fully charged and refreshed and then come tuesday night or rather early wednesday at 1:30 AM got a call from our tech support about a customer issue, yes I make sure everyone in the company can reach me any time. Its on my internal profile and I answer my cell phone no matter what time or day it is (even when I am on vacation). Why ? Customers both internal (integration teams) and external come first and its an un-written rule that they use it at odd times when its a critical situation. So that 1:30 AM call wednesday was a critical situation for a customer. I attend a conference call right away, it was like a doctor waking up and going into an emergency room right away. System Admins throwing log files, cpu usages.. DBAs throwing in sql traces, db connection usage patterns, business users throwing in functional issues, IT and Business units pitching what they want when they want.
It took us till today 3:00 PM to diagnose, reproduce, fix, test and get the customer up and running. All this time me and my manager literally never slept (an hour or two max per day). This has to be my toughest issue I encountered in 10 years of my professional career. Not just from the technical complexity of the issue but with all the other business complexities, integration complexities etc. We had to debug our stuff, we had to co-ordinate with other development teams, support, product assurance, off shore teams, get the data together, analyze, re-analyze etc and get the customer up and running.
It really felt like we were in an Emergency Room. Now its time to get back to sleep, I sure hope I do not get the 1:30 AM call tonight.
