Hacker News new | past | comments | ask | show | jobs | submit login

My comment I left there:

Lots of folks here are saying they should have fired the CTO or the DBA or the person who wrote the doc instead of the new dev. Let me offer a counter point. Not that it will happen here ;)

They should have run a post mortem. The idea behind it should be to understand the processes that led to a situation where this incident could happen. Gather stories, understand how things came to be.

With this information, folks can then address the issues. Maybe it shows that there is a serially incompetent individual who needs to be let go. Or maybe it shows a house of cards with each card placement making sense at the time and it is time for new, better processes and an audit of other systems.

The point being is that this is a massive learning opportunity for all those involved. The dev should not have been fired. The CTO should not have lost his shit. The DB should have regularly tested back ups. Permissions and access needs to be updated. Docs should be updated to not have sensitive information. The dev does need to contact the company to arrange surrender of the laptop. The dev should document everything just in case. The dev should have a beer with friends and relax for the weekend and get back on the job hunt next week. Later, laugh and tell of the time you destroyed prod on your first day (and what you learned from it).




The firing order, in theoretical order for preventing future problems:

1. CTO As the one in charge of the tech, allows loss of critical data. If anyone should be fired, it's the cto. And firing this guy apparently will have the greatest positive impact to the company. Assuming they can hire a better one. I think given how stupid this cto is, that should be straightforward.

2. The executives who hired the cto. People hire people similar to themselves, it seems the executives team are clueless about what kind of skills a cto should have. These people will continue fail the dev team by hiring incompetent people, or force them to work in a way that causes problem.

3. Senior devs in the team. Obviously these people did not test what they wrote. If anyone had ever dryrun the training doc, they should prevent the catastrophe. It's a must do in today's dev environment. The normal standard is to write automatic tests for every thing though.

This junior dev is the only one who should not be fired...


I'm amazed at how quickly everyone is trying to allocate blame, as if there must be someone upon whom to heap it all on. Commenters on both Reddit and HN are being high and mighty, offering wisdom that they would never have allowed this to take place, while eager to point fingers. I bet far more than half of these commenters have at one time or another worked for at least one company that had this kind of setup, and didn't immediately refuse to work on other tasks until the setup was patched. Hypocrites.

The fact is, this kind of scenario is extremely common. Most companies I have worked for have the production database accessible from the office. It's a very obvious "no no", but it's typical to see this at small to medium sized companies. The focus is on rushing every new feature under the sun, and infrastructure is only looked at if something blows up.

Nobody should have been fired. Not the developer, not the senior devs, not the sysadmins, and not the CTO. This should have been nothing more than a wake-up call to improve the infrastructure. That's it. The only blame here lies with the CTO - not for the event having taken place, but only because their immediate reaction was to throw the developer under the bus. A decent CTO would have simply said "oh shit guys, this can't happen again. please fix it". If the other executives can't understand that sometimes shit happens, and that a hammer doesn't need to be dropped on anyone, then they're not qualified to be running a business.


Well you need to consider the cto's reaction.

His reaction shows that he is the no1 to fire, and has a good reason.

What you said is true, but does not matter. The cto already show that he is clueless...


You are right that this is an opportunity to learn because it is a demonstration of incompetence at many levels. However, this incompetence has consequences that might be fatal for the company. How much time and effort will be required to level up ? As a CEO I would request an independent audit ASAP on this incident and see the real extend of the problem.

As my mother said, if you put a good apple with bad apples, it's not the bad apples that become good.


They are in no condition, yet, to run a post mortem. At this moment they're probably still trying to figure out how to get their data back or maybe just close up shop entirely.

You run a post mortem when you're back and running again. They may never be back and running again.


very fair point. My response was mostly in response to "fire the asshats."




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: