Hacker News new | past | comments | ask | show | jobs | submit login
Accidentally Saving the Day (jwatzman.org)
104 points by ahiknsr on May 24, 2022 | hide | past | favorite | 37 comments



I have had very similar work out in my favor early on. New engineer, some hacky idea that saves the day doing something that was "impossible" and got a bigger raise and promotion than I have gotten in any way other than saying I was leaving a job since. It was nice recognition. I have had some repeat of this recently, but not quite the same bump. It comes to be that fixing some urgent problem is way more $$ rewarding than running a team and making sure your team doesn't run into that stuff and just delivers on time. It is odd, but in a way it makes sense. You get more recognition for putting out a fire than preventing one. Everyone is watching while something is burning and recognition from someone above your direct manager is the way to get a better than nominal raise.


There's a lot of low hanging fruit out there, you just have to recognize it or be lucky enough to stumble facefirst into it.

I recently joined a budding open-source project (PhobGCC, a custom gamecube controller), saw that it was using a rather overbuilt full kalman filter, and within 2 days of work rewrote it into a less generalized filter that ran 7x faster (later improved even further) and performed even better at its filtering task.

It was super basic work, despite it having been almost a decade since I last studied control systems, but it made a really big difference for the project.


I really enjoyed this article, the style of writing, the story itself and the other article linked. "Be the change you want to see in the world."

Keep it up.


Author here -- thanks, glad you enjoyed it!


I did this at my first job. A director had a problem and no one was sure if a solution was practical. He was chatting about it in our team area and I said, oh, I can see a way to make that work. A few months later it was shipped and I basically got a promotion out of it.


> I basically got a promotion out of it

What? Your contribution to your company was actually recognized? I've never heard of that happening before. Where do you live? I'm guessing it's not the USA.


Hah, yeah, basically my thought process reading the article and the parent. Since when do people get promoted/recognized/raises for pulling a rabbit out of a magic hat? Certainly not anywhere I've worked and I've done exactly that a fair few times.


This was at Amazon.


You can be sure the manager DID tell the story like that and got a even bigger bonus. That's how you get promoted even higher up the corporate ladder


Reminds me of the last scene of LA Confidential. Paraphrasing: "you need more than one hero(es)"


> but more importantly it takes just doing fantastic work when the opportunities present themselves.

It has taken me a decade to realize this is the key to success. Just do your very best work, as often as possible, and let the rest figure itself out.

I accidentally created a data warehouse that became the necessary backbone to launch a massive new org (100M+ revenue). I was building it for a relatively small near realtime Elasticsearch cluster to generate data reports for an application. But I thought, gee, if I suck in a bunch of other data from other sources and clean it, I might find a use some day. Little did I know that someone else would piggy back on it to build a POC for a giant business expansion.


I've been stuck in the mud for a bit and had been slowly forgetting this one papercut at a time. Since this is usually not something I'm susceptible to, it has been troubling me.

Last year I pulled off a minor coup by making it cheaper and easier to run a big chunk of our code. One of the senior people was trying to get me to toot my own horn but the actual work was like one week of cleverness and five weeks of book-keeping, and so it felt too slog-like to celebrate.

But a couple weeks later when I was doing some housework it dawned on me that it was 'only' six weeks of work because of a bunch of fairly thankless things I'd invested a great deal of time in over the previous 18 months. I'd been dribbling out appetizers for a while, this was the first big dish, and I'm not sure it's the main course, so it's not like I missed my big chance, but I probably should have honored it more.

I'd had an initial 'I told you so' feeling in the heat of the moment that pushed down as feeling petty, and delivered instead a fairly lukewarm "this is what I am talking about." This is, in fact, one of the things I'm talking about, but I think I'm past trying to recruit people (all the fun people to recruit are gone). I'm sadly just following the Campsite Rule and splitting my priorities between things that help me do my job, and things that might help me do the next one.

In that respect, I've been chasing a dragon (two, in fact) for the last couple of months instead of acknowledging that paragraphs like the one above are a pretty good indication I should be focusing all of my energy on job hunting instead. Wisdom is on a continuum and there's a lot of room still for foolishness even if you've got a lot of things figured out.


Many years ago I worked at a company where I just didn't feel particularly useful or valued. I was working on a peripheral client-side Java app that was basically just used as a visual configuration for the meat of the system that was all on the back-end.

There were a bunch of more senior developers who I hugely respected and learned a lot from, including this guy who I'll refer to as $SMART_DUDE who joined at about the same time as me. He knew his stuff inside out when it came to C++ and I remember being absolutely fascinated by a talk he gave on memory management and different types of pointers one lunchtime. It's no exaggeration to say that I literally still refer to things I learned in that talk today, almost exactly twenty years later.

There were some pretty big personalities in that company and I was seriously intimidated by my line manager, who always seemed like he was pissed off with me. I spoke to him when he interviewed me, and then barely ever until my performance review came round nearly a year after I'd joined the company. I remember speaking up in a meeting once, him snapping at me, and I just shut the fuck up.

He surprised me in the performance review. We had 5 levels of bonus. I thought I was decidedly mediocre in his eyes but, out of nowhere, he gave me not the top level, but one down, which was only slightly less than the top level (we had a reasonable degree of transparency about percentages involved).

Still, I didn't think much of it and had started looking around for other roles. One of the motivators is that one or two people I really respected had or were leaving, including $SMART_DUDE, who'd handed in his notice. There was him and another guy, who I'll call, $SMART_DUDE2, who I thought - and still maintain - were just the real deal. I thought they were better than the people they worked for, way waaaaaaaay better than me, and better than everybody else in the company, although, to their credit, I think the people they worked for also though the same.

The other motivator was that, although I was earning slightly more, and on top of this the company had covered my rent for the first 12 months, I was still financially worse off than I had been where I'd lived before on the lower salary (higher cost of living in the new area).

Anyway, in the meantime I started hooking up our client with the backend using XML rather than CORBA. Not even using SOAP. I just realised there was an endpoint that we could communicate with firing XML documents back and forth according to a protocol that we agreed in a late afternoon/evening design session that would allow us to load and update whole trees of objects, rather than just individual properties (surprise, this was mind-blowingly slow), without having to make any major code changes on the back end. Really not that big a deal.

Whilst this was all going on I landed a contract. It paid something like 1.8x what I was earning after tax even with something like two months holiday factored in. It could have been more but I was so overawed by the low end of the pay band that when they asked me what I wanted I said £Y and they accepted without an argument. I was an idiot. Still, because I thought absolutely nobody would believe me, when I handed in my notice I told the company my new role would pay 1.35x my salary. I was an idiot for the second time. If I'd not been an idiot at all I could probably have got 2.2x.

It was at this point my line manager - bear in mind by this point (14 months in) I could still count the number of times we'd spoken on one hand - dropped the bombshell of, "Oh no, I mean you're really great. I think when you leave we'll feel the loss more than $SMART_DUDE. Let me see what I can do on salary." WTF?

I still maintain that $SMART_DUDE was a lot better and smarter than I was or will ever be, and I'd hire him in a flash - literally no hesitation other than to ask him if he might not get a bit bored by the work we're doing.

I still don't really get it, even now, but I suppose I delivered pretty consistently during the time I was there, didn't make a fuss (anyone who knows me now would find that very hard to believe - there's a history here that's too long and boring for HN), and solved a business problem that they had quickly and pragmatically without worrying too much about whether it was the best option from an engineering standpoint.

My advice: you don't need to be super-smart, but be just a little bit hungry, able (and stubborn enough) to deliver, and don't stand too much on engineering ceremonies.


I feel like this is a sweet spot of sorts as careers go. People hire you unproven and often have little or no real expectation. If you can deliver at all, even just by being consistent and not making a fuss then you will be rewarded. You’ve also got something to brag about now and it’s noticed by other employers. Your employer knows if they don’t pay you, someone else will. They probably also feel like you’re rather underpaid for the work you put in recently and can justify bonuses and such rather easily. Furthermore, if you leave they have a big interruption and void to fill. They have to roll the dice on another unproven or pay for someone experienced. They’re lacking leverage to keep you.

In this way, it’s not unusual to 2x or more your income in those first couple years across many different roles (even non technical). This is the reality of becoming what is often referred to as junior->senior.


At word "CORBA" the hair on my back lifted up. I hated that one with every fiber of my being 20 years ago when, on a job, I had to deal with it. I avoid it like plague after leaving there and I am so glad is a forgotten technology none uses anymore (maybe except some legacy projects, God have mercy on poor souls that have to do maintenance on those projects).


Ugh I had to do some work for the COBRA subsidy during COVID and to give you and idea how horrid this was, the frontend was an Access DB sourcing data from SAP.


Being the bearer of bad news, they're actually talking about CORBA (https://en.m.wikipedia.org/wiki/Common_Object_Request_Broker...) which is... Not universally loved.

Or are you talking about a real life example of a COBRA CORBA implementation?


Ah damn, I misread, now I do not have much experience with CORBA. I’ve toyed around with COM a deeper more than I should in these times.


I've done things like this. There's nothing like pulling a rabbit out of a hat. From a business perspective, it's... not ideal because you can't plan for this sort of thing, and an employee who has done it once might not do it again, but it's nice that when you need a rabbit and there's none to be found, it turns out you have a magician who can pull one from a hat.


I appreciate the story and the pride he must have felt. But this is the entirely wrong message. It perpetuates that the superstars are those jumping in last second in some heroic effort to save the day.

This is not how projects should work, and we should stop perpetuating these heroic stories. The actual superstars are all those people doing the less-than-glamorous grunt work. Planning things on time. Getting the infra ready, stable, well-tested, documented. Making schedules that have margins. The people carefully designing things so they are robust and don't break left and right. Thinking through their systematic testing. In short, preparing everything so that heroic acts are not necessary.

Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. Pushing all this against pressure that they are just delaying everything and should not worry so much and be more positive and agile. Imagine they had succeeded and everything had gone smoothly. This would have been much better for the company (since the existential risk wouldn't have existed in the first place). But that somebody wouldn't have ended being the superstar with the promotion and the bonus. It would likely have stuck to have been the guy just delaying things with their worries. That's what needs to change in sw eng culture.


Author here. For what it's worth, I (mostly) think you're right! IMO heroics were a huge problem at that company, and are a huge problem at the company I work at now too. The gruntwork is really really important.

I tried to get at this in the last section, that it's not about going out trying to be a hero, and it's unfortunate it didn't get across. That was my point that you shouldn't go look for company priorities to fix, to be a hero. Just look around you, see what's going on, and do good shit. Sometimes yeah that does involve heroics, sometimes it's just gruntwork.

I don't want to go back and edit the essay now but I think it could be much clearer indeed!

> Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. [...]

Disagree here, though it's maybe not obvious from the bits of the story I told why. ChaChing was one of... maybe three or four dozen experiments like it. All of the others failed. There was no reason to believe ChaChing would be any different, and if it did work, even the best-case expectations were well below what it ended up doing. It really was a lightning strike. So it would have been a huge waste of time to do all of that work, delaying Jabberwocky, for a bunch of things which never ended up shipping.


To give a different point of view, the message I got out of this was exactly what you were intending. I didn't come away with "go find opprotunities to be a hero". I found the opposite in fact, that you're saying ~don't~ go looking specifically for those heroic opprotunities. I feel that you reinforced this point in your closing statements quite well:

>It isn’t about creating bigger and bigger opportunities for yourself — it’s not about selfishly inventing self-serving projects. Rather, it’s about getting better at recognizing and taking advantage of bigger opportunities which are already there and just making things happen.


> you shouldn't go look for company priorities to fix, to be a hero

The "to be a hero" is the problem, you can always feel good about solving hard problems and finding your boundaries, but "avoiding a crisis" is a thankless job & we should thank those people more than the "only I can fix it" people.

The core problem for me is that the constructive "avoid crisis" people are baked out of these sort of events, rather than born risk averse in the first place.

> Just look around you, see what's going on, and do good shit.

One of the biggest side-effects of WFH was that this whole "who's around" mechanic has faded away without a proper replacement.

Most of my satisfying work came from random interactions like this (not the most impactful or the most purposeful).

I was a principal engineer and most of my 10 AM to 2 PM was taken up by meetings which mostly involved prevention of work duplication or to stop someone driving down to where I knew there was a dead-end (like, "I see you're using alpine docker images, why?" or more management boundary stuff like "the difference between these two ideas is ~140k over a year for all customers - can we agree that the expensive one is better for customers, because they don't see a 12$ month bill and it costs more than 140k$ to build the cheap one in NRE?").

Most of that work is tiring and always unproductive - the actual work is done by smart people who answer my questions carefully, the dumber the questions, usually the better the results.

But sometime around 3 pm, I'd be line to get a coffee and I would run into a PM or someone from support or dev who would have a question for me which sounds nonsensical at first - but having an hour & half to go down that rabbit hole looking over someone's shoulder usually ended up with mostly "hey, I know what's happening, I've seen this before" or the classic "hmm, that's strange, how could that even happen?".

Six to ten weeks later, I would be in a meeting where I would get some sense of "deja vu" when someone describes their approach and reach back to that day & that problem, to ask "how would you detect X" (something like "machine reboots, comes back with same hostname, but different IP" - would you restablish connections to IP or hostname, what about krb5 ... yada yada).

That part is the real value as an engineer, but entirely unappreciated from the management's point of view.

With WFH, I would go get a coffee, go for a run/walk and as good as it was for my state of well being, I would eventually get pulled into these crisis situations Thursday at 9 pm instead of Tuesday at 3.

I like WFH for the work I actually have to do, but this sort of temporal serendipity is not easy to recreate on Zoom.


So... why are Alpine docker images a dead end? :)


> why are Alpine docker images a dead end?

That's the most controversial thing at work too.

It comes up so often because it is a measurably good idea when you look at container sizes or startup times (gzip is terrible, terrible way to package container images).

An engineer tries it out, it works great and it does because alpine is lean, fast and generally good.

I work on performance, so any decision to make things slower gets routed through my desk - the perf buck stops at my desk. Now it's my thankless job to squash out her initiative and as gently as possible (there are lots of smart people in the industry, but being kind has a better ROI), so that it is not a "this is policy" form letter appeal to authority, but walk them through the entire list of tickets in my notes.

Because we hit a bunch of SEGVs with the JVM and the JDK team basically closes them as WONTFIX when reported.

And this sort of decision has a shelf life, I can be right for 18 months and wrong the week after - so this is not a hard line.

This specific thing is definitely not a permanent thing, but a temporary headache - the JDK team is working on Portola and that'll fix any issues they had.

https://openjdk.java.net/projects/portola/

If you're going to install glibc as a workaround then you're basically giving up the space savings anyway.

We still break things in production, but counterintuitively I'm the happier when the problem is bang in the middle of code I can commit to, rather than deep inside musl -> jdk interactions on what happens with a longjmp on a SIGSEGV.

Also the turnaround for the CVE reported was much faster across teams if everyone picked the exact same base image - standardizing on Redhat Universal Base worked out (& also AquaSec scans etc is easier if 6 different products have the same image down there).

Amazon Corretto does have alpine images, so this is sort of a 2020 advice with rapidly declining value (I'm funemployed for 2022, so my current "weakly held" opinions are entirely about rust syntax).


I think there's two stories intertwining here.

One which you've zeroed in on is "don't require heroism". We have a lot of these stories already. (Unfortunately also a lot of exciting stories of "heroism".)

The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking. IMO this is also really what "hero" stories are about: something goes so wrong that either people are finally forced to either look for once, or forced to bring in someone who will look.

As a senior engineer, for the past couple years I've tried to pitch parts of what I see "just there" intentionally, either before it becomes a crisis or just as a potential win. Sometimes people still don't see it; maybe they're stuck in "one year of experience, ten times" or sometimes they're just built out of stop energy. But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but".


> The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking.

Author here. Yes, this was exactly my point!

> But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but".

Indeed. I've mentored a bunch of otherwise-competent engineers trying to get to the next level, and my advice is similar: ask yourself "ok, now what's next". After doing the cool thing, there's inevitably half a dozen other things around it, and things might be obvious to you (as the now-expert in the cool thing) but not others.


> Author here. Yes, this was exactly my point!

Your original example shows you doing extra hours and long days to achieve this: if this was your point, the example is terrible.

I do agree with high-impact work just being there, but finding the time to do it other than "on your own time" is where the tension is between management and engineers who have a different belief of what the highest priority is.


> if this was your point, the example is terrible.

Yeah, I addressed this in another comment here. I attempted in the last section to separate the ideas of "I heroically dived in" and "I took an opportunity in front of me" -- you shouldn't go looking for heroics. This didn't quite get across, and I totally understand why. Ah well.


While this is true, it's also true that the people who are doing the less-than-glamorous grunt work are often unrecognized by management (which arguably is a management problem).

I remember being at an all-hands company meeting years ago and listening carefully to what the people who were given the top awards got them for, and about three in a row were for diving in, working nights and weekends to come up with emergency fixes to save projects or products that had gone seriously wrong, and these were different projects. It was so blatant then even some senior managers finally noticed "Hey, we have a problem".


And not to be discounted, all those prideful references to long days and late nights: another symptom of bad planning and management.

It also puts a bunch of people who have actual life commitments (like kids, health issues, family issues or whatever else) at a disadvantage simply because somebody can afford to spend the extra time on something. And if those are the people who properly plan and execute, they get a double whammy.

The moral of the story should not be that when there is an opportunity to make a significant contribution by exchanging your life for your work you do that, but that you should be able to excel at work and be recognized for it without exchanging your life for it.

Sure, the author of the post does not have anything to do with most of the management failures, except the failure to not work extra hours for a very artificial deadline (one does not need a hero to save face if you've messed up yourself — exposing the failure is usually a net win).


Firemen will always get more praise than people who change the batteries in smoke detectors.


False. People who quickly and cheaply build houses that kill everyone inside in case of small fires get all the praise, not firemen or whoever changes batteries (usually nobody)


Both of our analogies are true, but what they are analogs for is different.

In my case, the house is the business goals of the organization. In your case the house is the product made by the organization.


You are totally right. 100%. For over a decade, I basically sold myself as a hero-for-hire (or as one of my clients called me, a mercenary). My general schtick was two-fold: help the company/team out of the immediate problem they were facing, and, if they were game, spend some time with the team to help figure out how the problem came to be and how they might acquire the skills to prevent it (ideally) or solve it on their own next time.

Looking back, I am still pretty amazed at the turn-around I helped facilitate at a bunch of smaller companies.

Presently, I am an employee, and this is one of the most painful uphill battles I have ever had to fight. I have tried warning colleagues about issues that seem obvious to me, but face exceptional pushback all the time, only to have things blow up exactly how I predicted a few months (or sometimes weeks) later. In my own work, people generally let me be and do good work, but as soon as it becomes a group effort the “damn the torpedoes full speed ahead” minority steam rolls ahead and makes a mess of it.

The worst part is, I know exactly what the problem is, because I’ve been on the other side of it: organizations pay way more attention to external consultants than internal resources. 90% of the advice I have companies for how to change was suggestions gathered from their own employees, slightly curated and abridged into something easily digestible.

And so, while I’m exhausted from trying to change things from inside, when shit hits the fan, I step up and do the old mercenary thing to solve problems at the last minute and great expense. I keep trying because I do have a ton of exceptionally bright colleagues who I love working with, as well as an actually world changing technology. We’ll see… after this next demo… maybe we’ll take some time to breathe and get back on track.


[flagged]


"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."

https://news.ycombinator.com/newsguidelines.html


To clarify the ambiguity - he is white. https://twitter.com/jvwatzman

Make what you want of it...




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: