This is what I don't understand. When Tbray left, he said he was doing so in solidarity with the distribution center types, but he held to the line that AWS was great. This despite nightmare stories like this one regularly bubbling up, and I've heard similar from friends.
What is it about tech culture that people within it refuse to admit the level of abuse-by-design that happens at these companies? Is it that they want to believe that they are somehow fundamentally different from the distribution and grocery store workers, and fundamentally similar to Bezos? I can't help but feel it's just this kind of false consciousness that prevents developers from _getting organized_.
Sidenote, payroll tax on Amazon in Seattle trying to get revived. It's a long and winding story. Check out taxamazon.net
It’s worth pointing out why it needs to be “revived”: because the first iteration of the tax would have destroyed grocery stores and was protested by construction unions due to its disastrous policy.
This iteration is targeting the 800 largest employers in Seattle, many of whom aren’t currently profitable.
“Tax Amazon” is classic dishonesty — neither bill has been focused on Amazon, both would be a disaster for Seattle, and proponents try to focus on Amazon to keep people from noticing how damaging their out of control spending is.
In 2010, the Seattle budget was 3.8B; in 2020, it’s 6.5B. An increase of 2.7B and growth of 71% in a decade. (Per capita, the Seattle budget is 37% more now than in 2010.)
Seattle population has grown by around 23% [1] and inflation means $1 in 2010 has the spending power of $1.18 today [2], you'd expect a 45% increase (1.18*1.23) just accounting for population change. According to seattle's open government website, their budget was $6.24Bn in 2020, and $4.13Bn in 2010, a 51% increase. So just ball park spending has grown probably less than 0.5% per annum over inflation in the last decade.
As someone not from Seattle, the drama around that tax was quite ridiculous. The idea of a "head tax" is utterly nonsensical in that it's an incentive to NOT hire people, which is the entire point that companies are pulled into cities. Jobs. Jobs are one of the things cities care most about more than anything. A head tax actively disincentives hiring of new workers. A head tax can maybe make sense if you have permanently low unemployment and companies can't hire enough people, but that's not Seattle's problem.
Given how many people are moving to Seattle every year, far faster than new housing can be built, I'd say that an effectively negative unemployment rate has, in fact, been one of Seattle's problems, and that slowing hiring down would, in fact, help things.
> Jobs are one of the things cities care most about more than anything.
And look where that got cities.
One school of thought is "just keep importing jobs AND add a ton of new housing" but there's a sort of capitalistic dystopian feeling to that, like that's the path to Blade Runner. Just need to start shipping out the people without high-enough paying jobs (again, actually - Europeans moving to the Americas doesn't seem that dissimilar) and we're there!
That's from 2017 when it was introduced but it's a good non-partisan explanation, including the many problems with the proposal. https://sccinsight.com/tag/head-tax/ has newer posts.
Must be why all those small businesses that got affected (like the long-running local Bartell Drugs chain [0]) were protesting on the streets against this tax once it was implemented (shortly before it was repealed).
Writing off the opposition as solely "pro-amazon shills" is exactly what gets those bs laws passed in the first place. I wonder how much money was wasted on all of this, given that it was repealed shortly after. Seems to be very in-line with how wastefully Seattle City Council has been running things.
They increased about .5% inflation adjust per year from 2010-2020. That again does not mean that spending is out of control especially when you consider that 2010 was still in the depths of the great recession.
> What is it about tech culture that people within it refuse to admit the level of abuse-by-design that happens at these companies?
The Amazon example is extreme, but a lot of tech workers fit the stereotype of people who don't have a lot else to do.
I spend 100+ hours at my computer a week, so if you want long hours, I can live with that.
I am up at 1AM anyway, so if you need something done at that time, sure. I just need to switch to my remote desktop.
I have colleagues and friends who will log in on weekends or evenings out of boredom to do work. No pressure to do it, but they just want something to do.
I fill my time with internet commenting and various online contests, but if they need help at 9PM, I am fine with jumping on the VPN to help and have done that a few times myself.
Amazon still has problems beyond the long hours and the 3AM calls, but there are people ok with those things or at least ok enough with them to not put up a fuss.
Maybe it's just the type of people amazon hires? I've been programming for 5+ years. Know what I'm doing today at 5 pm? Headed over to Lake Michigan with the fiance. Yesterday I went on a 4 mile run. Personally I never check things outside of 8-5 and my employer is A-OK with that.
The biggest problem is when people like this article's author get a job and the HR people never make it clear that they may need to have "pager duty" or otherwise work outside of 8-5. Those different expectations is what caused him to have a very bad experience.
If the company thinks its applications are important enough that it needs people around 24/7 to work on them then the company should hire enough people to have 3 shifts a day.
Is it abusive if you willingly sign up for it? Doctors are on-call for emergencies. We compensate for this so I'm not clear on why its abusive if employees willingly sign up. If you don't want to be on-call, find another job.
I think if you are compensated for it, and allowed to recuperate from middle-of-the-night working, it's ok. If the employee knew this was part of the job though, and not something that was thrust upon them after they were hired.
One thing seems to have changed since the OP wrote their post -- managers at AWS are open about the on-call requirement. At least the ones in Portland. They were proud that they owned the product all the way from the code on the back to the customer at the front. So if you didn't want to be responsible for some level of ops, you knew that going in.
I didn't move forward to an interview, this was just from a meet-and-greet they did a few months ago (they do it every month IIRC). I have other reasons for not being at all interested in working for Amazon, but I figured it would be interesting to meet with them informally anyway.
> They were proud that they owned the product all the way from the code on the back to the customer at the front. So if you didn't want to be responsible for some level of ops, you knew that going in.
There's a huge difference between being responsible for (some level of) ops and literally climbing out of bed when the pager rings. At work, I'm responsible for the operations of the services I develop, which means that I'm doing third level support during normal business hours and I write the playbooks that first/second level support executes. You certainly won't catch me working after 5 PM.
I worked for a company where developers could opt in to on-call cover. Doing so meant you got some extra cash, and if you were called then you got so much for the first hour, then a different amount for each hour after that. Not being coy, I can’t remember the details.
I'm a pretty heavy sleeper, and would go to bed with my phone on silent, set to ring a long time before going to answerphone, and with my Apple Watch on my wrist. If my phone rang then my watch would buzz me awake, eventually. The only snag was if, in my sleep, I knocked the watch out of contact enough that it locked and wouldn’t buzz on calls, but in practice I was lucky enough that I never got a call when that happened.
Not that I was called often. The company had developers around the world so it was only if they needed some specific knowledge or skills. That meant that being on-call was mostly a fee to compensate for being available at short notice and not drinking, etc.
Ultimately they decided they weren’t getting enough calls to justify the cost of paying the on-call fees and scrapped the system, just relying upon whichever developers around the world were working.
They didn't for me - it meant my SO would wake me up after they'd been woken up.
I then started wearing headphones to bed so that I would hear the phone ring and not disturb them, but when you're rolling out of bed and hopping on a call at 3AM it's difficult.
What I seriously underestimated was the strain that being on-call can put on a relationship, it's a massive regret.
Personal attacks and name-calling are bannable offenses here. Would you mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting to HN? We'd be grateful.
yeah no. as someone who has worked at Amazon for 7 years I can tell you that it’s not true across the board and there are good “pockets” where there is respect for people’s boundaries.
To add to this, in my time there I worked during the weekend 3 times (i don’t count oncall here). My work day included 8 hours at most. I would get in at around 8AM and leave at 4PM. (optimizing the commute)
I started as an SDE 2 by the time I left I was a Principal SDE.
Do good work, be in a good org.
Amazon is not a huge, homogeneous entity. It’s more like hundreds of small companies that are forced to work together, bound together by senior leadership.
I'm trying to interpret your words as something other than "well, yeah, my weekends were interrupted, but it wasn't my choice so it's not worth mentioning." I can't come up with another way to read it, though.
So, don't you think that, in the context of having to give up nights and weekends, you should mention on-call? Your choice or not, the weekends and nights are affected, in fact isn't it kind of worse when it's not your choice?
i agree with you that the oncall is not ideal. it is also widespread in the industry. so what’s the point to complain about something that is an essential part of the job? (advertised or not, if you don’t want to do it you will have to look for another job)
i also think there is a huge difference between being oncall and having to respond in case something breaks vs mandatory death marches all day, every day.
I guess it depends on your definition of widespread. It is relatively common, but it's also far from universal, and there are companies that use on call which have a vastly different experience that what many people at Amazon report. In fact it seems like even different teams inside of Amazon have drastically different experience of on call. Treating it like a fact of life, like the weather, leaves you in a frame of mind that makes it impossible to advocate for change. Your own phrasing "be the change you want to see in the world" could as easily apply to on call as much as it does anything else -- but you won't take that stance if you think it's inescapable.
hmm. no. you do have agency. you have agency during writing the code. you have agency during the code review. you have agency during wiring of alerts (if you do stupid shit you will receive a lot of pages. don’t so stupid shit). you have agency during the events. you also have agency after the events when you do a coe/post mortem. you can defined prioritize things to improve the life quality of other developers.
to give you an example. the team i was in at amazon had 3000 tickets in the queue when i started. anything except sev2s were basically ignored. lower severity tickets would escalate when shit hit the fan. i advocated for fixing classes of issues instead of myopically focusing on one-offs. by the time i left the queue was tens tickets and mostly feature request or higher level investigations.
to give you another example: i would basically remove all alerting that was not actionable. the worst possible thing that you can do is wake up in the middle of the night and not be able to do anything. i would ask for runbooks and the test was “if i take a developer from another team and put them oncall can they function independently 95% of the time”. i would think about what the experience of being oncall was (ie you don’t take people and throw them in the deep end of the pool and wonder why they drown)
so i guess what i’m saying is that oncall for me wasn’t that bad or stressful. it sucks having to be near a computer but I was rarely paged for stuff that broke or needed to be fixed right NOW. (once stabilized our team had 1 sev2 every other week)
Okay, at the risk of putting way too much effort into this, I'm going to assume you have 5 people on your team, therefore you're on call 10 times a year roughly. 5 weeks out of the year then you'd be oncall during a sev2. Assuming sev2's are uniformly distributed you are probably interrupted outside normal working hours ~80% of the time, so call it 4 times a year you have your work-life balance negatively affected by on-call.
you need to generate some 3d graphs and after that write a paper about using gradient descent to improve the WLB of SW developers :)
our team had 7 people and at some point we started sharing the oncall load with a sister team (ie you were oncall for the services on both teams), meaning that you would be oncall roughly once every 3 months. not ideal - but not the worst thing in the world.
I am sure it is not all of Amazon, but that is where the horror stories are coming from. So Amazon as a whole may not have this culture, but the teams with this culture seem to disproportionately be at Amazon.
so, the bigger an org, the higher the likelihood of stories like this popping up.
also, the incentives and their structure play a part in how this unfolds.
It’s also very curious to me that only a certain type of person runs into situations like these: either they don’t know better or they have accepted the conditioning that work is more important than themselves. it’s sad really.
i think it’s really really important to have a good mentor with a strong set of values to help you navigate the corporate landscape.
i also think it’s important to pull a Ghandi (ie be the change you want to see) as often as possible. for example: one think i would inoculate in young devs is that it’s not acceptable to set meetings earlier than 9am or later than 4pm. meetings during lunch were not accepted. i would be pretty vocal about morons that worked over the weekend because they had some free time. i would also push really hard against management that had their sick little practices to extract more work out of the drones. this mostly worked because i got the job done - but again: you have a voice. you are NOT a machine
law of numbers; Amazon Seattle headcount alone is 50K+ people. also, amazon has a service-oriented management model to match the general SoA architecture, where small teams are given massive leeway as to how exactly they run their services. managers and your leadership are thus very make or break.
this is very informative. I wouldn't have believed it, without the first person perspective. Before i read this, it was simply unbelievable to me that people could be that bored without any life outside work. Speaking as someone who really values free time and has tons of hobbies and things to do outside of work, it's just unbelievable.
Most of the people I know are fairly new graduates, so that would skew things.
My friends and I spent most of our lives since grade 9 doing school stuff or stuff to get into school stuff. In my case, couple that with introversion and an aversion to most normal activities and the easiest thing to do is be good at the get into school stuff.
In university, most of my social activities revolved around the startup world, hackathons, or other competition-oriented activities as that is what I was good at. I went to one hackathon a weekend in my final academic year.
So I became good at a lot of these types of things which all revolved around the academic world.
I am not in university anymore, but I can still fill my time with the kinds of competitions and challenges in that world. If I couldn't, I doubt that I would know how to use my time. Even with those, I spend a ton of time commenting online.
Most of the people I know are in the same kind of boat. They were good at one or two things (might be science fair, research, conference organizing, etc) back in university which consumed their time. Then they left and are now left wondering, "now what?"
Thank you for sharing. I can hear a shy cry for help. I think it's fine to spend a lot of time with computer, in general, I do it myself. There is so much to do, learn and help others. On the other hand, I think it is good to realize when it eventually becomes a replacement activity for lack of motivation/opportunities to do something harder. Then try to break that, take a challenge, do something new, either with computer (a personal project to make something cool?) or better, without it - for example, try some manual work with wood, do some science project with a friend, or get a pet and teach him tricks or something. Just don't spend yourself fully on the online/academic life, that sounds too ephemeral and can be a bit disappointing later on. I for sure regret spending so much time in academia.
If you can't sleep at 3AM and would like to, plan to do anything else afternoon than staring at the screen - like, try to walk and find a path to a moderately distant place in your city you know/like but have no idea how to get there on feet, without using a phone unless you really get lost. This can be fun and long walk and navigation will tire you down. You can think about what you're gonna do next time too.
Like taking up some challenges in life. Besides maintaining physical fitness and getting tired in the process, one thing I like is to create contraptions (does not have to be a "product") that could be useful to you or people you know. An air purifier, a charging station for a phone, a piece of wooden furniture, or anything else. Get some tools and get working.
If you prefer social activity, plan to do the above with a friend.
You move across the country to a city you don’t know, you end up there with no social skills and no real entry point it the social scene so you binge watch Netflix enroll in a masters degree and stare out of your condo window.
Then you ssh in and work to fill the hole. Now I don’t even get to see my co-workers.
> Very sad when you think about it. Wasted potential.
Eh, you leave the university environment of all its clubs, hackathons, etc, likely to another city where you know far fewer people. You need to wake up the next morning, so gaming until 5AM with your friends no longer works. Someone is also on on-call, so they can't join, so you postpose it further.
I do believe there are people out there for whom "living their best life" truly does entail spending almost all their time in front of a computer.
Nobody shamed Chopin for spending his life in front of the piano. Some people choose to dedicate their lives to one single thing, and often those are the people who end up contributing the most.
That's not the comparison as far as I understand it. The analogy is of someone that dedicates all their time to becoming the best at their craft, like Chopin. It's not about being a genius.
Every company is full of nightmares, especially anyone doing something big enough to be interesting. For instance that guy who started blogging about how janky Tesla's software was, most of us who have been around a while read all that and shrugged. Not that bad. It is easy to imagine perfection or competence but very hard to actually achieve it at large scales with humans.
So at the same time it can be true that company X is a great place and also true that company X is a nightmare.
This. Amazon maybe has thousands of developers working in various departments. The departments can be totally different from each other, have different culture, values whatever. Even in very small companies I have seen this effect.
The divide between AWS employees and everyone else employed at Amazon seems to be "2007 Google employee" against "sweatshop"; they're run mostly-separately, from what I've read.
This was also four years ago; the gist is a copy of a post from 2015, by the above reddit poster.
> The divide between AWS employees and everyone else employed at Amazon seems to be "2007 Google employee" against "sweatshop"; they're run mostly-separately, from what I've read.
This is not true. Amazon and AWS are exactly the same organizationally speaking. You could hardly tell who works in an Amazon team and who works in an AWS team. Maybe if they tell you their building.
The cultural drift happens in the teams and orgs, not the parent "company". Everyone at Amazon knows this and that’s why many people know changing teams is a gamble and prefer staying at the same team doing the same menial tasks, than venturing into finding another team.
Amazon and AWS have both great teams and shitty teams. There’s just not an easy way to tell, except for one internal tool that allows you to see how people are rating their managers but this is all bs because some teams don’t have enough data or any data at all. Also the data itself is probably skewed, since people are afraid of giving poor rates to their managers (this is a general problem in the modern workplace dynamics, not at an Amazon problem)
The other way is asking a friend but I don’t know how people do that since I only knew people from my team and teams that sat nearby. Maybe I’m too socially incompetent.
> The divide between AWS employees and everyone else employed at Amazon seems to be "2007 Google employee" against "sweatshop"; they're run mostly-separately, from what I've read.
This is incredibly offensive. I work in CDO and have friends in retail and AWS and the biggest difference day-to-day is that we have different systems to manage our AWS accounts.
Equivocating me to a Wipro employee is honestly really disgusting. These sorts of microaggressions are the worst part of working at Amazon, the WLB is fine.
If psychologists surveyed one of these companies, like Amazon, to look at the sorts of attachements the employees have now, and had as a child, I think you will find distinct deficit of people who had a happy childhoods with secure attachments sticking around at these companies.
The bravado and masochism so common in such places, in my opinion, is a form of self-bullying. You don't have to look hard to find managers exploiting this characteristic, and I suspect the most of the ones who thrive in such an environment do it. I worked a brief contract there and it was clear that the manager was used to making statements designed to trigger his subordinates' insecurities.
When your self worth is tied to the quality of your work, but your ability to self-evaluate is not solid. you are vulnerable to manipulation. I have that trigger too, it's just harder to hit, especially by some punk who is used to saying 'frog' and having people jump. I felt sometimes like he was waiting for me to react to some subtly passive-aggressive statement he just made and I just shrugged it off and stopped talking or kept going.
A master can be very tied to the quality of their work, and hand you your ass if you try to disagree with them. This doesn't develop overnight, and it's a winding road, but it's very undeveloped in a bunch of 25 year olds with a toxic cocktail of Impostor and Dunning-Kruger, and any 28 year old who spent more time honing their interpersonal skills in college has some ideas how to get what they want out of such people.
And those people think this is okay. Normal. That it's only what they deserve. And then we have a culture of avoiding psychotherapists, who would tell you two sessions in that this is bullshit and you shouldn't have to live this way.
I worked at AWS. Your experience may vary significantly depending on which team you are in, either at AWS or plain-Amazon. Some people are having the time of their lives there. Some are working horrible hours, maintaining crap old code that they hate, and struggling with performance reviews that saps their morale and prevents them from seeking a better team or leaving the company. Some people genuinely like the competitiveness, the politics, and know the Leadership Principles like it's the law.
Amazon is a fascinating company, it's Big Capitalism in a way no other company is today, and as such it's writing History, for better or worse. For the average developer it's not great, though.
Tbray is going to have a different experience at any org he joins.
He's not coming in as a no name peon. He's financially well off, internet famous and would (and could) probably leave at the first sign of mistreatment.
He's in Canada, do health insurance isn't tied to job.
>Is it that they want to believe that they are somehow fundamentally different from the distribution and grocery store workers, and fundamentally similar to Bezos?
I think this might be a big part of it. The idea that "I'm college educated and don't perform physical labor means I'm not like them" seems to be a pervasive societal thought not even strictly related to tech.
Perhaps the first thing they should arrange is to have lunch breaks together with the low-wage work-force, i.e. in the same physical space. Or perhaps a company outing. That could result in some awareness and perhaps solidarity.
If these developers don't ever see or talk to the people who are suffering from the company strategy, then the problem may remain to appear imaginary or abstract.
It's good to be on top. The reason AWS is so great for those with voices whom we pay attention to, is because of the misery of a whole lot of people below them.
It's not even programmers like OP but also all the servicepeople and staff at datacenters, and the rest of the crew which makes up the infrastructure that makes it work.
Payroll taxes are always borne by the worker. They may not make the payments, but when I hire people, I set aside an all-in budget, and their salary is whatever is left after benefits and taxes. If the payroll tax goes up, it means their raise is smaller.
That's how every business works. All per employee costs are borne by the employee one way or another.
This is only true in the sense that there is a cost to move goods, but it is not true in the sense that retailers charge more to move the good to you. If it were, I wouldn't be able to reliably purchase goods shipped to my door for equal or less than what I pay in physical stores. When shipping is included in the price, that means that the shipper bears the cost of shipping by taking a smaller margin. If I were to bear the cost, it would appear as an explicit cost or higher prices, neither of which are happening.
By the same argument, customers are paying the costs for Best Buy's internal logistics and retail floor operations to enable “free browse and instant pickup”. If those costs are higher than Amazon/NewEgg's costs for "free shipping", I'd expect to see exactly the relative prices that you're seeing.
I'm sorry, but I can't track the point you are trying to make.
I'll reiterate my point that shipping is a cost that has to be paid, but who bears that cost is not necessarily the customer. If it was the case that the customer was the one bearing that cost, there would be a difference in the price you pay for goods at a physical store vs. goods shipped to your home, implicitly or explicitly. There is generally no difference, and in many cases, ship to home is cheaper because you can select from more retailers.
This is not evidence that payroll taxes are borne solely by the worker. I'm not claiming that workers don't bear any of the burden. Just that they don't bear all of the burden. this would be especially true in an asymmetrical payroll tax system in competitive labor markets like the Seattle Software Developer market.
I'm not sure how a competitive labor market makes any difference. They all have to pay the same payroll taxes. If anything it's a great equalizer -- all companies must bear that employee cost. So they will all take it out of their employee's earnings equally.
In this context, we are talking about the Seattle Head tax, which is NOT applied equally.
In the larger discussion of payroll taxes, to disprove that workers bear the entire burden of payroll taxes, just go through the thought experiment where we eliminate the employer side of the SS tax and whether wages would rise 6.2%. They wouldn't. Workers were willing to work for $N previously, there is no reason that they would stop working/change jobs unless they were paid $N + 6.2%.
Of course salaries wouldn't go up. Because you removed the cost from the equation.
But now do the thought experiment where the employee must explicitly pay the extra 6.2%. Now of course you have to give them a 6.2% raise, because they would demand it to cover their increased costs.
Numbers: I pay the worker $10000 a month. Let's say my tax cost is $500 a month, and their tax cost is $500/mo. So they get a check for $9500, they see a $500 tax deduction, and they net $9000.
My all in cost is $10000.
Now the city says "you don't have to pay your part of the tax anymore, the worker does!"
So now I pay them $10000, they see a deduction of $1000 on their paycheck, and net $9000. My all in cost is still $10000, their net is still $9000. Nothing has changed.
I'm still paying $10000 to have that worker. And if my budget is 100000, I can still only afford 10 workers.
The loaner thing is so true - having two bosses is THE PITS! That and not sleeping.
My first job out of college I had two bosses. HORRIBLE.
This came up recently, a manager, trying to be helpful, not sure they could manage someone said why don't we both manage them. NO!
The other things - trading sleep and shutting off friends - the first will destroy your quality of life and the second you will regret for a LONG time no matter how critical the day to day was.
I'm not perfect. Setting boundries. I focus on situations in my case - what situation lead to bad results. Also - you can sometimes trade time for reduced money - which in some of these situations is a NO BRAINER. I'm trying to do that. Society really IS NOT setup to support that.
I go home even if I'm behind. I sleep next to my wife and play with my kid. Covid has made catching back up with friends harder.
I think what folks from the outside don't realize is that folks making big money+ - you put a lot of pressure ON YOURSELF. You are given lots of money, often lots of responsibility, your work impacts lots of people (internally and externally). This is even if Jeff Bezos is not your boss.
My favorite example of having two bosses was as an intern. I was getting loaned out to a team that thought I was working full-time when I was really working 20 hours per week. I wasn't aware of this until I started getting super negative feedback from the manager. She basically thought I was the worst developer alive haha - then when I clarified I was a part-time intern and full-time student she totally lost it on my og manager. Of course he then was furious at me for blowing his scheme... not sure exactly how the internal budget/trading hours/money worked, but they were offloading 40 hours of full-time budget for my part-time efforts.
I appreciate this being shared, but I have some qualms.
I think talking about the Leadership Principles as a 'typical corporate motto' is unfair. Amazon lives their leadership principles far more than the other companies I've seen. They don't include things they don't actually care about, like most corporate mottos do (you'll notice the lack of 'we care about our employees' or 'do no evil' LPs).
At least in my experience, you only need to pay back Amazon proportional to (2 years minus duration employed) so it really isn't that bad (maybe that wasn't the case when this was written).
Maybe it's changed, but the signing bonuses I saw were essentially increased salary until your stock starts vesting seriously. It gets paid out and vests every paycheck so there isn't anything you need to pay back.
My signing bonus was structured in 13 separate payments, one attached to my first paycheck roughly equal to half the total bonus, and the other half distributed evenly over each paycheck in my second year of employment. If I had quit in my first year, I would have had to pay back the lump sum payment proportional to how long I had been at the company. The "bonus" paid out over the course of the second year never needs to be paid back under any circumstances.
Relocation bonus would have had to have been repaid in full if I had left during my first year.
From the perspective of not taking advantage of the financial irresponsibility of recent grads, this is an improvement over the structure described in TFA, but the right solution is make sure that recent grads know what compensation agreement they're signing and can resist the urge to spend it all, that can't be left to employers, whose every incentives push against that. Big lump sum payment right out of college are very useful as emergency slush funds even if you can't responsibly do much with them for a few months.
Not sure anyone will see this since the post is rather old now and off the front page, but just to add my two cents:
I was made an offer for L5 at Amazon earlier this year. For compensation, it's like you suggested: the offer is structured such that the first two years weigh more heavily toward cash (a separate signing bonus for each of the first two years, which IMO was pretty good and _does not_ need to be paid back, it's just combined with your normal bi-weekly paychecks) and then the latter two years are where you get the majority of your RSU rewards, which wind up being worth the same or more than the cash bonuses of the first two years (assuming I suppose that the stock price continues to be stable and/or increase from what it was at time of signing).
A note about the signing bonuses, the offer letter details basically do not say anything about needing to pay them back for leaving early. If your employment is terminated (by yourself or Amazon itself), then you just end with the prorated portion for the current pay cycle and that's it, you don't get any more of it (fairly obvious anyway if you leave a place).
I can confirm that unlike most other companies' "mission" the Leadership Principles are a real thing(tm) that even those of us who are skeptical about corporate yadda yadda were caught up using and being influenced by.
Two things about it: it's not feel-good bullshit like the parent said. That makes it real. And another: so many principles are at direct odds. Do you "Insist on the highest standards" and delay the launch or do you "Deliver results" now? How do you "Think Big" while also obeying the mandate for "Frugality"? There is a lot of tension in the principles which matches real-world choices. It's not the shallow corporatese of most mission statements.
You're right. The proportional compensation is something I was very wrong about.
At the time, I was also in debt due to being straight out of college and having been dumb AF with money (student loans, car, & credit cards) and so owing anything would've been bankrupting.
It was so easy to be stupid with money when you're making an engineer's salary.
Not shifting blame, I did that debt to myself and I dug myself out of it after educating myself.
How so? Google is facing employee issues and protests because they sold the 'do no evil' ideal, but the employees don't think some of their projects follow that. I think you would struggle to find an equivalent portion of Amazon employees who think Amazon isn't customer obsessed.
> I mean in the sense that being customer obsessed should mean you do not evil
I think there is a real tension between customer obsession and 'do no evil'. Serving the individual needs of consumers can come at the expense of the public good. I think example of that are the amount of waste that Amazon packages generate by shipping as fast as possible and the amount of environmental damage AWS can quietly cause via their datacenters.
I think it depends on the size of the signing bonus. Smaller ones are paid upfront as a lump sum, larger ones are paid out over time. The latter of course is just an abuse of language by Amazon to disguise the fact that they're cutting your salary in year 3/4 in exchange for stock.
When my Uncle found out how much the average entry-level Amazon software developer at Amazon makes, he was completely shocked, because it's higher than his salary as a physician in practice for many years. So all things considered, when it's mentioned how doctors are the only other profession where the employees are on-call 24/7, most FANG devs are making more than the average doctor.
$3000 for 1bd in NYC is pretty common, it can extend to >= $4000 for 5-10 minute walking distance to work. Tech salary can seem much less impressive servicing that kind of recurring cost.
Seriously, my dad made $150k as a physician... in suburban, central Virginia (not NoVA). This would be what, $400k in the Bay Area? We lived on 6 acres with a barn in a 3 story house with a pool, surrounded by forest.
Yep, it's crazy when doctors like this look at tech and they even complain that we're overpaid.
You can make a ton of money as a doctor nearly anywhere in the United States. Software engineers don't get that freedom. Maybe now with more WFH being common place.
However, now to get to the point of being a physical you need to go under a ton of debt. That wasn't the case for our parents and grandparents.
The original OP's write-up misses the main questions and helpful perspective. Doctors (we have several in the extended family), lawyers (worked with some), entrepreneurs (have been one) all put in very long hours. So do grad students, and undergrad's working on tough assignments.
It's not work per se; the issues are larger, and different:
- do you enjoy what you do? If you're new, out of school you may not know this without experience. This guy got experience. We are aware that plenty of people make career changes inside and outside their industry right? Because of they realized the enjoyment and fulfillment wasn't there, right?
- do you have friends, hobbies, a social life, a wife (husband) and family? Do you do anything with them? If not, why not? Why does one work all the time?
- do you have self awareness of your red lines? If no, why not? If yes, why were they repeatedly crossed? Individuals have choice, agency if they are relatively self-aware.
- Why would a team put up with so many bugs, and so many late calls? There are a zillion developer teams ... we all have support issues/challenges, and part of the job is reducing outages which feeds into software engineering, quality, management, cross functional management with business people providing requirements, corporate culture, and many other facets.
I didn't find this article helpful. I do not cherish or dismiss the trauma reported herein, but something bad X happened at company Y is neither new, insightful, or explanative in any ultimate way that counts.
> Remember when I mentioned Amazon's moving and signing bonus? Little caveat on that, if you leave or are fired within 2 years, you have to pay it back.
Paying back a signing bonus if they fire you seems super rough. Is this common elsewhere? 2 years also seems high - the time I got a signing bonus, it was "pay back if you leave voluntarily in 1 year" which was much nicer than the described.
Unfortunately, much of this seems pretty common, especially months-of-crunch-time. Being able to figure out projects to avoid (if you're at a big corp like this where such things are possible), and being able to avoid them is a very good career skill to cultivate.
On-call can work quite well if the team has sufficient ownership to improve the underlying issues, not just keep the lights on. I've done this with good success. I'm often disappointed by how many managers and companies I've seen who don't get that aspect.
Believing too much in mantras like "adding manpower to a late software project makes it later" can be dangerous, too, though. How well it works depends on the manager and the person scoping and planning the work. Another critical skill to develop to escape being the person doing the grunt work.
> Paying back a signing bonus if they fire you seems super rough. Is this common elsewhere? 2 years also seems high - the time I got a signing bonus, it was "pay back if you leave voluntarily in 1 year" which was much nicer than the described.
I actually don't think this is true. My understanding is if you're PIPed they don't really care. The signing bonus is paid out monthly the 2nd year (and for people hired at SDE2+ its first year too) so you don't have to pay it back after the first year.
My experience with Amazon is that my signing bonus was in two parts:
Part one is in your first year where you get a lump sum signing bonus with your first paycheck (paid monthly) with the understanding that if you leave you would pay back a pro-rated amount proportional to how many months were left in the year.
Part two is in your second year and is paid partially with every pay check and if you leave you don’t pay anything back, you just don’t get the remainder of the money to be paid out the rest of the year.
After year two stock vests tend to vastly outpace the signing bonus pay but I’d imagine this varies on the stock price.
I fear OP may have misunderstood how the signing bonus works and it seems to have caused him undue stress.
I've interviewed at Amazon and had a lot of conversations about compensation. I don't know about the terms of the relocation bonus, but for the signing bonus, you don't pay it back if you are fired and even if you leave voluntarily, you only pay back a pro-rated amount. For example if your signing bonus is $20k, and you leave at the 6 months mark, you owe $10k back but you get to keep the other $10k.
I'm surprised at how many people in this thread are misunderstanding this. Every tech company I've seen has had a stipulation saying you have to pay back signing bonus or relocation if you leave within 1 year.
Yea, that's something else I forgot to mention. I chalk it up to OP possibly being inexperienced/naive (I think this was his first job out of college), but I have worked with a lot of companies and all of them require you to pay back at least part of your signing bonus if you quit within a year.
When I graduated college, all of my friend group discussed it and we all took out a small chunk of the bonus to go on a small group vacation, but the rest of the bonus got put into an investment account and not touched for at least a year. That way if we did decide to leave, the money was easily accessible to be paid back. I know not everyone has that flexibility, but it's probably the best way to approach signing bonuses if you're not 100% sure you will be at the company long term.
I've heard that companies rarely enforce these clauses unless you really burned a bridge. If anything they won't give you severance if you quit within the first year.
Kinda unrelated to topic at hand but I always wanted to ask this to other programmers:
Author mentions that he traded sleep for code, and I hear this from so many people but I really cannot understand how well they manage to pull this off. I simply cannot code productively if I get less than 7 hours of sleep or code for more than 8 hours straight, and caffeine doesn’t help me. So how do you people manage to pull this off? I’m really envious that people can code all day long but does it really provide good code? Doesn’t that increase the bug count of the code?
Easy, most of the code we write is pretty mindless. Boilerplate tying together of various internal libraries to make it play nice that you've done too many times to count. Following someone else's wiki page instructions on how to implement some "best practice" that needs to be done for a production system. Writing CRUD operations and such. Writing unit tests for all these things. Sure mistakes happen when you're exhausted but the stuff is hard from mentally demanding given the bar that they're supposedly hiring for.
I don't say that disparagingly, this is 90% of my day to day life as a mid-level engineer at a FAANG. The other 10% I might get to do something "neat" that I can justify somehow. It depends on the team of course, but I find the Author's experience believable.
Lots of IT jobs that deal with critical infrastructure have on-call duties.
Try working for some IT firm that supports healthcare, utility, defense, or what not, and I can assure you that there's rotation duty where you need to be on-call within xx minutes, 24/7.
During college I was interning for IT firm that provided networking for local hospitals, and they had the same rotation. My friend who got me the gig, worked there FT, and he basically had to forego social activities every 3 or 4 week, because he'd be on call 24/7.
Yeah, I also thought the on-call/pager thing was a little overblown. It immediately raised a red flag about how wet behind the ears OP is.
Not to say that there aren't toxic on-call rotations and such, only that it's a lot more common than you'd think, and is not limited to just doctors eyeroll
Any time a business unit needs to be accountable to another party after-hours, some sort of call schedule is created.
> During the hiring process, on-call is not mentioned in any way other than the usual salaried catch-all "are you willing to work nights and/or weekends."
It should be very clear what the actual expectation is, though, instead of a nebulous "we may need you to work extra sometimes." It's a guarantee that you will work extra at a fixed interval, and that extra work happens to be psychologically draining.
I ask for quite the premium in salary if I know I'm going to be on call.
I found that weird. On-call is so central to the technical identity of Amazon that I'm surprised you could get through the hiring process without knowing that you'd be on-call.
> After being on the team for a couple of months, I was put on the on-call rotation
I don't work at Amazon but the situation is very similar to my job. On-call can be brutal, even if there are no incidents during your week (which is rare.)
Just the fact that you have to be 24/7 _ready_ during a whole week sucks, especially when you have a family. During my on-call week, I can't go get groceries, I can't take my kid to her hockey practice, we can't go out for lunch over the weekend, etc.
The 1st time I worked at a company with "on call" we had this concern while trying to define the policy/rotation/etc. It was always "well, yeah I can be on call sure, but this specific Tuesday at 7pm I have this thing". We solved it in 5 minutes: just have a primary and a secondary that gets the page after 10 minutes. Solved. Go get your groceries, if it's bad luck and the page happens during the 45 minutes where you're out, your buddy will get it.
Yep. This was the solution for us. The secondary is the next-in-line for primary. Secondary is also first in line for helping out during incidents. This means communication with other teams, a second incident popping up (rare, but has happened), etc.
I’ve found that just having someone else to commiserate with at midnight, if needed, really helps the mental state of on-call.
Absolutely. We had the same policy. If primary wasn't sure what to do, they could ping secondary to regroup. We even had a third "last line of defense" which was basically always the same 3 people, pinged if nothing was acked in the first 30mn.
At my last position they tried to offer me a "promotion". More responsibility, more accountability, mentoring juniors AND on call, all for the grand pay raise of $0.
"on call" was super, super vague and I pushed back hard on it. If they require me to work outside something reasonable like 6am-6pm, they will pay me at least double for it. No exceptions.
I wish everyone would push back that hard, and that on-call was an official opt-in "extra pay" scenario.
Needless to say I turned down their "promotion" even after months of them trying to make me take it.
Note: I am NOT saying I'll work 12 hour days. I'm just saying if I have to come in early for a meeting for 6am, or stay until 6pm sometimes, that's fine. If I'm expected to come in or be on call at 2am, that's a different story.
There was one period in my life at a 401k administration place where I was responsible for making sure the transaction data for all of the clients made it to AIG every night. For months, I had to do the transmission manually at 7pm each night, which meant not much fun for me ever. I never even got a thank you, and when I finally managed to get it automated, my position was eliminated. At least they gave me a pretty good severance.
A 5 year old gist from 1 employee (out of the current ~70k SWE's employed at the company + all that have come and gone in the last 5 years) just to shit on the quality of Amazon engineering? And folks wonder why I have depression -it's not because i'm burnt out, it's because the HN crowd thinks I'm some sort of cognitive defective
for working at Amazon.
Fellow happy Amazon SDE - hopefully you're not being literal saying you have depression from HN type crowds displaying their typical amount of empathy. just in case you are, take this as a friendly reminder that you're not the only person with that experience and it's easy to just stop reading the nonsense
Yeah, I know. But even when talking to friends at top companies it's like a stream of microaggressions when I've worked just as hard as they have to be where I am.
This post convinced me to apply to Amazon. At the end, he says working there overall is pretty good. The pay, the impact. "Burned out" here doesn't mean he quit, he just scaled back his workload.
I will add to that, I have a friend at Amazon. I visited him in Seattle this previous winter and he had to work on Christmas Eve while most other top companies do Two-day holidays over Christmas.
I used to be like that and then one day I realized it's just a CS job, I'm not doing open heart surgery so I shouldn't stress about it. Since then my life has been much better.
I can chime in that a lot of this rings true for me. I was also an SDE-II ( started as one ). I also was loaned out to other teams. I also suffered the idiocy of being locked into 2 years of work or get saddled with a huge amount of money to pay back.
Also, I consider that my child is dead because of what happened.
I'm not sure if you're legally bound not to say certain things by the terms of the NDA you signed, or if you've simply chosen not to share some details. But I read that entire page, and who is in the wrong in that story hinges entirely on the exact events of the day you requested to go home, and exactly what you said that was considered a death threat. The extreme vagueness around those details leaves me with no ability to form an opinion on whether your manager is a whip-cracking sociopath, if you have serious anger management issues and are blaming the consequences of them on anyone but yourself, or somewhere in the middle.
I wrote a long response to this. Hacker news then presented me with a message "Your comment was too long".
Short version: Think what you will. My story serves as a warning to anyone who thinks you can survive the Amazon hell. I may have issues of various sorts, but no one should have to tolerate the "Amazon way" or be subjected to their insanity.
Amazon's 401k plan is terrible when compared to competing companies. I cancelled my Amazon interview the second I saw the terms of it. I had to find the details for myself because the info packet the recruiter sent me, conveniently left out any details except that there was a 401k plan and some matching.
Only 50% match, and the maximum match they will pay is 2% of salary. (Competitors match 1:1, high or no limit).
Three years vesting on the match. (Immediate vesting elsewhere, e.g. Google).
No match on catch-up contributions. (WTF? Naked age discrimination as policy?)
It was accurate until this year - Amazon switched 401k providers from Vanguard to Fidelity. I don't remember all the details but after tax contributions (which are required for doing a mega backdoor) are possible now, but weren't before.
I was at Amazon for four years and enjoyed my time there. I had one stint on a high profile high-pressure team that lasted about 6 months (no on-call, though). That was the busiest time I had at Amazon but it was kinda known going into that project.
I generally worked 45-50 hour weeks, with the occasional 50-60 hour weeks but that was pretty rare.
It's a big company. It really depends a lot on your manager, your team, and your org.
No, they won't run out of people to hire but the cost to hire them will go up. At a certain point, they may have to pay more and offer more perks to attract the level of talent they want. The culture will mellow once the payroll expense starts climbing.
> I mention on-call duty because it is “peculiar” in that the only other profession requiring this kind of responsiveness is doctors
Uh, not even close. I used to get up at 4am EST to trade European markets for a hedge fund. I was on call 24/7/365. I had my laptop and Bloomberg fingerprint scanner with me on any trip I took.
And this is just my experience. I have had many friends in engineering, management consulting, fin tech, etc who have similar demands made of them.
I’m not saying it’s right, but it is part of the rules of the game that we all sign up to. There’s a reason all of the aforementioned jobs pay so well.
In the military, we had a 2 hour recall for a large chunk of our battalion, about 200 people for a few months at a time. If you got a call, you were expected to be ready to go in less than 2 hours. Social norms meant the expectation was closer to 45 minutes than 2 hours.
I think oncall is a common requirement for anything involving large sums of money, treatment of high risk injuries, or geopolitical power struggles.
One I hadn’t even considered (typical stereotype here: no friends who are in the armed forces). But yeah I imagine loads of others (military, EMT, police) are in a similar situation.
> I mention on-call duty because it is “peculiar” in that the only other profession requiring this kind of responsiveness is doctors, literal lifesavers. When you go on-call for the first time, it’s terrifying and tells you “holy crap, this is serious.”
A key difference is that the oncall engineer is usually able to directly influence the quality and robustness of the systems being watched. Sort of like preventative medicine, but much more direct.
Crappy teams make crappy software that pages often (or not at all if their ops fails to write any meaningful alerts)
I don't trust an engineer that can't troubleshoot their own software running in production. I think the "you write it, you own it" adage is really smart and raises all tides when engineers are empowered to root cause each issue.
Crappy managers stress delivery over quality, and then engineers pay for it when they're oncall. The manager doesn't care how much their team gets paged because they'll be sound asleep while their engineer bangs their head against the wall at 2am, and the manager gets their pat on the head for delivering more projects.
Well... some of us join teams that have problem software that is going to take a long time to fix, and don't have the manpower for it. And sometimes the alarms are overly sensitive.
The on call stuff sounds outright dystopian to me, it's not even legal in my country. More places should think about adopting something akin to France's El Khomri law's (https://en.wikipedia.org/wiki/Right_to_disconnect) and ensure that every citizen has enough time to recuperate from work and isn't pestered by their employer 24/7.
How are you proposing to run a complex system that essentially powers a big chunk of the entire internet and can go down any second without on-call?
If a big selling point of your system is the ability to be on 24/7, on-call is inevitable. But it definitely can be way less gruesome than it currently is at Amazon for a lot of teams right now. For example, my on-call schedule is extremely relaxed and low-stress, it is never a worry I actually have or something I dread ahead of time. But getting rid of it completely is simply impossible for a lot of "always online" services.
Hiring more people is a great idea for making on-call less stressful, because it allows for much sparser on-call shifts (e.g., one week every 8 months vs. one week every 2 months). But I don't see how hiring more people gets rid of on-call completely. Someone will always have to be on-call.
You cannot just hire people who do on-call-only duties and nothing else, you have to be very closely familiar with the code and features being written in order to resolve issues effectively. Which usually means that you need to be someone who actually worked on those things, not just some glorified extra-fancy customer-support worker.
According to TFA, programmers at Amazon are required to be on call for a full week at a time. More programmers could mean sparser on call shifts, but it could also mean shorter on call shifts -- i.e. instead of going from one week every two months to one every eight months, go to two or three days every two months.
Being effectively at work 24 hours a day for seven days straight sure sounds like a recipe for burnout to me. As TFA notes, these people aren't doctors, the only thing that happens if a glitch takes two hours to fix rather than one is that Amazon loses some money. Amazon naturally would prefer not to lose that money, but its workers presumably would prefer to be able to sleep too.
What was stated seems pretty typical on small teams who manage critical systems. How are critical systems managed otherwise without having to 3x your team (assuming you want 24hr coverage with a 8hr workday)?
>How are critical systems managed otherwise without having to 3x your team (assuming you want 24hr coverage with a 8hr workday)?
They typically aren't, you need to hire more people then. In anything defined as critical I would suggest that being well rested is also a requirement rather than having a team of people who are so overworked they develop mental illness.
Doesn't make sense financially. You're gonna triple the size of your team, just to make sure that this one time of the week someone will restart the DB?
I would much rather bite the bullet, be on call a week a month or so and use the budget for something else.
You don't need to keep the primary development team as primary oncall 24/7. "Hire people to keep an eye on things off-hours, follow playbooks when necessary, and escalate to the right people if they can't resolve" is a reasonable pattern that folks have done for decades the world over.
You get less incentive alignment perhaps, but it really isn't hard to come up with ways to make creating work for another team unappealing.
This is a law to protect the mental and physical well being of citizens and workers, obviously not Amazon's bottom line, that's literally the point. If you treat people like disposable commodities then yes this does not make any sense.
You know what impact the mental and physical well being of workers? Have their company go under because they had to triple the size of the team just to implement an on call policy.
And I'm not talking about Amazon here. Of course Amazon can afford to hire everybody and their mom to cover 24 hours in a day. I'm under the impression that your original comment was really a critic of the practice in general, not just this specific example.
Some small companies build products they try to sell to bigger companies. These bigger companies have SLAs, so they need SLAs from their providers to sign a contract. Even at a small startup on call may be necessary to get customers.
>I'm under the impression that your original comment was really a critic of the practice in general, not just this specific example.
It absolutely was criticism of the practise in general. Yes, small companies that can't ensure that there workers aren't overworked don't really make sense with protections like these. But the same is true for environmental protection.
Critical infrastructure arguably ought to be handled by companies large enough and with enough staff to comply with regulation like this.
For anything else I don't see the issue. Some random smartphone app company doesn't need to fix anything at 3 am in the morning, it can be fixed the next day. That's exactly the culture I'm critizing. 24/7 readiness to work for some random product at the expense of mental health is awful.
I feel like we're just going in circles because you're not addressing any of the examples I'm providing.
I gave you examples of how to implement on call without overworking your employees and even giving them the option to run a few errands here and there. So what's the problem? You seem to be against the practice just for the principle, not even really for any bad effets due to bad implementations.
As for which products are worth implementing "on call" for, yes I agree, some companies are too quick to think that the world can't function 30 minutes without their product.
Anyways, I've seen it implemented successfully at companies with very low attrition and I do believe it is a necessary tool sometimes for small companies to grow and sign big whale customers.
Why are we assuming that Amazon, the third largest corporation in the world by market cap, needs to limit itself to small teams?
They could afford to 3x the teams if they wanted to. They just don't want to, because management prefers to just keep the money and burn out their subordinates. (There's always more where they came from.)
Then that's the cost of managing the service. The only reason companies don't pay that now is because of the significant leverage they have over software development labor. We may be better compensated than many other roles, but we're still just entirely expendable resources.
The only reason companies don't do that is because it doesn't make sense. What are you gonna do with a team that is 3 times the ideal size just to have enough people to cover a 7/24 on call?
It absolutely does make sense if the service is that critical. If it's not that critical, it doesn't need 24/7 on-call. There are some "in between" circumstances, sure, and those might require additional resources or occasionally work outside of normal hours.
It doesn't make financial sense because companies can get away with exploitative behavior because software engineers are quite naive (or even actively detrimental to themselves and their peers) when it comes to labor relations.
So you have 5 engineers building your product. Now you hire 10 more and organize them in 3 shifts. What work do you give to the 10 new engineers? Are you supposed to now hire 200% more PMs, designers, QA, etc and make up some new projects for these guys?
I see this comment again and again on this thread "if it's really critical spend the money", but the thing is even small companies sometimes have critical systems. Simply because they're trying to compete with bigger ones, or they're partnering with companies that do have critical systems and the SLA gets passed down the chain of dependencies.
More members on the team would increase the gap between 2 on call periods for team member (e.g. every 4 weeks instead of 2), it wouldn't get rid of on call entirely.
Where I'm at we have an on-call schedule like the one mentioned in the post. It isn't too bad, and we rarely ever get paged. Most of the issues that the on-call person has to deal with, at least on my team, will come up during the normal working hours.
I don't understand what's so dystopian about it. Some systems are critical and need to be kept online at all time, or at least have very minimal downtime.
The description in the post is not even that bad. A page a week is nothing. And it is clearly state that your workload is reduced during your on call period.
On my team, net workload is not reduced, it just becomes radically different. It's only your time spent on longer term projects that gets reduced (to zero).
The dystopianism comes in because the vast majority of teams are decidedly _not_ 27/4 critical. Sure, some teams really run the 1-Click Buy button or the S3 API. Most run some DB behind some API behind another 3 APIs behind 5 other teams that run a widget.
In practice, your workload may not be reduced during on call because it's "expected" and you still have to meet your normal deliverables while the team isn't staffed with "extra" people. It also takes a toll on employees to be mentally-available at all times in case some triviality emerges and raises an alarm. You'll find many engineers not willing to leave a small area or even go out for dinner without bringing their laptop and a hotspot, just in case.
Your "in practice" doesn't match my on call experience over the past 15 years. During "on call" workload is reduced (the author of the post even mentions it). There are ways to implement sane on call policies that benefit everyone. A few I've seen:
- use a primary and secondary on call so that the primary can still go get groceries, or play in the yard with their kids, etc
- overlap on call and work hours. I did that at a job and I was fine with working 6pm - 2am just to cover this time slot. Won't work for everyone, but may be a nice compromise for some.
Like everything else, there are ways to do it right, and way to go complete bananas with it.
> Like everything else, there are ways to do it right, and way to go complete bananas with it.
Sure, agreed. Sometimes it's implemented well and sometimes it isn't. I've experienced some of both.
When it isn't well-implemented and is particularly non-critical, it can feel a bit dystopian to the engineers. If done well and for a good cause, it should be fine.
I'm not sure where they got the notion that emergency services are the only other field where people go on-call. Plenty of industries operate 24/7.
I worked in high volume manufacturing where process engineers could be on-call 24/7-365 unless you explicitly asked a backup to cover you during a vacation or weekend. Depending on the part of the process you owned, that might mean a middle-of-the-night call a couple times a month or several times a week. No human lives on the line, just lots of money.
A part of me feels bad sometimes cause I doubt I'm good enough for some FAANG company...then I read stuff like this and I think, yah, ability or no, I don't think I'm type of person who would want to partake in this culture.
I was a contractor at a company that announced that developers would have to work through the nights and on weekends at times, and they couldn't take any extra work days off. So, if you had to do an upgrade to production and it took all night, you were still expected to work all day too. Oh, and you didn't get paid any extra for the overtime either. One person asked about work-life balance, and the response from that manager was that when you chose a career as a developer, you signed up for this. This is what happens when this kind of abuse becomes rampant. The managers then declare it comes with the job. Such BS! BTW, that guy quit along with anyone else reasonably competent.
That blows. Seems like the company was wholly incompetent and rotten with comments like that. And as a contractor, you should always bill overtime so your unavailable time isn’t abused.
> On-call wasn’t (and isn’t) too terrible for my team. At first we averaged 1 page every 2 weeks; now we’re up to about 1 a week.
Wow, I wish I had once a week. This is way better than my previous employer/role. I was on-call for months straight, and would get paged multiple times a night, every day, even at 2am, 4am. I had to carry my laptop to dinners, date nights, game nights, etc.
I got burned out and moved careers into a role that never has on-call. I don't get to code much anymore, but I've found other things to enjoy
Yeah On-call can vary greatly by company, although how On-call issues are handled can have a huge impact on your perception of a job.
At one previous employer, you were on a 2-week rotation for on-call.
But Every time you were on call, it was the same set of problems, always at the same inconvenient times (Mid sunday afternoon, random weekdays but always between 3-5AM, etc.) Admittedly often they were NOPs (Oh, alert came in, but when you check the system it's fine.)
But we didn't fix the false alarms. And we didn't fix the recurring real problems either; the Org's view was that was the entire purpose of the On-call person; that middle child you send to pay the loan shark not-quite enough because you don't care what happens to them.
At later employer, I actually had to deal with -more- on call issues, both in severity and frequency. And yet, I hate being on-call less, because at the very least when we find problems on-call the org puts a focus on fixing them.
I don't know how absurd this sounds but on-call is the only reason I have stayed away from backend or frontend/web development. I had worked on both for few months and ran back to native app development. There might be a place where the company has a "follow the sun" policy for on-call but I haven't come across any.
Dunno, I read through half of the post and have not figured what actually the problem there? If 15 minutes you have not responded while on "pager duty" your manager get notified? Phew. I mean if you are infrastructure worker, you kinda need to go and fix the infrastructure that is expected by everyone to be efficient and up 24/7. Imagine, your internet goes down once a month, you will be among first to complain how sh... I mean bad, your internet provider is and threaten to switch to other option. Internet goes down often, and there is a person or a group that goes to the affected location and fixes the issue during day/night/rain/snow. Now think about people who's livelihoods depend on AWS to be up, or customers being able to purchase products on amazon.com... business owner, track driver, manufacturing worker, and bunch of people who have work and income because those workers do. If programmer's job was so easy, everyone would be able to do it. Problem with today's society is that we see people's success and desire a piece of it, we do not see work and sweat that was put into achieving that success. Programming not for everyone, if you not used to stare at the screen 16 hours/day, maybe there is different career for u somewhere.
>There is something great about having code you wrote be actually executed by thousands of people every day. So few people get to show their friends and family a site they visit often and say "I did that thing there." Maybe it will wear off eventually, but it's still my favorite.
It has never worn off for me! I've worked in automotive for over 20 years, and it feels great every time I see something I worked on drive by.
> Revisit your bonus/relocation payback strategy ... if that's what you need to keep employees, that's fucked.
The reason for it isn't employee retention. It's to stop people from accepting a job from Amazon just so someone will pay for their move to Seattle and give them a free bonus.
Would people do this? Yes. I've seen them try. It also discourages people who aren't serious about the job.
I don't want to minimize what this guy went through - anything that takes you to the edge is too much - but I can't help but interpret this story as a story of professional & personal growth rather than an example of why Amazon sucks (Sidenote: I happen to agree they suck, just not for this particular reason).
How many people in other professions have had similar experiences from the perspective of intensity/amount of work? I-banking is _built_ around this level of intensity for years, and I've heard similar stories from people at other tech companies, plus VC, PE, consulting, medical residencies...
My big takeaway here is much less that Amazon sucks than that in a high pressure job, it's critical to know your limits: "I can go so far and no further." The author seems to acknowledge this; why are so many comments implying Amazon is uniquely terrible?
As other have pointed out, it's probably 1000x worse to be a doctor saddled with med school debt who is literally operating on people. I will not argue that, ever.
But that's also not me and so I told the story I could; mine.
I really want this to be more about how vital professional and personal boundaries are.
While also highlighting how certain company behaviors make it challenging (on/call, etc.)
Before Amazon I had worked at a company that (due to the nature of the work) necessitated a strict 9-5. You literally couldn't work a minute more.
Before that, I was in college and had hourly jobs.
So you could say this was my first big-boy job with salary-hours, as well as real responsibilities (rent, car, wife, life)
I have had a different experience with working at AWS. I've heard really terrible things working in retail, but AWS has been a pretty good place to work, with normal 9-5 hours. I can go more into detail about how my experience has been different from the article if people want.
I'm a software engineer at Amazon in Retail. (This is just my opinion, I don't speak for Amazon in any official capacity, etc.)
Different software products have different severity levels. For my team, the oncall burden is not so bad. We have an internally facing tool. When we do get paged, it's usually requires acknowledging a ticket, a few minutes to validate something, sending an email out, and that's it. For teams with more externally visible products, a particular page can be stressful, but my understanding is they also spend a lot of time in ensuring system robustness in order to limit the number of pages. The "15 minute" response time the post referred to is generally to acknowledgement - the expected time to resolution can be quite variable.
The working hours are generally pretty good as well. The general expectation is that you're not working crazy hours.
I've ran into "project crunch time" at Amazon, but the experience at Amazon was like everywhere else I've worked. This also will vary by team: teams that need to make changes to support Black Friday obviously have a pretty hard deadline. Or if your team is getting ready for a re:invent announcement, there's no way to work around that. But from what I've heard, teams that have such a hard deadline also have a corresponding quiet period in December or January.
Most people I know within Amazon that get burned out on one team and feel like there's no resolution within that team (or just feel like they've had enough with one area) solve that by switching teams. This is generally pretty easy, since most teams are hiring at any time, and they would rather take an internal transfer than an outside hire.
Amazon has a pretty good system around having a senior individual contributor track (I haven't worked for any of the other big tech companies, my understanding is that they're similar). Part of the benefit of this system is that SDE1's/SDE2's have someone to reach out to for helping navigate management. Anyone at Amazon who reads this who is experiencing something similar to the OP should feel free to reach out to one of the senior engineers for help.
Overall, I don't doubt that the post in the original article is that person's experience, but it has not been my experience.
'Encourage employees to be critical not just of ideas, but also of expectations.'
From my experience as a developer; I've never really been involved in the process of setting expectations. Something I find incredibly frustrating at times. It's like sure you can give your opinion but it's too late to make any difference. I think it comes back to the main point in the article about open communication.
I think open communication needs to be inclusive. However, that's rarely the case at least in terms of setting business expectations. It's only natural to want to be involved in shaping some objective or some expectation that impacts you and for which you may be one of the more qualified to discuss.
>I've never really been involved in the process of setting expectations
Nobody will ask you. You have to be upfront about it from the start.
Request time estimations and fully fledged requirement definitions from the start. Review them before accepting a project, push back against overly ambitious plans and propose changes that allow you to deliver value quickly and iteratively. It's easier to react to surprises if you are able to test the ideas as you go.
Stick to the plan as much as you can. Push back against changes to the requirements if they're not vital. Let others know about blockers clearly, as soon as you run into them. If you get behind schedule, communicate it properly. Stuff happens, unfortunately, and deadlines not being met is a part of life. You just get better at estimating with experience.
There's no way the upper echelons will adapt if they don't receive input timely and loudly enough. If your boss is any kind of decent, they will listen to you and trust your input. If they don't listen, then you should reconsider your current job.
This is why you don't let upper management walk all over you.
I would personally love to tell Bezos to "fuck off" if the ask was unreasonable. Worst case scenario, I have to find a new job (which I usually have a few companies/recruiters in my back pocket), or best case scenario I get "respected" and the deadline is pushed to a more reasonable date. Typically massive corporations like this put you on a "performance improvement plan" which can give you few weeks to a few months to find a new job (or "improve").
Also learn to make friends with your teammates in the trenches. Presenting a push back to management in a unified front is much more compelling than a 1:1.
You could be shown the door in no-time. Amazon has forced attrition and the manager decides who gets the cull. They may make a mountain out of a mole to further their objectives.
As someone who came from a similar environment (MSFT during the stack ranking days), the forced attrition and manager's arbitrarily blackballing employees are the ingredients of the toxic stew. If a manager hates your guts, there's really nothing you can do, even if you are great at your job.
Does your manager love your guts? Are you good at your job? Hopefully that paycheck can bleach the guilt of your colleagues losing their jobs because you are good at yours. That toxicity could also be because they are either selecting or creating people who are okay with this.
Yes, yes, and thankfully I left that environment. Now I work in the public sector, I guess you could say. Mostly because of the gas-lighting of reviews and performance "feedback." I've been on both sides of it, and seeing that I'm the same person, it really made it obvious how arbitrary it is.
I feel like it's common wisdom now that forced culling is not a great strategy and has lots of perverse incentives, but still not universal.
> GitHub and Gists seems to alot of people's go to place for writing these days. I'll take it over Medium but still seems a bit odd.
IMO it's the lowest effort blogging platform (for people who already know how to use git). You don't have to go to a webpage - just write some text in the editor of your choice.
You don't even need to use something like Hugo since Github will render markdown files properly.
If you're just writing some throwaway content then sure, I guess the low effort is worth it. But if you want to write more than a single post it's probably better to spin up either your own blog or somewhere else.
This is actually from Pastebin which I used to really enjoy finding random anonymous stuff on with the trending filter. It was mostly random logs, stack traces or email dumps but occasionally you'd stumble on an interesting anonymous memo like this.
Did they get rid of that feature or was it replaced by Public pastes on the right hand side?
I've only ever used Pastebin for gaming leaks and I don't think the interface leads itself well for long form posts. Probably just me being picky though.
I'm not sure, I quit checking for awhile then went back and it was gone or hidden maybe 3 months ago, I feel like I did find it but maybe it's just the public pastes at the top now.
>I mention on-call duty because it is “peculiar” in that the only other profession requiring this kind of responsiveness is doctors, literal lifesavers. When you go on-call for the first time, it’s terrifying and tells you “holy crap, this is serious.”
... or literally thousands of people in the military right now, oil rig work, countless other IT help desk jobs, etc.
> During the hiring process, on-call is not mentioned in any way
Then next:
> are you willing to work nights and/or weekends
To be fair I'd say this is mention enough.
If someone is willing to (or has to) look the other way at the time of hiring when something like this was explicitly said, then even a "hey, we have on-call here, you fine with that?" would be okay.
I think it goes both ways (in fact, most of this post seems to go both ways in that Amazon sounds brutal but OP could have taken actions to mitigate it). If your team is going to be requiring on-call, you should say that during the interview process. "Are you willing to work nights and/or weekends" is indicative, but is still too vague. Some might interpret that as just "sometimes you might have to work late if you are on a tight deadline", not as an on-call situation.
However on the interviewee's side, if you see something in a job description that asks if you're willing to work nights and weekends, that should be an immediate indicator to you that you should ask why you possibly would have to work nights/weekends and in what capacity. It's the interviewee's responsibility to ask questions like this even if the employer isn't immediately forthcoming.
>Be upfront and as precise as possible with what a position entails. "Working alongside smart and passionate people" is lazy hiring.
I understand that this person was coming straight out of school, but the onus is on the applicant to figure out more about the job during the interview process.
Interviewers, if they're coached properly by the organization, will lie. "Sell the candidate on the company" is a thing. Human beings aren't conditioned to have their skepticism alarms ramped up to 100% constantly in every interaction. At some point a certain level of trust to be honest with one another is required for society to have any cohesion. In my view businesses, especially software/tech companies, are quite poor at this on a good day and frequently work in ways that undermine it.
Based on my experience with the various AWS SDK libraries, I don't think Amazon has very good developers, or perhaps they have good developers but a poor development culture. The architecture is convoluted, heavily OOP and just generally reminds me of Java in the early 2000s.
> Based on my experience with the various AWS SDK libraries, I don't think Amazon has very good developers, or perhaps they have good developers but a poor development culture
Dude, have you fucking seen the Android libraries? They're awful! At least the AWS SDK libraries are like that because they're all autogenerated from a spec.
Anyone working over their contracted hours needs to realise they're being exploited by their employer. Sometimes this is justified: we pay more than the going rate because we expect you to step up occasionally. Maybe it's kind of expected with Amazon in particular.
But when it descends into perma death march and your manager flings you "a few" days holiday in recompense... does that make up for it? If it doesn't, to whom has the profit gone?
They have a contract, but the contract will specify them as "exempt salaried" employees. That's pretty standard across the board in the tech world (as well as many other professional roles).
Exempt salaried employees aren't paid by the hour, and don't have a set number of hours they are expected to work. There is no overtime, but there's also no (contractual) penalty for working fewer than 40 hours in a week.
As a bit of background upfront, I'm a 6+ year AWS employee who went from an intern to full-time SDEI after graduation to relatively recently an SDEIII. All of the below also specifically applies to white-collar tech employees.
There are two big issues with a lot of employees at AWS, not knowing their worth and not having backbone.
It is not a privilege to work for Amazon. It is a privilege for Amazon that you work for them. In this particular case, this person says that he was an SDE II. That's great. You should be able to pretty easily go get a job at any of the other large tech companies. You're not locked into Amazon. You likely have a CS degree. You've got a couple years of industry experience in a field where most companies can't hire fast enough.
That said, don't leave Amazon. Leave your team. Internal transfers are easy and no longer gated on you being on a team for a certain amount of time. Start talking to other people from other teams and find a good fit. Most internal transfers are about culture fit and you likely won't be asked any coding questions.
You are not obligated by anything to stay on your team. End of. Amazon is one of the largest companies in the world. Don't feel some sort of personal responsibility to stay on one team. That's just not reasonable. They'll find a way to make do without you and if they can't why do you really care? At some local mom and pop shop, sure you may not want to leaving the owner hanging but Amazon ain't that.
Yes, there are rare occurrences where an org may try to block you from leaving. In which case, again, you are a tech employee of Amazon. You should be able to transfer to another large tech company.
As far as the loaner situation goes, have backbone. I've been asked to help out another team before. I did some research and decided I didn't want to. I told my manager and that was that. If he really pressed, again, see the prior options.
Don't be afraid to escalate. Don't care about politics. Your manager may hate you for trying to escalate on them but what the effect of that in the long run? You only get a 1% pay increase on your 150K annual salary instead of a 5%? You have to go work for Microsoft for 150K instead of Amazon?
Rough oncall? Change teams.
Oncall but want to go places? Do it anyways. All teams should offer a hot spot. Yes, this probably means you can't go hiking but it doesn't mean you can't go other places. Being oncall doesn't really ever change my habits other than having to bring my laptop with me. Sure, if you're on a team getting paged regularly, that won't work but I'm not sure why you want to be on a team that gets paged regularly after hours.
Come on people. If you work for a FAANG company, you are a very highly paid employee in one of, if not the, hottest career field in the world. You are very likely in the top 10% of world wide salaries. (118K+ year according to numbers from US Social Security Admin)
on-call is standard industry practice, there's nothing especially egregious about this. the difference is that some places like google pay extra for it.
What? Not without extra pay, that's for damn sure...not just "some" companies. That's ridiculous. I got overtime when I worked retail for crying out loud...
So, I wrote this. I'm the original author. I'm amznymous (still proud of that username)
It's wild to see this pop-up at the top of HackerNews 5 years later, but I'd be more than happy to answer any questions anyone has.
I'm still at Amazon and have been on a couple of different teams since then, but this was definitely written at my lowest time at Amazon. I've had low times since then and high ones as well.
I'm still an SDE II but on-track to become SDE III soon.
I've interviewed elsewhere but was either unimpressed with the company (Facebook) or the company was not impressed enough with me and asked to try again later (Google). Facebook interviewers all looked absolutely exhausted and stressed and lifeless. They zoned out in the interview and just in general left a bad taste.
One low point was when I found out (years after 2015) that the whole project that I had brutalized myself over was shutdown. Deleted. Scrapped. All of it.
It was devastating and cathartic. I've become a bit more balanced, jaded, and care-free since then. Not totally, but just a bit.
It's all just code. It's all just business and sometimes the business fails and everything you did doesn't matter anyway.
I'm still glad I wrote the paste, if it helped one person out there. Based on reddit responses, I wasn't the only one going through this.
=============================
Since then things that help:
0. Get that "Fuck You Money" I pinched pennies and followed Dave Ramsey's approach to get 100% debt free and thriving to ensure I never was locked in as I felt I was.
1. Being absolute about my sleep schedule.
2. Finding a good team. When you find a good team/manager, stick with it. Manager matters 10x more than the company
3. Therapy. duh
4. "Man's Search for Meaning" by Viktor E. Frankl
5. "The Depression Cure" by Ilardi. This is NOT A SUBSTITUTE for therapy or medication. ONLY use it as a supplement . The tl;dr is take care of your body or your mind will pay the price.
6. Soft skills: developing skills on how to provide estimates, pushing back on deadlines, and finding compromises between what business wants and what's feasible. It sounds obvious, but sometimes it really is just as easy as saying "that will take 2 weeks, what can be cut?"
(Obvs, my opinions are my own and not my employer's)
Really just gossip is what you have to go off of initially.
After you've been at the company for a bit, when you're interested in a new team, you can usually get some meet-and-greet meeting with current members on the team.
That's the time to ask really tough questions, even potentially awkward questions: how many times do you get paged? What's the worst project you've worked on? What's the average tenure of team members? When people leave the team, do they leave Amazon? (If they leave Amazon, then the team is less to blame, otherwise, beware).
I would have guessed AWS from your post since it's the more notorious one about bad WLB. As a new grad SDE1, things are bit horrifying especially since I've onboarded remotely.
What is it about tech culture that people within it refuse to admit the level of abuse-by-design that happens at these companies? Is it that they want to believe that they are somehow fundamentally different from the distribution and grocery store workers, and fundamentally similar to Bezos? I can't help but feel it's just this kind of false consciousness that prevents developers from _getting organized_.
Sidenote, payroll tax on Amazon in Seattle trying to get revived. It's a long and winding story. Check out taxamazon.net