Worked for AWS for just under a year and a half; this mirrors my experience exactly.
Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter. Automated tooling and tests that were ostensibly created to make life easier were the number one pain point and constantly failed in obscure ways that required cutting tickets to teams responsible for the automation.
More often than not these tickets would be ignored and issues would persist for weeks beyond what's reasonable.
I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
That, paired with the awful corporate culture and constant fear of PIP/job loss forced me to finally leave.
Generally a very unpleasant experience all around, sans the compensation and name on the resume.
>I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
I did not mention this in my blog post, but I have actually done similar. That's incredible. Thanks for sharing this comment. Our tenures were almost the exact same length too.
When I worked for a BigTechCo, I was doing exercism every single morning for ~30 minutes for the same reason. It really is incredible. I know a few of my former colleagues were doing the same. Today I'm on a team with 4 other engineers, so I keep busy coding.
This is the general feeling of what I see from AWS support as well as a consumer of the platform.
I'm battling something for weeks now which is a completely broken ass piece of shit inside AWS. I spent the first 3 weeks trying to get attention for this via support tickets and someone to take it seriously, resulting in escalating to account management. I've been on calls with the programme managers, been apologised to constantly and absolutely nothing has been done that is productive. Not only that they sheepishly suggested other customers were in the same shit.
My conclusion at the end is that critical bits of AWS are held together with only a couple of people who actually know how it works and they don't even have the ability to do fix anything...
Not that the consuming company I work for isn't a complete mess either but I expected better from AWS.
I know the types of companies that are Microsoft shops come with their own problems, but my quality of life via tooling at two Microsoft shops has been great.
I think they mean places which use Visual Studio, (usually) C#, Windows Server, Azure, etc. Lots of components of this stack tend to work a bit better if you use all the others. Common in finance, because of Excel.
Poorly designed, failure-prone, brittle internal tooling was a time and energy sink
Sometimes I get frustrated at my own job for some reason or another, but I appreciate learning that the grass isn't really greener anywhere else & even places where tech is the core business are having a lot of the same problems.
Completely agree. Interesting how the internal dev satisfaction polling always shows that the vast majority of devs just absolutely love the internal tooling and processes. Might have something to do with the polls not actually being anonymous. When hr/polling teams are asked about anonymity they generally skirt or ignore the question, but I've learned that enough metadata is collected to identify any responder in pretty much any internal poll.
There is very little transparency as to how the polling data is used. I'm sure many employees don't feel comfortable giving honest answers on many of the questions. Also the surveys go far beyond tooling question - they ask about future career plans, happiness with the company, happiness with management, whether employees are interviewing for other jobs, etc. It's not hard to see how certain answers to those questions could color the responder in a very bad light from managements perspective.
You should have just switched teams. Amazon has amazing teams working on deep technology, they also have huge systems with technical debt and stress. People dont realize how different it can be inside the company.
They all look shining from the outside, problems start when you join the new team and realize the pile of st you will be dealing with.
Btw this happens in most companies, tons of tech debt. And those who created that st are off onto new projects recreating the exact same mess all over again.
It is a cycle that never ends.
Only chance is to join early and be in for the long run.
I've made a great career of being a code janitor and doing what others don't want, while having a good attitude. In the last few years, I just rewrite everything I possibly can in Rust and I've yet to regret it.
Most people don't know how to do this, because it's actually hard, and I don't think it's being taught much in programmer education.
I certainly had to learn "on the job", and probably spent 10 years before I got good at it.
I learned the most by fixing bugs. After a while, I started seeing the patterns of why this bug had occurred, and started writing code do avoid the traps.
Before I left Amazon I tried transferring to multiple teams after speaking with hiring managers. They all (IIRC 4 separate ones) wanted either a full on-site loop or at minimum two coding interviews).
I was an L5 with about 5-6 years of experience, 1.5 at that point at Amazon.
Hiring managers have a lot of latitude at Amazon, so there isn't a consistent standard. I don't want to share too much PII but I was in a similar situation. I was able to transfer teams after an informal chat with a few engineers on the team (discussing past projects, the new team's product vision, etc.). There was no whiteboarding involved.
The hiring managers are themselves managers of the team e.g. for a team of devs, a software development manager. In parent comment's case, the manager likely asked for opinions from the team after the chats before giving the final approval.
IME, the best managers are the ones who trust and enable their teams to make the right decisions. It would be a red flag if the manager was just giving out offers without allowing the team to provide their input.
There are so many data sources that are all queryable via API. There are a couple basic greasemonkey scripts already floating around but as an SDE it really isn't that hard to write something yourself... It does take a bit of experience to get a good heuristic on weighting the different data points but you can do way better than blindly guessing.
I will say as someone who HAS intelligently looked through this data for most teams at Amazon, there is an absolutely MASSIVE difference between the top 10% of teams and the median, and an even larger difference between than and the bottom 10%, which by all metrics appear to truly be hell on earth.
Amazon runs an annual survey with typically quite high participation rates, it is very detailed, and the results are often illuminating (and like many things at Amazon, widely accessible to all people).
Find out what they drink for "Team alcohol evenings."
Whiskey team? Ok, probably somewhat stressful but doable. Wine team? Plush and cushy, with a line of people who want to be on that team out the door. Vodka team? Oh hell no. Etc.
I've been on 4 different teams at Amazon (now migrating to a 5th). It depends entirely on the team and as well as your current level. Some do loops just like with external candidates. Others barely require more than a conversation with someone on the team -- though this one is a huge red flag. Teams looking for warm bodies are seldom good teams.
Most of the time its somewhere in between. You'll speak with the manager, speak with the team, you each review each other's artifacts, and if everyone's happy, you swap over. The balance of power is pretty nice. I've dropped out of the process many, many times after starting talking with management. Or even just learning more about what the team itself does (I couldn't live with myself if I worked on pre-roll ads, for example. So I noped out of that one).
Christ, I wish PIPs were that easy where I work. Some days it feels like the public sector trying to get rid of employees that coast for months at a time before we can work our way through the process to can them.
that can't be right (but i have no idea). In the new group, nobody has any desire to at least do a reference check with the previous group? That would be the first thing as a hiring manager i would try to check for, say "this person is a d-bag that the entire group hated, and we were about to pip him/her anyway"? That would be kind of an equivalent of an approval, even if there is no formal approval.
I mean, yes, having your manager say "I will destroy your career if you think about leaving" would be a kind of approval needed, but short of that level of viciousness it sounds like no?
Not familiar with Amazon culture so I don't know if that'd be considered acceptable behavior. I hope not!
> have you considered that the manager may rightly have bad things to say? or that is just never warranted.
The manager should put those bad things _in writing_ during performance reviews instead of trying to prevent their employees from leaving by dunking on them after they are informed about the fact.
Unless you’re in Focus or Pivot, you’re a free agent. When Amazon is performance managing you in any sense it isn’t up to your existing manager if you can move, it would be an L10 (VP) exception on the receiving side to allow you to move, even if the hiring manager thinks you’re great.
I worked a contract there and the team barely understood how some of those configuration panes functioned. It was always a case of staring at someone else's config and then at yours until you found enlightenment. And the target kept changing, so there was a very bad week where my computer worked for 80 minutes total. I coded like hell during that time, pair programmed with the other contractors or did math on a whiteboard for the rest.
Another guy on the team intimated once in a while that he didn't like the team dynamics, but he played his cards pretty close to the chest. After he gave notice, on what we all thought was supposed to be his last day, we showed up in the morning just to find his computer and all of his equipment in a pile on a desk. We never did figure out if we or he were off by one or it was his middle finger to the team.
All the more mysterious because later that day we did discover his middle finger: The lead dev, who seemed to be one of the more reasonable people on the team, and the resident know-it-all both discovered that someone had fiddled with their configurations in ways that broke the system but produced obscure errors and hard to notice differences (like punctuation).
I took a couple lessons away from that project, and reinforced some that I'd already had. Perhaps the most important of which is: Bragging rights for being first mover are very, very short-lived. Once everyone else has copied it, the most accurate adjective for your version is 'old', or 'primitive', not 'first', and absolutely under no circumstances 'best'. Developers don't like bullshit. Don't pretend like your liabilities are assets.
Man, that's crazy. You can go to big boy jail for that, haha.
When I quit Amazon, they ran to my desk and unplugged my dev machine, since they thought I was going to leave a software "time bomb". I thought that was the stupidest concern in the world, but maybe it does actually happen.
It was maybe his second gig. A lot of us are still little monsters at that point. Not to say that’s okay, but just to say I wasn’t that surprised. What I was surprised by was that I didn’t think he had that kind of animosity. Pretty good at hiding his feelings.
I never understood that “shut them out” reaction for people giving notice. Getting laid off, maybe. But I pick the time and place where I hand over my 2 weeks. I have all day or even a few days to do it, but I have known for a month or three that this was coming, I’ve known for days when it was likely to come, and I’ve known for hours or even days that it was happening today.
You don’t think if I was going to do something shitty that I would have done it first? You guys are really underestimating my intelligence, or at least my ability to plan. But then that’s probably part of why I’m leaving.
Also they say that liars think everyone else is lying, thieves think everyone else is stealing. So I wonder what this says about the paranoid person? Are they a danger to the company?
> Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter.
Considering rather good quality APIs they expose to the external world, this is shocking to read.
Everyone says this, but in terms of discoverability of the actual apis I have found the opposite is true. Azure tends to have docs with small mistakes, but they are extremely thorough. With AWS it seems to be a tossup: You can be lucky and have complete and accurate information, or you can be fucked.
Someone on another Amazon related thread mentioned that 50% leave after 2 years and 80% leave after 4.
If you do the math on their TC to see how it actually works out from month to month, it makes sense. Unless you get more RSU's, you take a huge hit in pay, unless you weren't cashing in on them to make ends meet.
You get most of your RSU comp after two years; vesting is back-loaded. Many (most?) people wash out before they collect, which obviously works out well for Amazon.
My Amazon experience was similar. Especially the lack of coding due to the emphasis on devops/meeting culture. I have since felt that once a product reaches maturity, additional headcount (butts in seats) are operators first and secondly a swarm for something like a broadly scoped breaking change, runtime upgrade or platform migration. There is plenty of fat to trim even in aws despite it being among the more secure jobs to have at the company.
I’ve seen work slow down across a few teams and it just leaves me in a place of boredom and complacency with a major technical itch. Despite this itch looking to be scratched, I cannot seem to find the time/energy to pursue coding regularly outside of work in the limited free time I have. Maybe if I were single, no kids, dogs…
I thought that the deployment pipelines are part of their core business considering AWS has such products that are meant to be used by customers. I assume that the internal tools are different than what AWS customers use for deployments?
A lot of internal teams don't use the AWS software for various reasons. So like, CodePipeline might be great, but there's an internal analogue (I'm not sure the detail I can go into here) that is awful that a lot of teams use. There's been an internal movement to try to get all teams onto AWS services, but it's incredibly slow moving and there's no timeline that I know of.
Oh man, we disagree! I miss Pipelines (the internal version of CodePipeline). Having worked at startups and now at Google, that tool is honestly best in class.
FWIW, I don't miss Quilt, or VersionSets, or Apollo, or Hydra, or TOD, or NAWS Pipelines bridge. But Pipelines itself, brining all of those tools together is amazing!
I was once pitched a startup by some Amazon vets whose elevator pitch was "Apollo for everyone". I literally laughed out loud, which was followed by some very awkward assurances from them that they were not, in fact, trying to be funny.
The entire CodeBuild suite kind of sucks when you stack it up to options available on the market including GitHub Actions, GitLab Runners, Azure DevOps, literally almost anything.
If the internal analogue is worse than the CodeStar suite, then yikes.
Codestar is simpler than and has too many feature gaps compared to the internal offerings, which makes building and deploying Amazon scale multi-tier, multi-environment services difficult.
I worked at Amazon from 1998 to 2003. Those were exactly my experiences. Clearly they don't care about the internal development experience and probably never will.
> but also a job that made me feel more unhappy than any other job I’ve ever had.
I can relate. I worked for many years for one "those" companies that is a household name. The pay wasn't outstanding, and the job was not one that gave me much fulfillment, but it was at a marquee brand. My business card opened a lot of doors.
When that job finally ended (after almost 27 years), I started looking at working for other companies. I would have been a particularly good fit for startups, as I was financially secure (thus, didn't need to be paid too much), and have a huge arsenal of skills and experience, that can be quite useful to any small outfit that has "many hats" employees.
What I found, is that us older folks are quite unpopular, and got tired of slamming doors and passive-aggressive insults.
So I just decided that I was in early retirement.
Best decision I ever made.
I would have liked having the extra money, but the soul-sucking aspect would have probably killed me, by now.
I am in a good position. I have enough set aside to be OK, but not at an exorbitant level, and I get to write awesome code every single day. It's basically, my "dream job."
I feel terrible for over-leveraged young folks, having to deal with this kind of thing, early on. I feel even worse, for the ones that have "crossed over" into "old," and are still over-leveraged. They won't feel old, but they will be treated as such.
it's weird to hate on older generations, particularly when you consider so much of what we use today in terms of algorithms was thought of in the 70s or older. none of that became irrelevant.
the only maybe maybe advantage that younger folk have is that they might use newer languages or frameworks, but then when those go out of style and they only have 1 tool in their belt and you've collected dozens over the years... it really won't matter if they're more proficient in one tiny area.
but let's be real, the languages and frameworks were never the problem to begin with. its the knowledge and foresight to see where things are going to go wrong which you can only get by years of experience
> What I found, is that us older folks are quite unpopular, and got tired of slamming doors and passive-aggressive insults.
Some of the older engineers I've worked with are among the most brilliant and productive people in the groups. In those cases I have seen bar none, they have been revered and respected among the other members.
Age discrimination is definitely a thing, but not everywhere. Maybe that rotted mindset of prejudice has taken hold in the "I'm saving the world broken by boomers by working at a corporation that sells advertisements on the internet" crowd.
As a slightly older engineer, I believe there are 2 sides to the coin. Age discrimination can happen. On the other hand , older engineers don't always help themselves. In my last job, a colleague who was early 50s kept talking about 1980s TV sitcoms (which included various racist, sexist content that would at least come with a warning nowadays). This made him look like a dinosaur to younger colleagues. I generally find, younger engineers can sometimes play it a bit cocky and make out they know more than their older colleagues but when you show 'em a few old skool command line tricks or rapid edits in vim, and generally be a nice person to pair with, they quickly appreciate what you've got to offer, and the combination of youth and experience can be a great team dynamic. But its true I guess, people shouldn't have to get past initial prejudice based on age, and work harder to prove themselves, same as they shouldn't have to for race, gender etc.
> This made him look like a dinosaur to younger colleagues.
Can't be helped. We will always look like that, no matter how hard we try to "fit in."[0]
Tension between younger and older folks is as old as humankind.
I think that a team is best served, with a combination of youthful enthusiasm and creativity, and experienced caution and completionism. I think that the whole from these teams, is greater than the sum of the parts.
If we have just older folks, stuff gets done, but it may not be that interesting.
If we have nothing but younger folks, we have ... FTX.
The difference, this time around, seems to be extremely young C-suite execs.
In "the old days," when we watched Archie Bunker rag The Meathead, companies were generally run by folks in their fifties (there were plenty of problems -not exactly the Halcyon days). These folks didn't have anything against older folks, except, maybe, that we were relatively expensive (but not, compared to these days). If they discriminated against older folks, it wasn't personal.
These days, we have people in their twenties and thirties, at the top, and their discrimination is personal. They also make it OK for their employees to have the same attitude.
What's that saying that the "olds" have? "The fish rots from the head down."?
My experience has been similar at other large companies (MSFT, Google). My advice is to look for founder-led teams (not companies). If there are plenty of founders still on the team, they will likely care a lot about the product and code and hold everyone to high standards which make the work easier in the long run. On the other hand when I've been on teams where all the original founders left, it's been a constant struggle to make anyone care about anything. At that point there is no ownership and most people are there for a paycheck. Lots of sloppy code gets shipped, everyone approves anything that looks like code, and eventually adding a single line anywhere feels like playing Russian roulette. Extreme short term thinking takes hold and no one cares that in 2-3 years the code will be unmaintainable because they don't plan to be on the team by then.
Yup, and the other flipside is devs being extremely nitpicky and regurgitating irrelevant Amazon principles BS in code reviews because they're trying to get promoted.
That one is my favorite. At Amazon, "bias for action" is a knee-jerk managerial phrase that typically means, "I'm going to spew ambiguous/ignorant bullshit into the room and you do something meaningful with it."
It was much better 5+ years ago. It's impossible to maintain high code standards when a company grows as fast as Google has. Eventually the law of large numbers applies. It's probably still better due to starting from a stronger base, but they are all converging.
There’s really a percent growth rate above which you just can’t build watchers fast enough to keep making sure the new people integrate. The more time the watchers spend trying to watch the less time they have to demonstrate the culture by their actions, and less time to reduce code rot personally.
So you either hope that others will be inspired to do better by your example, or you become a gatekeeper who loses respect because nobody likes to be lectured by someone who isn’t walking the walk.
I spent 10 years at Google (left in April) and I do think it's much better than how Amazon is described in this thread. Not perfect, but really still quite good.
Why better than Meta? My impression was that Meta have outstanding engineering (regardless of what else is happening with their products and company direction).
Meta has a more a hacker culture "move fast and break things". On the other hand, Google is well-known for its engineering culture (they wrote numerous articles and books about it).
I think you might have cause and effect mixed up a bit there. I’ve absolutely worked with people who left because they were unappreciated, and they were unappreciated because they wrote unmaintainable code and were defensive about it. Being a quitter and writing shitty code are very compatible personality traits.
To an extent then, if the original authors are still around, they were either appreciated or feared, but you probably won’t know which just from the interview. If they split it may be good riddance.
I do have a couple of mystery cases where I believe that someone who previously thought very highly of their own work had a change of heart, and realizing what they’ve done and how hard it would be to fix it, have decided it would be easier to start over someplace else. That may be true but I would recommend that it’s a character building exercise if you at least try to clean up your mess before leaving. There will be other messes, from other blind spots.
>adding a single line anywhere feels like playing Russian roulette
I worked at a company where all 4 founders were still there (15 years in) and that was very much our reality. I think the founders still cared, but that didn't matter - beyond a certain size (>100 employees), the majority of the people in the org didn't care the same way.
Jamie Zawinski thought there were two kinds of people. Those who want to make a company successful, and those who want to work at a successful company. He was salty enough about what happened at Netscape Corp that he quit the industry and opened a bar.
Sigh... another loss at the hands of, "Code is Art!"
It's not art. Art requires no function, it exists as a representation. (Professional) code, and software engineering generally, must produce valuable things, and sometimes the process of making those valuable things isn't "fun", but that's never been part of the equation. Aligning "fun" with "valuable" is nearly impossible, and when you try you're dooming yourself to almost certain failure.
This is really burnout, and the mental health effects that cause burnout. I'm sorry the author has gotten to this place, but wrapping it up in "passion" is a disguise.
A few years ago I visited New York City and went to the MoMA. At the time, they had an entire floor dedicated to mid century design. I was always particularly taken with the Kennedy Armchair and the Eames lounge chair. Those pieces to me are undoubtedly art. But they also serve a function. They're chairs.
A business analyst once told me - "you programmers have it so easy. It either works or it doesn't!" as if the vague 2 sentence requirement from their spreadsheet that I have to turn in reality doesn't require any creativity to implement.
But what about the edge case they missed? How should I design the control flow of this? How should it handle errors? In a lot of cases, how should it look, because I only received a wireframe? Should I write this as a class or a function? Can I write this code in a way this is easy to read and performant? What if it's a really specific piece of business logic, that doesn't make sense unless you know the reasoning for it?
Good software is art the way a well designed car or watch is art. They exist to accomplish a function, but the path to get there often requires quite a bit of creativity and expression. Your assertion that art requires no function doesn't make sense.
Is a dish or menu designed by a chef not art? Is a house designed by an architect not art? I feel sorry for people who can't see the art in those things. And especially sorry for people who can't see the art in writing code.
So you've listed houses, cars, watches, food. All these things indeed serve a purpose as well as allow for creativity and expression.
But the difference is, all those things have sale values that DIFFER based on that creativity and beauty. Code does not. My customer does not care if my backend code is beautiful, and neither does my CEO because it won't make more money.
> Is a dish or menu designed by a chef not art? Is a house designed by an architect not art?
Of course I see the art in those things, and in response i'll pay a lot more for a Frank Lloyd Wright than a mcmansion. But the customer of my B2B CRUD app does not care if my code looks like a FLW or a mcmansion. They care if it works.
>all those things have sale values that DIFFER based on that creativity and beauty. Code does not.
I wouldn't be so sure about that. I certainly prefer to use software that has a nice interface, is fast and responsive and has thoughtful features. I remember the first time my iPhone opened a pop up at just the right time asking if I wanted to share a wifi password with my Mac. Wow! Delightful. And I am willing to pay more money for such things.
>But the customer of my B2B CRUD app does not care if my code looks like a FLW or a mcmansion. They care if it works.
What is "works"? I'm not being disingenuous here. The software development process is often plagued by things like scope creep, unreasonable asks from stakeholders, cutting things for budget or time etc...
"works" is a subjective concept in your hypothetical customer's mind. Likely molded by you or your project manager setting expectations, pushing back on feature requests, etc...
My real point is, it's not just the value of the consumer or "customer". But the artist as well. It's the satisfaction you can get from designing a performant service that handles requirements and has the capacity for future expansion, etc...
Just because some people are philistines doesn't make the creation any less valuable as a piece of art.
I’m sorry but this is complete nonsense; there is no pleasure derived from viewing “beautiful” code from the perspective of the consumer such as in your other examples.
A program is not like a car, a chair, or a house, and if any of this were brought up in a design meeting for code, you’d rightfully be laughed out of the meeting.
But it’s fine to disagree about these things up to the moment where you attempt to slow progress towards delivery for these values. At that point, the point at which functionality is hindered in any way by your artistry, are you now a problem on a development team.
There are plenty of productive ways to deal with problems, but make no mistake, on any competent software team you will be disabused of this “art” notion, not the other way around.
>I’m sorry but this is complete nonsense; there is no pleasure derived from viewing “beautiful” code from the perspective of the consumer such as in your other examples.
You're taking it too literally. "viewing" in the case of software is "using". When I'm using a well designed, performant piece of software I can feel the art in it. When I'm using something thrown together I can feel that too.
"Performant" is not the same as "well designed". What you feel is a pleasant UX, which is art, or closer to it at least, and would fit your analogy drawn against a chair or a car.
However what we're discussing here is how the code and otherwise hidden implementation is designed and built. OP is not a UX designer, OP is a software engineer, and should not waste time building things that are "beautiful" in that capacity.
UX and systems design are almost entirely unrelated, and should not spend time in one another's domain.
Your writing reminds me a lot of "Zen and the Art of Motorcycle Maintenance".
'The real cycle you're working on is a cycle called yourself. The machine that appears to be "out there" and the person that appears to be "in here" are not two separate things. They grow toward Quality or fall away from Quality together.'
> My customer does not care if my backend code is beautiful
Apple can only sustain the high margins they do because you’re wrong. Some people care a great deal. Some of those people have deep pockets. If none of your customers seem to feel that way it’s probably because they are Apple customers.
But you also have companies like Anker and at certain points in their history Sony, Samsung and perhaps LG. They got it. At least for a while.
People here don’t complain about Apple for silly reasons as much as they used to. But I’ve been laughing at people jealously predicting their imminent demise while cashing dividend checks ever since their shares were $40 pre-split (7x and counting IIRC). I’m going to get to retire at least a couple years early just on AAPL, even having done dollar cost averaging.
> B2B crud app
Ah. Businesses have a weird split brain problem and it’s difficult or impossible there. They want custom software for less than or around the price of off the shelf solutions. The OTS ones can be beautiful and occasionally get away with it. If you care about art of even ergonomics you’re gonna have a bad time. I encourage you to seek out a new vertical.
> Apple can only sustain the high margins they do because you’re wrong. Some people care a great deal.
Are you suggesting that some people care a great deal about how pretty Apple's code appears, despite having zero way of knowing anything at all about that as a consumer?
They do start to care when it stops working or it takes forever to change due to it being a mess. This is the natural state most projects end up at unless the devs are putting in a ton of extra effort to keep it clean, organized and maintained. Devs that are thinking about keeping things beautiful, succinct and performant are much more likely to do that.
My Observer bias of late has been about open or nearly open feedback loops. People don’t adjust their strategies when the consequences come due later. I’ve been on a number of projects and seen a few more where management keeps turning screws to get stuff faster and then goes pikachu face when the whole thing grinds to a halt. Few if any of them see themselves as the agents of chaos in this situation. It’s not their fault, it’s someone else’s. And even if you can document it, it doesn’t sink in. Because that was something they did years ago and we don’t think about old things.
> as if the vague 2 sentence requirement from their spreadsheet that I have to turn in reality doesn't require any creativity to implement
Do you really need creativity, though? Unless you’re working in uncharted territory and you’re just building some CRUD app like 90%+ of software in the world, you don’t. Software engineering patterns exist, and even so much of the innovation that supposedly came out in the last 5 years are mere iterations of the same patterns.
And the purpose of being creative with a solution is not creativity for creativity’s sake. Clarity, maintainability, scalability, should strictly be the primary goal of every line of code that you are writing. I wouldn’t debate that a code that is so clear and intuitive and easy to scale for so many people has achieved some degree of art, but that should be a side-effect, not the goal in itself.
I'm getting a little tired of all the "creativity" in UI myself, a.k.a. reinventing the wheel.
I use a dozen different-looking web interfaces per day, often internally inconsistent in the same application. Here, it's OK Cancel, there it's Cancel OK, another one is Yes No.
There are two buttons on the "modal" dialog in a web page. One is a different color. Does pressing Return mean anything? Escape? Can I tab through them? Does the modal with one required text field (e.g. 2FA entry) focus that field? Does Ctrl-V work? Is it some bizarre custom design that can't keep up with a normal rate of typing?
Will it allow my password manager to autofill fields? I saw an interface recently using Angular that didn't even provide name or id on the text fields. Thanks.
The desktop app side of things is equally annoying. Either it's electron/nwjs, or it's attempting to look cool instead of just following the guidelines.
You know what takes creativity? Shoe-horning new code/requirements into code that is not abstracted in any sane way. The end result doesn't make me look like an artist though, it makes me look like an asshole for not rewriting the whole mess. But alas, that wasn't factored into the current epic.
Would you pay more for a house where the internal plumbing is beautifully laid out? Not functionally better, just more beautiful (in some abstract sense).
It doesn’t sound like burnout to me. It sounds more like organizational issues that have left the author with a very limited amount of time to work on a sluggish product they can’t take pride in. The name brand and pay sometimes just don’t make up for that especially for some people.
There’s nothing wrong with wanting to take pride in your work and to have working conditions that facilitate that. Get enough people together who feel that way and can find something useful to build and you get a huge competitive advantage. Software that is useful and performant is like gold but takes a lot of extra effort and passion to keep it that way.
You’re right about the “fun” part though. Making great software (and art) is hard work! “Satisfying” is probably a better metric to shoot for.
Coding is pretty straightforwardly a craft - something that is done for use value but allows for artistic expression and a convergence of form and function. It isn't art.
However, I disagree that aiming to get enjoyment out of a coding job is necessarily a mistake. But it does often have to be weighed against competing factors like ease of getting a job and pay.
He compared it to writing a novel. A novel can serve more functions than just being a representation, such as a puzzle, an entertainment device, an empathy exercise or a learning tool.
I read it more simply like, "Code is a craft that deserves careful attention."
I sometimes hire marketing writers where I work (I have many years of experience as a marketing writer). One of the most important moments in the first interview is when I explain that what they will do is not creative writing.
We can find ways to do our writing jobs creatively, but fundamentally we type to get things done. We write to influence people. It’s not art, not even a little bit.
Nothing requires a function, you can have code without a function.
I think good art (to me) requires some function. And code is art. Like architecture , or furniture design, or gardening, or dance, or many art where the design and precision and purpose is all tied into what the artist is doing.
I don’t think coding is pointless and think that coding for the purpose of being beautiful can be dangerous and painful to support or refactor. But I’ve read a lot of code and appreciated it on some aesthetic level and remember thinking, “that’s neat.”
Beauty doesn't put food on the table, is my point, and I'm beyond sick of working with people who put "code beauty" before "code usefulness".
Ugly code that works pays bills. Beautiful code that doesn't work is, in a very literal sense, worthless.
I'm not against writing clean code (there's utility in that), but I feel like some folks lose the plot and quit their jobs when they can't make "beautiful" things anymore, and I think that's more often a sign of burnout than it is a desire to "return" to creating art.
Why is the dichotomy "ugly code that works" and "beautiful code that doesn't work"
Surely there's also "ugly code that doesn't work" and "beautiful code that does work." And I'd argue there's a lot of in between as well. Unless the software literally does exactly what it needs to do, no more, no less then there's also the variants of "works well, somewhat well, etc."
>Ugly code that works pays bills. Beautiful code that doesn't work is, in a very literal sense, worthless
what about good code that work? All places where I see your argument prevail, ended up to be not just 'ugly code', but unmaintainable code. And systems that cannot evolve at all.
Personally I don't call coding an art, but I've seen countless times how people people choose bad solution even though better one costs exactly the same, and in a long run - actually cheaper. And they _always_ use this argument - 'code is just a tool, if it solves the problem, it is good'. And then they either leave or have to spend weekends to write even more dirty code just to solve problem they wouldn't have in the first place if they spent a little bit more time thinking about the code.
Don’t mistake “beautiful” code for “clean” code. Clean code has value in its ease of maintenance, it’s easy to work with. “Beautiful” code falls into the completely arbitrary land of “what is beautiful?”
Wars have been fought over defining beauty, but the important thing to note is that beauty is orthogonal to function. Clean code has clear criteria and value, beautiful code does not.
Code isn't Art, it's Craft, or, to use the current buzzword "artisinal".
Software development isn't "engineering" by a long stretch. There is still another 40-50 years of development and consolidation before it could be considered the equivalent of other engineering practice.
I've lived through numerous lifecycles of "this time for sure" regarding software as engineering, from "modular programming" to "CASE" to "iterative development" to "UML" to (god help us all) "Agile".
What a lot of practitioners think of as "engineering" is actually "project management". Engineering is about defined practise, measurable inputs and outcomes, responsibility for signoffs and approvals that ensure that the defined practise has been followed.
I hate this whole idea that coding is a creative (in the artistic sense) activity.
There's a limited number of solutions to a programming task, it's about finding the one that meets most constraints, it's about cost, effort, trade offs analysis and finally implementation.
Getting code to do the right thing is an engineering activity. Getting code to make sense, to communicate an accurate and useful sense of the problem and solution space to yourself and others across time and change, is a humanities activity.
Thank you! Writing code is a creative exercise not because there are infinite solutions to a given problem but because your solution must be one that clearly communicates it’s own intent both to the machine and to the next humans who read it and give certainty to those humans that they can make changes without tripping over load bearing coconuts.
It stops being a humanities activity when you shape your comms strategy around the science behind how the human mind learns, as you should with code documentation and even with the way that your code reads.
It is once you enter the real world of: costs (e.g. has to take N people X amount of time), resources available (may run only in a specific environment), performance requirements (has to solve the problem in a certain time), tools availables (you may have to solve it using a specific set of technologies), conventions (from coding style to required coverage to documentation), security and many other less measurable constraints (readability and maintainability) the pool of solutions to a certain problem is small, extremely small.
Coding is mostly optimizing and compromising between all those factors, rather than staring at the wall waiting for inspiration expecting the solution is an act of poetry.
This is not to say that engineering is void of creativity, some sort of beauty, cleverness, etc, it obviously is, but it's not the same kind of creativity involved when painting or writing music.
Art has the goal of touching, moving and expressing the author's self.
That's just not the goal of writing a form, a serializer, a validator, a logger, a queue, a shader or a git hook.
Tell me you don't know what knowledge it takes to compose or work on a painting without telling me. Either of those tasks that we class as art also aren't simply "staring at a wall" waiting for inspiration either.
In either case you didn't specify constraints and even with those constraints it's still technically false unless you're limiting yourself to a non-turing form of computation. I'll take all your tasks with their constraints and add no-op operations onto them to make them ugly but still functional. I'll add in all kinds of ill-thought out logic that will never be reached or is "functionally" useless.
And even an "artist" has these "real world" constraints which are "must sell before starving to death" or the work won't be "completed"
> This is not counting the weeks where I was “on call” and forced to drop all of this to work on a backlog of DevOps related issues.
I want to say sorry at the beginning if I misread what the author (you?) intended when you wrote this. And I'm coming at this from someone who is not a developer and who, compared to the lofty Software Engineer salaries in this industry, feels underpaid and underappreciated for doing the operations/administration work.
With that said: I truly wish those who are leaving their roles where this kind of requirement exists--feels "forced"--would speak more loudly about the need to have actual people doing system administration work. This is especially true, I feel, in large companies who operate "the Cloud" and who ought to know better; someone needs to be doing that work and it can't always, or usually, be the people who are writing the features and implementing the updates to the core product you are selling.
But what seems like has happened is everyone in the tech industry has forgotten it, or named it "Site Reliability Engineer" with a job of 55% failure analysis, 35% coding, and 10% "are the infrastructure and products actually online and functional." And then this role gets looked down on and paid less because the people in the role--people like me--are not seen as "delivering" "value".
Which culminates in the proper software developers seeing it as a thing they are forced to do, resulting in disdain, and furthering the cycle.
How, pray tell, is someone utterly unfamiliar with a code base supposed to be able to deal with unforeseen issues in production for that service?
(Worse, the incentives for improvements to production quality, and thus on-call quality, are utterly mis-aligned.)
Ironically, I say this as someone presently on-call for a bunch of stuff for which I am utterly unfamiliar with the code base of, and have no time to become familiar with. It's going predictably badly.
So just page that person directly (the dev, for the purposes of this argument).
(And because I feel like this is bound to draw a strawman, steelman this: while there are definitely pages that might not get routed to BE eng in particular, assume we're routing pages to the person responsible for that system. I.e., in the face of infrastructural problems, those pages get routed to something like "infra eng", although IME there's very little that can be done with those in the middle of the night…)
The SRE role comes from Google, where they are neither looked down on nor underpaid. Perhaps other companies have corrupted that title to mean something else.
Ideally incident handling should "just" be rolling back the broken change. Fixing the problem should be done in the morning with no time pressure, not in the middle of the night half asleep with customers on the other side of the world yelling at you. Of course it's not always that simple, but most of the time that's what on call should be about
It would be nice if things only broke during "business" hours and didn't have real world impact. Nevermind impact millions of people around the world. But if you look at the customers of say code that is running cloud infrastructure it is running airlines reservations/checkins, government workloads, banks, hospitals, critical infrastructure, netflix, gaming services. That's a lot of things that can't typically wait for morning.
This is the pat answer Amazon gives to defend this absurd practice, but it breaks down really easily.
>If your code breaks something, you should fix that code. Who else should?
What if it wasn't my code, but code written by someone 3 years ago who quit because most people only work at the company for 2 years? And it's in a part of the codebase I've never touched. That's a much more likely scenario.
The problem is that he has a big pile of half-working spaghetti code that he never has time to touch except when it malfunctions in the middle of the night
The problem behind that is that Amazon is a completely dysfunctional corporate hellscape. Like TFA said, you just don't have time or resources to actually fix things
You should join Amazon and do that, and you can come back here and apologize in a couple of years when you get pipped for wasting too much time on legacy code
I did not feel super sympathetic when reading the breakdown of percentages (with only 10% "actually writing code"). Yeah, those specific percentages were a bit egregious, but working on a big project it's not uncommon at all to spend more time thinking and talking things through than to actually write code. There's a lot of complexity to be tamed, and in my experience that can be a very enjoyable part of actually engineering.
This however:
> And the worst part is, I feel like the product we were shipping was really bad. It was slow and kludgy. It was the opposite of the qualities I value in software products. I didn’t feel like I was adding any value, and I was miserable.
Yeah, there is absolutely no excuse for that. And if that is the case about the product you're working on, what I said just before about "enjoyable engineering" will never apply. Good for getting out of there.
>There's a lot of complexity to be tamed, and in my experience that can be a very enjoyable part of actually engineering
In my experience there is almost no correlation between complexity of the project and time needed to do 'non-technical' work. Quite contrary, the most complex things I did as a dev required me to do a lot of coding - to prototype, to experiment, to load testing and so on.
And at the same time the easiest project in the larger corporations required me to write down design docs for trivial stuff and spent months in meetings to 'align' with people, who don't have time to even read your design doc, but want you to have bi-weekly meeting about it.
So no, I don't buy this excuse anymore. Unless you are building space ship or do very unique work, there shouldn't be any need to spend 80% of your time in non-strictly-technical work.
I fundamentally disagree with the notion that design and discussion, of often very detailed technical aspects, is "non-technical work" of any kind, or that sitting down and writing code is the only "technical work".
Doing design and discussion without writing code just feels like deliberately hampering yourself. Code is the best language we have for describing things unambiguously! Weeks of plain-English discussion can save you hours of prototyping.
That works in mostly self-contained projects. As soon as you interface with other teams, or even hardware that is still in development, you better make sure that everyone is aligned on the critical details before plowing ahead. Otherwise it can get very expensive (in terms of time, money, and motivation) to prototype yourself into a corner.
Just this week I witnessed some team having built feature without consulting another critical team, and now it has to be reworked and shipped at a much later date.
> As soon as you interface with other teams, or even hardware that is still in development, you better make sure that everyone is aligned on the critical details before plowing ahead. Otherwise it can get very expensive (in terms of time, money, and motivation) to prototype yourself into a corner.
Even then, writing a prototype is often a quicker and more effective way to flush out disagreement than endless discussion.
But "endless" discussions or disagreements weren't implied at all. For a sufficiently large and complex project, where every participant (individual or entire teams) is only able to fully grasp a relatively small piece of it[1], there is sometimes just a lot of details to talk through, even if all participants are in full agreement and perfectly willing throughout that entire process.
To stay in the embedded world for an example, sometimes the actual code is just a few dozen lines with a handful of assembly instructions, or even less. But the danger and potential pain those lines of codes might incur on countless involved subsystems, if not properly thought out, even if seemingly "working" at first, can be immense.
[1] To give a sense of scale of complexity in the embedded world for example, the ARM Architecture Reference Manual alone is well over 10000 pages now. And that isn't any actual implementation, and only the "main CPU" itself!
Maybe so, so long as we disagree with the notion that that the most senior engineers, the ones who are having "discussions" and making technical decisions, should not have to be active in the code, work with the development teams, and have a deep technical understanding about it works today.
They don't have to have the highest commit counts, but if they're not the ones the rest of the development team look to for help solving critical bugs and advice about new features and designs etc., then they have no business evaluating technical concerns.
Okay, just making sure. I've had experiences where people making technical decisions hadn't written a serious line of code for 10 years. Maybe they used to be good, but they wound up resting on their laurels. They frequently do make great decisions. Some can just muddle though things if they listen to the real technical people, the ones who are incapable of even that are a disaster.
The problem is essentially the following: in a know-all situation (prototype, reduced domain, startup,..) situation it is easy to ditch the 90%. No sync needed, no docs needed, no tests needed, ...
Once you scale up to large systems (ignore complexity) with hundreds of developers then you do and can not know it all. At that moment the churn starts.
And unfortunately, once you get big, regulators, lawyers, audits and other things become a drama as well.
Amazon's internal tooling really is the ninth circle of hell. It seems unsalvageable too; ostensibly the teams are supposed to be customer obsessed and the developers are the customer, but there's no accountability for anything to work well at all.
Is this just me or aside from 40% spent on tooling (which I don't know what the deal is in this case but tbh I've seen some folks really spent their time inefficiently trying to make everything perfect) the breakdown looks completely normal for an established project at medium to big corp?
While it's not just you, and there are lots of folks who think that's just fine: if someone hires a dev and then make them spend only 10% doing the job they applied for, whoever hired them lied about the job, and the product they're hiring for has fundamental problems that you can either commit to solve, or burn bodies over in the hopes that you get transferred out to a different position and it's not your problem anymore. One of those may be good for business, but it's not the one that makes for a good company.
So 20% spent on tests and another 20% on meetings where presumably you decide what to build is not part of the job they applied for? Nonsense! If we’re doing hot takes here there’s no way everyone else on his team is spending 40% of their time on tooling without management doing anything about it
What a lovely thought, I do hope that's true, but everything I've seen about this story and people replying to it points to "no, actually, everyone else apparently spends 40% on fighting their internal tooling, too, and not a single manager seems to give a fuck". So there's that.
It's really difficult getting hundreds or thousands of people working together. Tooling, communication, etc become really critical. Otherwise, it's just chaos. As a result, you spend a lot less time "actually working" and a lot more time "collaborating" and "project managing". Spending less than 50% of your time on "actually working" is very normal in large companies
Was thinking that same. The 40% spent on tooling to deploy code and of course on call time is never fun, but the other time seems normal. But leaving only 10% of real development is not great and the author can probably do better. It is a good reminder that if you get to do greenfield development or generally enjoy your work, don't be quick to jump ship just for higher pay/prestige.
I left Amazon in June of this year, in a similar-ish position to the author. I didn't have anywhere near that level of problems with the internal tooling[0], my frustrations were more related to seeing poor product prioritization decisions, neglect of tech debt, and repeated inefficiencies and mistakes in org-wide projects. Nonetheless, the feeling choosing to prioritize one's own mental health and recovery really resonates. Good for you, author! Take a break, get back in touch with yourself and what you care about, prioritize your joy, and I wish you all the best of luck in your next project!
[0] which I've found to be pretty-to-extremely-good compared with what I've discovered since leaving, and would actively welcome new perspectives on what flaws I'm missing.
Curious, did you go somewhere that at least matched your salary? I know Amazon compensates very well, so much so that it's probably difficult to find something else unless it's another FAANG company. I have a friend working there who basically deals with all the not-so-good things mostly because the compensation is so good.
Haven't moved on yet (intentionally took some time off to relax and refresh), but I expect I'll have to take a bit of a pay cut.
> I know Amazon compensates very well
That's not been my understanding - I thought it was noticeably below other FAANG companies (at least until the pay hike earlier this year)? I have only just started reaching out to possible employers so I don't really have any points of comparison.
Amazon's compensation wasn't anything special. The first two years were awesome because I got a $100K+ sign-on bonus prorated and tacked onto my paycheck every month. Once my first two years ended, my regular salary was pretty lackluster and I was easily able to find a job making a lot more elsewhere.
It's all about the RSUs. That all kicks in after year 2. If you're a lower level maybe you don't feel it as much. But for a high performing L6+ RSUs typically are -- or at least were designed to be -- the majority of your income compared to base pay.
I was an L6 and, at the time, my RSUs certainly gave me pause for leaving, but in the end the urgency of my mental health won out. My timescale was dictated by getting my Green Card, not (primarily) by finances.
yeah...different story now. The target comp numbers ) assume 15% stock growth per year from when it's allocated to you. That model is now broke for the first time in a decade+. How it gets fixed... I'm not sure, but, there is history of cash payments to compensate.
Point being, if you're not making more in year 3 than year 2 -- something is broke and/or your manager wasn't supporting you appropriately, or not hitting high enough on the performance ladder.
Why did you waste 10 years of your life there if it was that bad to you? Legit curious here. I can’t imagine spending more than a couple months in an awful situation.
> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc…
This gives the lie to Jeff Bezos' whole "Day 1" philosophy. If you're at a Day 1 company you don't spend 40% of your time fighting with the crappy internal tools.
Not sure what the Amzn situation is but something I've noticed at few previous gigs that there're some folks who claim they spent significant time wrestling with tooling but when you actually go in and make them formulate the issue it's either something really obvious or a 10-min google/github search like 9/10 times. These also weren't coming from people who I considered lazy or unintelligent (well, for the most part) so I can't really explain it logically. I suspect it may have something to do with motivation or cultural issues ("not my job" sort of folks).
I've seen people complain about it who were wrong, but I've also seen people fail to complain about it when they really should have been.
I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
Your example does not sound as outrageous as you seem to think - I can imagine a number of scenarios where this would be totally acceptable solution assuming we’re talking about software engineers here. Consider the fact that most startups don’t even have dedicated teams running CI unlike at fangs and it’s up to devs to set everything up. Yet I don’t really hear about folks complaining about internal tooling at early startups
The fact that you don't think that's ridiculous is a great example of terrible tooling be accepted as "normal."
Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for.
But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day).
If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
If you're smaller you'd hopefully use something like Datadog, where you can easily ingest logs and have alert triggers from log lines. It's probably harder at bigger firms, where internal politics get in the way of having nice things.
Or you reach out to the owning teams/oncalls and half the time they have no idea how their own code/product works for anything nontrivial because most of it was written several generations of attrition ago.
When I worked at google you could =) j/k moma was terrible but most issues were easily searchable via a combination of email list/bug tracker/codesearch
Haha, I found this quite funny :) I'm sure there's an element of truth to it considering how complicated the code base must be for the various services
I quit Amazon a couple weeks ago to start a startup, and I felt like I was reading my own story at some points. The tooling was the biggest thing dragging me back. It's hard to get excited about what you are building when you have to wait 2-4 minutes to preview your changes when most industry standard tools / stacks can do it instantly.
I left this feedback a bunch of times with different people in the org and I really hope they scrap their janky "frameworks" and dev tooling. They should just move to more standard open source tools options that evolve and get better quicker than barely maintained internal tooling.
Open Source tools also have bigger communities and resources online to debug and solve issues, compared to mediocre documentation from internal wiki. If absolutely needed, make thin wrappers above open source tools. Some other teams within Amazon had the luxury of using better tools, but I bet a lot of people are in the shoes I was in.
I hated the internal tooling so much that I switched to a manager role almost immediately and only went back to IC when I was sufficiently situated to not have to code anything that wasn't a flashy proof-of-concept.
I suggest, in this discussion, to not consider a single company a real, monolithic entity, where everything works the same way everywhere on the planet.
I was at AWS 2008-2014, two years each in EMEA, APAC, Americas. I didn't contribute to code directly, but I built my own software demos (I was the tech evangelist for these regions).
My experience has been vastly different between these 3 "regions", and also when interacting with HQ in Seattle (which I visited very frequently).
There are certain things that unfortunately heavily affect the company - the internal tools were already considered quite bad back then, and I'm not surprised it hasn't changed much.
There are others, though, that in my view are even more important. I had a total of 4 managers, and three of them were amazing, average, and quite bad.
But in all honesty, I was probably not perfect all the time (real modesty, uh? :D), and it might be that the "bad" manager and me were just a poor match, or that I failed to understand things that made me perceive him as bad.
I will never work for someone else ever again in my life (I can afford to, both economically and career-wise; in fact, I just launched a VC fund in Europe), and I love that. But if I had to go back to a corporate job, my single best piece of advice would be: pick your manager wisely, especially if you're early in your career. Everything else is important, yes, but your manager is going to have a huge influence on how you develop, how happy you are, etc.
If you're curious, this is how I got hired by Amazon back in 2008 [0]. I find that post a bit naive in retrospect, but I also wrote it ~15 years ago.
To be completely honest, what he describes seems to be what I've run into at nearly every stop I've been at as a consultant. I honestly never thought of it as hell, but I've also never been in a much better, tech tooling-friendly environment like at one of the FAAMNGs.
I suspect the people who end up leaving those environments for a more normal F500 company are going to be in for a shock in terms of work ergonomics and pay. This guy was lucky that he was able to squirrel away so much tech-worker money to be able to quit without a backup plan.
No voluntary layoff plan you could take advantage of to get some severance pay? Might have been worth holding off a week to see if they rolled one of those out. Water under the bridge now, though.
Yup - I heard about this from a friend of mine. The offer seemed pretty good: lump sum 4 - 9 months base salary depending on tenure, extra for COBRA for 12 weeks, benefits through end of the year, etc.
This mirrors my experience with my prestigious job I had at SiemensVDO in 2005/2006. Same stuff of wasting time on anything else than coding. He said 10% of his time was coding. I suspect mine was like 20% and I felt miserable after a year. The honeymoon of getting to write embedded code for real cars, cars that would be driven by millions of people only lasted like 3 to 4 months. After that the job had nothing else to teach me and over a little year later I quit too, with a total of one month shy of 1.5 years there.
I spent just over a year up here at the Seattle mothership back in the mid-20-teens. I met a few really smart / kind people that I'm still friends with (I even liked my boss, who I'm going to have coffee with after the holiday craziness.) I last a year. Most everyone I know there I know left.
My experience was similar. On the first day of orientation we were trying to build an environment to run some tests in. But some aspect of it was borked. I got to file a Sev 2 ticket my first day of work. The component that completed the dependency closure was borked (what was that, brazil? can't remember the name.) But the whole company spent two hours not building software.
I love that all teams advertise their capabilities through online services, but man.. it would have been cool if maybe they tested that update to the build system since we were all forced to use it.
The other big hilarious problem was when we ran out of IP addresses for new sites. Long story short someone made a mistake three or four teams up the chain and it only became obvious when we tried to allocate a new routable (public) IP address. The problem was in the DNS team, but we didn't use their service interface directly, so we had to wait while the team upstream of us responded. And then they had to file a ticket with the team upstream of them. And so on.
It always seemed to me that while they were trying to prevent their interfaces from being brittle, they missed out on some opportunities to understand what was going on by looking at data flowing from one team to another. I hope there's SOMEONE in the company doing that, but we never met them.
Not all my work relationships were toxic, but I saw quite a bit of toxicity while there.
I quit the day after my then new boss wore a MAGA hat to work and made comments about killing trans-folk (I'm f2m.) Our HR rep was out on vacation and the main office indicated they weren't interested in doing anything about it. I probably should have sued, but it wasn't clear I had a strong case (he said/xe said). At the very least I could have forced everyone to take another sensitivity training class.
Which is to say... There are some amazing people there. Also some complete shits. If you have a high tolerance for death threats, you'll probably do fine there. Some of the technology is amazing. Some of it is completely bonkers. When things are good, they're amazing and very good. When they're bad, they're hellish.
I completely understand where the author is coming from. I did literally the same thing. I am fortunate to be in a similar circumstance where I have the savings to weather the storm. The only difference is I was working for a highly dysfunctional manager and team. Towards the last days before I resigned, I was nearly on the brink of panic attacks from every meeting I had with my manager. I could see that my skills as a developer in doing actual, meaningful work had severely atrophied in a short period of time. To anyone that had to make difficult decisions like this, do what is best for you and your loved ones because a job shouldn't define you.
"40% of my time trying to tame the bad internal tooling I was forced to use to submit my code" .. I experienced something similar while working for a big Korean multinational. This combined with development practices from the 80ies was (when looking back) the cause of severe physiological issues that took months resolve.
Amazon has a lot of bad internal tools, but this person's experience doesn't match mine (being here for 8 years) at all
> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc…
The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic. Google is better. Most companies aren't.
I have never spent 40% of my time battling internal tools....
> 20% of my time in meetings
Developers complain when they're not invited to meetings, and they complain when they're invited. On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process.
> 20% of my time writing unit tests to hit the 100% coverage requirement of the codebase I worked on.
This makes no sense. This isn't a company mandate, every team is free to determine what code coverage percentage makes sense for them. Give this feedback to your tech lead, nearest Sr. SDE or PE -> 100% test coverage should never be "required"
> 10% of my time tracking down bugs in other team’s codebases for either internal tools or frameworks and trying to get them to acknowledge the problem by filing tickets.
> On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process
Oh yeah I love the multiple hours we have spend every week 'introspecting' processes, just to throw out one of the dozen we'd already defined and add another one. And this 'introspection' typically boils down to the loudest, most ambitious mouthbreathers forcing their BS down everyones throats so that they can jot down their amazing process contributions in their promo doc. Brutal is the right word.
But it's the difference between an individual carpenter making a rocking chair for himself and his family, or maybe making a couple to sell to his friends, and being a structural engineer.
Bikeshedding is not a necessary antipattern to the process, but large software projects absolutely need group collaboration, and a discussion of processes, tools, and best practices.
'Buidling things with code' is not just coding. It can include design, architecture, collaboration, planning etc. There are many high quality, highly complex, open source projects that don't rely on many of the 'processes', meetings, toxic competiveness, thought-policing, and beuracracy found in AWS.
Based on your comment history (which is consists of about 0% technically interesting topics, and about 100% pro-amazon 'tales from a super-senior principal engineer guyyyss') tells me you don't like engineering, you like politics and beuracracy. Which is okay!
>I have never spent 40% of my time battling internal tools....
In my experience, not at Amazon, long tenure employees get used to the quirky tools but the impact on new employees can be massive. Same with bad code bases, bad documentation and so on.
Maybe so, but don't you think I talk to new employees? It's half of my job to support my whole team and deliver through others.
I battled those tools when I started. I watched them get better.
I've seen what new hires struggled with 5 years ago and what they struggle with 1 year ago.
Night and day.
The tools have gotten a lot better.
Here's the other ugly truth: That "40% of struggling with internal tools" may be saving the engineer 300% of time of having to implement the same from scratch themselves. Software engineering isn't all algorithms and data structures. A lot of it is just boilerplate code hooking up A to B. And better leave that boilerplate code to the internal tool that you have to figure out how to configure than implement it yourself.
> Maybe so, but don't you think I talk to new employees?
Everyone below a certain level talks to new employees. Do I believe you take their concerns seriously and actively try to help? Based on my own experience with PEs as well as your comment history I think you absolutely do not.
In general my experience with PEs at Amazon led me to conclude that the vast majority of them are:
- entitled
- lazy
- egotistical
- less technically useful than the average l5 engineer
I really think this is true. Like compare the CR process to Github's PR process. To me, it's horrendous. But to others who are embedded, maybe they love it.
Yeah, comparatively, I find Github's process to be bizarre. Why do I have to fork a repo just to propose a change to it? Why does my "Pull Request" (really it's a Push Request, but it's called a Pull Request because of the 2 repo requirement) have to be associated with a remote repo in the first place?
(I wouldn't say I love the CR process - making multi-package CRs is definitely flawed, and I had a Sage question open with the Builder Team for nearly a year where they admitted as much - but I certainly prefer it to Github's)
But I'm pretty new to the outside world and really keen to understand alternative perspectives. What's good about Github's process to you?
You can do PRs from branches in the same repo on GitHub just fine if you want to. Having them in a separate repo makes more sense because 1) it doesn't require the dev submitting a PR to have anything above and beyond simple read access to the target repo, and 2) there's no concern that someone might depend on WIP branches created in those private forks, so they can be deleted or rebased at will.
The reason why it's called a "pull request" is because you're requesting the owner to pull your changes in; I don't see what this has to do with the number of repos in the picture.
> You can do PRs from branches in the same repo on GitHub just fine if you want to. Having them in a separate repo makes more sense because 1) it doesn't require the dev submitting a PR to have anything above and beyond simple read access to the target repo
Right, that's my point, though - why is there no distinction between "ability to create a real, actual, complete branch on a repo" and "ability to create a 'fake' branch that only exists for the purposes of diffs for a change request"?
For contrast, the flow I was used to inside Amazon was:
1. Clone the repo locally
2. Make my changes locally
3. Run a command that creates a 'fake branch' on the main upstream repo, which is used as the reference for the change. This works even if I don't have push-permissions on the repo (in which case, I can still push my change once it gets approval by clicking a button in the UI, whereupon a service account will push "on my behalf")
Whereas for Github, the process (if you don't have push-permissions) appears to be:
1. Fork the repo to my own Github account
2. Clone that locally (equivalent of 1. above)
3. Make my changes locally (equivalent of 2. above)
4. Push my changes to my own forked repo
5. Run a command (either CLI or UI) that creates a Pull Request from my repo to the target repo
Sure, it's only two extra steps - but I don't see why that friction has to exist in the first place. It gets _much_ worse if your change is open for a while (which will happen to coincide with the case when you don't have push permissions - that is, when you're contributing to code that you don't own), because then you need to resolve rebase locally and push to your fork rather than just being able to update in the UI (Github UI doesn't support rebase-pulls, only merges).
I see your point. FWIW I personally like it because it lets me move between my desktop and my laptop almost seamlessly - just push any changes on one end, and pull them on the other. It also means that, if any (or even all) devices suddenly die on me, whatever I was working on is still safely in my private repo.
But, yes, this does mean pushing things routinely a lot, not just when it's time to make a PR.
Fair enough! I only have a single development machine, so that advantage was invisible for me - but that makes sense!
You could theoretically get the benefits of both approaches, though: have your own personal repo to which you push "in-progress" commits for durability and portability, but maintain Amazon's tooling which generates PRs with a diff between a _local_ commit and the target (by, behind the scenes, generating the ephemeral fork from which to Request a Pull), and permitting updates to that PR from local (not necessarily "pushed to an online repo") commits. That's _still_ advantageous over GitHub's model, because:
* If you don't want to have a personal repo, you don't have to
* Even if you do, the process of updating a PR is simpler and more flexible when executed purely with local Git commands rather than by manipulating a remote repo
Fair enough! I thought I remembered it being a (likewise invisible) branch on the upstream repo, not a separate repo entirely - but I've been out for a while and might be misremembering. Regardless, as you say, it's immaterial - so long as the tooling makes that invisible, the UX is the same.
I don't think Github's PR process is that great. E.g. it misses the ability to diff 2 revisions of a CR if there is only one commit - which would happen if the next revision is a fixup and doesn't create a new commit. I usually used this to review changes incrementally. I find it rather sad that github doesn't have it and one needs to push multiple commits for the sake of making something incrementally reviewable.
> The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic.
THANK YOU. I feel like I'm taking crazy pills whenever I see people bash Builder Tools - Pipelines in particular. Compared with what seems to be available in the Open Source world, it's fucking stupendous, and has silky-smooth integration.
I want to offer a different perspective. I work on EC2 and my experience is completely different.
Lots of good internal tooling. When something critical to us breaks, we usually cut high severity ticket to get other team's attention.
We rarely have actual customer affecting issues in production, there's no PIP threat, promotions feel achievable within a couple years if you really want them and the extra responsibility, automation is very helpful and we're constantly looking for ways to reduce our operational load.
Yes, testing could be better. Yes, we could have more engineers. Yes, oncall could be less stressful. Yes, I spend 10+ hours on meetings every week. Yes, I write a lot of documents and do a lot if ops to debug issues. I have probably merged just 10-20 lines if code in the last couple months.
But EC2 dataplane is very complicated. Every small change needs to be given a lot of thought. Success and scale bring responsibility. Plus, I've learned so much here. I'm never bored.
And my work life balance is good overall. Slightly less than 40 serious pomodoro hours a week except when oncall.
Ec2 is a very stable system and we keep making it more stable for our customers. Do use it. It's not brittle at all. Expect occasional instance failures and design around that with cross-AZ redundancy and you should be golden.
> ... 100% coverage requirement of the codebase ...
Is 100% really practical? Probably in life or finance critical code.
What about those "should never happen" cases? Incredibly hard to trigger because you only know from gut feeling it can happen in some really weird way.
I know it would feel very nice to have 100% coverage, but my gut feeling is that it will also affect the code base in a negative way. Developers might skip catching errors that would be very hard to reach.
People focus on coverage but I think that has diminishing returns. There is a lot of 'art' to building clean reusable code which you can't easily standardize but what you can do is pay attention to things like:
* Build times
* Testing times
* Test Flakiness
* High quality fast unit and integration tests
* High quality fakes
* Fast, high fidelity non-prod environments
* Time from commit to production, number of cherry picks/rollbacks
* Good Coverage, but focus where it matters.
If any of these go out of whack it needs to be a P0-P1 to fix because it really is debt that grows to become unmanageable (not to mention repelling to the best people).
In most companies, even big tech, this is difficult because non-coders set priorities and allocate the resources. Even once-upon-a-time engineers who make it up the org chart find themselves preoccupied with other things and invariably sacrifice these things for features, compliance, conformity, toolchain complexity/sophistication, security, outage/incident scar tissue, etc.
(And maybe rightly so as there are few successful businesses that manage this consistently)
1) Work is work. The percentage breakdowns here don't seem too far off from any other company. It would be interesting to see what the author feels the percentages should be. My gut feeling for anyone willing to walk away from a job is that they probably do need a break for a few months.
2) If you're looking for reasons to be unhappy, or trying to justify unhappiness... you may be depressed. Switching jobs may help, but if you walk around smelling shit all day remember to check your own shoes. Everything is going to smell like shit until you get the help you need to get back to being healthy. Getting help is hard, but it really makes life (and work) a lot easier.
3) Amazon... I've written about this in the past, but I really have a love / hate around my time there. Looking back, lots of talent, lots of cool projects. But I just felt like Amazon was perhaps the worst place I ever worked for people giving praise or sharing appreciation. Amazon... I can't put my finger on it exactly, but it did often feel like their mission was to suck all the fun out of the room.
Been there done that too.The thing that hit me really hard was Software Engineering vs Writing code/Computer Science are two very different things. Yes they overlap in theory and behavior of systems.There's no deny Amazon chooses to amplify the worst parts of Software Engineering today (poor tooling etc), but most places aren't much better and even if it is, its not forever.
Has the tooling gotten dramatically worse in the past 5 years? I willingly left almost exactly 5 years ago because my team's responsibilities had shifted too much, but I remember the tooling being pretty great. Also, everyone being more or less on the same stack also meant that there was a ton of resources to solve problems, just search the wiki for your error message. I still miss how well Odin worked for service credential management, nothing I've used since comes remotely close to how seamless it worked.
Also, I found the PIP process to be overly dramatized relative to my experience. I had a PIP imposed on me a couple years before I willingly left, and I "graduated" out of it with support and cooperation of my manager. Everyone involved seemed to want me to succeed.
Also, when I left, I opted into a second PIP, because HR and my manager wanted me to leave on good terms. Doing so let me get severance while quitting (it was ~3 months worth based on my 5+ year tenure). I'm not sure if this option still works though.
I left about a year ago, but the universality of tools and stack was not a thing in the ~3 years I was there. Retail and AWS are split and use different stacks. Additionally, the company was in a drawn out multi-year process of deprecating old tooling (including Odin), so you had teams or orgs that were in-between these old internal services and new internal services (or off internal services altogether and on "native AWS"). If cross-team interoperability demanded that certain tooling was to be used, many times you could inadvertently hold back or screw over another team by deciding to stay or move to the new tooling.
Not to say it was all bad - some of it was certainly necessary. Especially for security - IMO the systems of Amazon are great with security, but not necessarily the spirit of the company. Without all that bureaucratic annoyance, there 100% would be existential leaks or breaches coming out of Amazon with how fast and loose everybody treated security in the name of "Deliver Results" .
I just laid myself off from Amazon too after 2+ years. I just couldn't take the janky-as-shit internal tooling, the myriad pointless/redundant meetings, and pointlessly stressful and toxic work environment anymore.
Also, I enjoyed my first two managers at Amazon, but my third manager was a tool, which is a systemic problem at Amazon.
I'm glad to have Amazon in the rearview mirror of my career.
This mirrored my (much longer) experience at Amazon a bit. The internal tooling is just absolutely awful, _especially_ the newer stuff NAWS stuff. The log4j incident was an absolute nightmare for many teams in Amazon, and the sad thing is that I don't think any lessons were learned by tech leadership.
One key lesson from these kinds of stories - money gives you options.
Pile that money up high (in something like index funds) while you're earning the big bucks, and there comes a time where you can choose whether you want to keep doing it, or move to the beach and surf and still be young enough to enjoy it.
If it works for you fine, but most of us just find another job and then resign. Few situations are intolerable for a few more months, although Twitter seems to qualify right now. As you said, you have a simpler financial situation that enables more flexibility, good for you!
I quit Amazon over a year ago with my managers encouraging me to stay. I was on a cross road of decisions between being stuck in an engineering work that intensify my depression vs taking the high road and hope for the best in uncharted startup territory.
Engineering at Amazon is not for a fainted heart, and internal tooling is definitely the source of most people's miseries. On multiple occasions, I had to fix external tools at the expense of making progress on my own work, in the name of exhibiting a "deep dive" ethos.
On another hand, however, I think Amazon is a good place to work for people who will eventually get into startups.
20% meetings is two story points per 2 week sprint. I hope you have a recurring task for this. I do this and does it go down like a lead balloon with project managers.
It's all about perception. If I had 2 less story points per sprint then I would get more tasks to fill up those 2 story points. PMs don't like it when the whole day isn't accounted for.
Where did this image of “prestige” come from if everyone has the same stories: the place was dysfunctional and the tooling and processes were mediocre ?
same reasons here that contributed towards deciding leaving Amazon despite it being a dream job entering the industry (i did have a significant "pull factor" to another job that accelerated the timing). when you realize you wouldn't be happy even with the best case, it's time to go. However, i've heard many reports of talented, smart people doing fulfilling, challenging work at Amazon and its probably the luck of the draw on what team you get staffed on. So the case could be made that you couldve simply tried for a transfer.
i suspect that, if you work on a "slow and kludgy product", the responsible thing to do would have been to write a memo laying out why it sucks and what needs to be done. I've seen a rare few folks actually get heard and get the mandate to do what needed to be done. but I know it can feel overwhelming for most and ultimately its the responsibility of the GM/PM to know whether this is true or just the whining of an underling with no context.
also kudos for having the guts/integrity to do this before thanksgiving. make it count!
Same situation here, and I don't work for a big tech but for a little company.
I'm unable to realize how things could change in a similar environment: the sensation is that there's a lot of time wasted in calls and similar activities. And also into the process of "how to do things".
This is very stressful, in particular when what there's to do is clear and glaring.
I can only wish you the best. 6 months ago, I was in your shoes. Today, I have my own startup building applications I am passionate about. I have never been happier. I’m grateful for everything that I’ve learned and endured so far.
I just wanted to tell you that I hope you stay positive even during the dark days and that you maintain your integrity. Truly, I wish you the very best.
Sadly where I work I split my time very similarly and I thought it was a sign of a dysfunctional organization and a sign that I should leave. It's been my only professional software job, so I thought it's likely going to be better somewhere else, but now I am concerned this might be more normal than I thought. Is that so?
Former Amazonian here. The learning is unmatched. I've been in different FAANGs but the way that Amazon pushes one to learn and challenge the status quo is unthinkable. I never saw "no, you can't do that, that system/business/area is sacred". Everything is up for grabs all the time. While there are pathological side effects to this (AWS promotion-per-launch, constantly battle for scope), the amount of knowledge one can gain in different areas from world-class people is pretty incredible. One of my systems at Amazon had more traffic in one day than my system at another FAANG had in 5 years. Working on a system that needs to support 5, 6, 7 MILLION TPs to perform complex operations was a great lesson for me.
Yet the internal tooling has been cited by several here as being one of the major time wasters and reasons they don't want to work there. It seems sacred to me...
I've been at AWS for almost 6 years now. It depends on what you mean by "internal tooling". But considering all the tooling I've used over the years at least 75% of the internal tooling that was in use when I joined is no longer in use as it has been replaced by newer, better versions of what came before.
Things definitely move at a slower pace than at new startups, but systems within AWS still change at a pretty fast pace for such a large organization. A fair number of employees don't stay past two years because of how compensation works, so they don't stay long enough to observe tooling changes, and are left with the assumption that tools are sacred. The reality is that it takes about two years between major rewritten versions of most tools. Then there is a leap forward, stasis for a while while feedback is gathered and the limits of the existing tooling are found, then in roughly two years there is another leap forward, etc. This churn requires work to keep up with, so some teams also fall behind if their product team prioritizes features over keeping up, so they may also be stuck on old versions of tooling for even longer.
So in summary I'd say yes in the short term some of the tools appear sacred. In the long term pretty much every internal tool or framework is discarded and replaced by newer versions on a fairly regular cadence.
I would say that 1) the experience varies by team 2) there's a lot of truth to that. But it's not a matter of being sacred, but the (sad) fact that Amazon being a ruthlessly data-driven business, investment on the tools and on reducing technical debt (real or perceived) is always prioritized down in OP1s.
Correct. If you're a PM on AWS and you launch a service, promotion chances will increase substantially. This is the main reason why the product portfolio is so bloated with services that are minor variations of other, existing services. Someone sees a minor use case there, PRFAQs it, voilà!
If you're thinking of joining big tech, try to join a team that develops an internal tool.
You have as much impact and are exposed to as much scale (maybe not in terms of TPS but in terms of data, etc.), but you usually have much lighter development processes and a much more frequent release cycle. Your ops will also be a lot lighter.
I'm an early career software engineer and am curious - why do people say working "at scale" like it's a good thing? Why is that desirable? What are you doing differently from someone who doesn't work "at scale"?
Making software work for single use is relatively simple, making software that runs at 100% uptime, with large numbers of "users"/"tasks"/etc is an entirely different level of development and design.
The actual functionality is a very small part of software development.
Making it work within its constraints (memory/CPU/network/storage in the small scale, ability to handle peaks while maintaining uptime at a larger scale), as well as dealing with the "non-functionals" like AuthN/AuthZ, obervability, performance, etc is actually the "hard part".
Then add in business "non-functionals" like "minimize CapEx/OpEx", "meet external SLAs with measurable and reportable KPIs", "meet corporate and regulatory standards and requirements" etc.
All of that is much larger than the actual function being developed.
it just means they deal with some obvious and non-obvious problems having to do with being A Big Deal. (there's only a little bragging involved.) the most usual example is scaling up. say you have a website which you run on a server, and it's humming along serving users. everything's great. but then you get popular, and your one lil server can no longer keep up. So you swap that in for a bigger server and everything's great again. but then you get even more popular. and suddenly there don't exist bigger computers. so then you split up the programs on that one computer so that you can use two big ones and serve more people than you could before with one. and then three servers. and so on.
Someone who's dealt with websites at scale before is going to be able to skip the first 20 of that process, so if you're a startup dreaming to get big, you find someone that's worked "at scale" eg Amazon or Google to design and build your systems and avoid some pain points like the website going down because the system can't keep up.
If you look at tools in general as something that's intended to help other people solve some problem - and derive some enjoyment from that aspect of your job as a tool writer - tooling that operates "at scale" usually also helps more people.
IMHO you shouldn't have worked for Amazon in the first place. Most of what they do is more negative than positive for society overall. Good that you're out of there - please consider doing something more redeeming with your skills.
I wouldn't say that AWS is negative. That is seperate to the business of Amazon, which exploits warehouse and delivery employees, but is not unusual for the US.
So twice as much time writing unit tests for your code as writing your code (20% of total vs 10% of total), and that's a complaint? Got it. But, sure, it's everyone elses code that is a shitshow. Ok.
I would like to say, I am going through a very similar experience at Meta. Is there any Big Tech company where internal tooling isn't broken for front-end development?
How was the work/life balance though? Did you find you were able to have downtime and rest or were you constantly worrying about unfinished work and looming deadlines?
I don't think I've ever heard a testimony of Amazon having good work life balance. The best you hear is that some people get lucky and things are "ok" on that front.
Hi, recently ex-Amazonian here. It very much varies team-to-team. I was both careful and lucky enough to always be on teams where work-life balance was actively prioritized, with managers literally telling people to take more PTO, ensuring that stressful oncall experiences were a) compensated with unofficial time-off and b) taken as a prompt to invest in paying down tech debt to avoid repeat experiences. My experiences seem to have been better than the industry average.
But then I saw how some teams (predominantly, but not exclusively, based in Seattle) worked, and....yeah. Overall, not great.
My work life balance at Amazon as an SDE is great but I don't have 24/7 oncall, make of that what you will.
That being said, I agree that the internal tooling at Amazon is not that great. It's actually a place where newer employees tend to clash with older Amazonian because it compares unfavourably with GitHub actions or any modern CI/CD while being significantly better than anything that existed in the early 2010s.
I've heard a lot of testimonies of SDEs maintaining pipelines that are making them miserable.
This is really baffling to me. In fact, since leaving Amazon in June, I've been really frustrated by how much extra work setting up a CI/CD pipeline is in (say) Drone and Argo/Flux than with Amazon's internal tooling. You can set up a standard Amazon pipeline with a single command and answering a few prompts about naming, whereas with the open-source systems you have to hand-craft everything yourself, hack in a way to update the infra repo every time the source repo changes (I blogged more about that here: https://blog.scubbo.org/posts/ci-cd-cd-oh-my/), and it seems like Drone literally doesn't have metrics that indicate broken builds. To say nothing of how well-integrated Pipelines is with alerting and ticketing - yet _another_ integration you would have to build for yourself in OSS-land.
Literally the only advantage I'm aware of for OSS systems is that they allow you to use whatever language, dependencies, build system, etc. that you want - which, yes, fair, if that's something that you need, then that's a deal-breaker, but for the 99.99...% of internal cases that Pipelines works for, it (seems to me to) work flawlessly and smoothly.
What are some unfavourable comparisons that I'm missing?
I've actually really been of two minds about this since leaving AWS, though it's been a few years. On one hand, I remember spending an absurd amount of time debugging esoteric internal tooling errors that were un-googleable. The cases where you were lucky enough to find someone else who had the same issue on the internal stack-overflow or wiki were the good ones.
On the other hand, I've actually found myself missing features from amazon internal tools at the jobs I've had since. One example: I remember a pipelines feature where you could look at any change and quickly tell exactly how far that change had made it in your pipeline. At my current job, determining exactly when change X deployed to region Y has been a multi-step exercise in manually comparing commit hashes.
Right!? The observability on OSS CI/CD pipelines seems to be pretty lacking. Maybe I'll look for an opportunity there for my next position... :)
In fairness, I guess this is the tradeoff you get for being forced into "one and only one way to do it". The benefit is that all the tools are likely to play nice with one another "all the way through the flow", because they only have one representation format to be compatible with. The downside is, well, there's only one way to do it; and if that way happens to suck for you (as, for instance, for an ex-coworker who needed to build multiple different versions of the same code against different base OS images for...reasons), the tools are going to hinder more than they help.
Perhaps it's just a green engineer thing. I know GitHub actions, CircleCI, Drone, etc... I don't know pipelines and sometimes it feels like dark magic to me.
As far as the other tooling goes, I often find that they are very Java-oriented (which is fine considering the AWS background) and everything in Python feels janky.
Ah! OK, now "dark magic" I can totally understand. Pipelines _is_ a little opaque, which is a totally fair criticism. In my experience it does _most_ of the things that you would want if you know a few fairly common (and easily searchable) incantations ("How do I add a new stage to this pipeline?", "How do I put tests after this stage?", etc.) - but I can definitely sympathise with the feeling of "this tool is doing a bunch of stuff for me and I don't understand how it's doing it". Thanks!
One point that I've found _really_ strange after transitioning from Pipelines to other CI/CD (I'm using Drone and Argo, but I think this applies for Flux as well) is that infrastructure-definition repos define the _image-version_ that they deploy to a particular stage, rather than just defining the source repo (and automatically deploying the latest image-version which has passed all preceding tests). That seems...odd. Unless you hack together your own automation[0], that means you need two actions to get new App Code out to an environment:
* Push the App Code commit (and have a new image-version built)
* Make a change to your Infra Code, updating the desired image version
Am I missing something?
---
I'm surprised to hear that you consider AWS to be Java-oriented. When I think AWS, I think TypeScript - to the extent that the primary reason I started learning TypeScript was to be able to write CDK more fluently (and, then, to write Lambdas without a whole bunch of Java type boilerplate - though Python also works there).
[0] Which I've done, but the fact that I had to hand-build it suggests I'm doing something that the tool author's discourage
Does anyone know what role or title the the author might have had there? They mentioned DevOps but then also in another paragraph they mentioned front-end development.
Yes I am well-aware of how SDEs work at Amazon. However there is a role at AWS called SDE - System Development Engineer which is very much DevOps which is very different than SRE and the reason I was asking.
This reminds me of a place I used to work. I used to get my real work done between useless meetings. A couple months after I left, they laid off like 20%.
You must have not tried god mode, the internal tool that flips the table on all of that SDE-0 bullshit. If you work at AMZN, search broadcast for "god mode hackathon" and start using it. If you are external, watch the hype videos here: https://photos.app.goo.gl/8enVGZLYFvpcgj2V9
Let's face it. Most programing jobs are a bullshit job. I've been chasing obscure bugs for weeks and ended up fixing a few lines of code some guy wrote years ago and it improve the software so little. It's really difficult to keep the motivation besides the money.
This author sounds insufferable. Tooling sucks? Fix it. It's important to lead with a culture of quality and impact. Wasting 40 hours a month on something broken? It'd literally be faster to stop your feature work, fix that thing, and then go back to your feature work.
I can only assume you've never worked at a company even close to the scale of Amazon. There's no such things as "fixing it" with a lot of these internal tools. They'll often have entire teams dedicated just to developing these tools. More often than not you won't even have access to the source code and even if you did it would probably be incredibly difficult to make any meaningful changes.
You say it would literally be faster to stop feature work and fix the broken thing which is a laughably naive view on how these companies operate. That's just not an option unless you do it all on your own time in which case you will still be incredibly limited on what you can change and even if you make a significant fix there will be no extra reward for doing so. The incentives are all wrong.
I'm sorry but you honestly have no idea of what you are talking about. The tooling in Amazon is developed by hundreds of engineers and it doesn't suck because of small bugs here and there but rather because of systemic issues related to how it's been architected.
It's not something that is fixable at all, let alone by a lone engineer.
That said, I also want to be more tempered than the author, I don't find the tooling to be that bad. Most teams are on full Native AWS now and the few non-native tools actually do make your life easier because of how well they're integrated in the whole Amazon ecosystem.
Some tooling does suck, but it's more like 2% of my time spent fighting bad tools, not 40%.
My team was not native AWS at all. So yeah, huge variance based on what your team uses.
Also the guy you’re replying to has no idea what he’s talking about. If I had tried to do that my manager would have said “that’s not the work we have planned for you, that’s not the work our team does at all” and if I had just done it anyways (not that I think I could have) I’d have gotten fired.
Lol. If the author had really tried to fix the internal tooling by himself in such a top-don huge corp like Amazon, he would've gotten himself laid off much quicker.
You are grossly underestimating the effort it takes to change things at a company this size. Ignoring the scope of the technical work to be done, you cannot just send diffs to another team and expect them to just accept them.
But the problem is, fixing existing broken stuff might not be rewarded as much as working on the "new feature". This means that if I focus on just fixing the broken stuff, I'd be managed out. Even say I put in the extra effort to fix the broken stuff, and deliver my new features, other thousands of engineers won't do that same- meaning I can't make a meaningful dent.
(i dont work for amazon, but I can imagine it's similar everywhere)
it is when the focus is on having 100% coverage and not on test quality. When 100% becomes the metric it tends to get gamed pretty heavily with tests that have such a huge amount of mocks as to make the tests useless, or doing something to make it hit a branch, but ignoring actually verifying it cause that branch is just a log statement.
I work closely with the CR-time coverage/enforcement tool. The tool expects 70% new line coverage. The number was chosen arbitrarily, but individual teams/managers can choose to avoid the rule or set a different threshold.
It's not ridiculous to expect some, *configurable*, amount of test coverage for newly generated code, is it?
No, reasonable levels of testing are part of the job, it's when you get hard and fast rules from leadership like "don't lower coverage, reward increasing coverage" that it starts breaking down, we had that rule and someone gamed it using the methods I described up to 100% coverage.
everybody on my team. Are you seriously implying I was put on some kind of special "extra testing required" probation system where the unit test coverage requirement was upped from 95% to 100%?
It sounds kind of absurd really. I've never once been asked to sign a contract that required me to do 100% code coverage. As others have noted in the thread, it is like some mythical number any way. It sounds to me like this wasn't going to end well since the expectations were wonky from the start.
It is absolutely possible to set the coverage threshold in Jest to 100% for all categories and they are set to that in the codebase I worked on. I would spend hours trying to that last 0.02% covered sometimes.
I still don't understand why that would cause testing to be hard, but since you won't explain it... I guess it leaves me thinking the problem might not be entirely them.
It's quite possible to get line coverage to 100%. It doesn't really mean anything because it might be covering those lines getting executed in one very particular order - and all bets are off if that order is different.
He's a digital nomad who took off to Mexico (and Belize) and seems a bit lost in where he is going (he says it himself). I've been there myself (except I moved to Vietnam), so I understand it.
This is a lot deeper than just writing tests or 40% internal tooling issues. I also suspect that being remote, he feels like his hands are tied in a company that isn't used to working remotely. Or at least, if I was struggling with tooling, I'd work to find a way to make it better.
I also never complain about writing tests. I can't tell you how many times tests have saved my bacon or resulted in writing cleaner code.
Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter. Automated tooling and tests that were ostensibly created to make life easier were the number one pain point and constantly failed in obscure ways that required cutting tickets to teams responsible for the automation.
More often than not these tickets would be ignored and issues would persist for weeks beyond what's reasonable.
I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
That, paired with the awful corporate culture and constant fear of PIP/job loss forced me to finally leave.
Generally a very unpleasant experience all around, sans the compensation and name on the resume.