To me this is just a simple artifact of size & attention.
Another example of this is stuff like Bluesky. There's a lot of reasons to hate Twitter/X, but people going "Wow, Bluesky is so amazing, there's no ads and it's so much less toxic!" aren't complimenting Bluesky, they're just noting that it's smaller, has less attention, and so they don't have ads or the toxic masses YET.
GenAI image generation is an obvious vector for all sorts of problems, from copyrighted material, to real life people, to porn, and so on. OpenAI and Google have to be extraordinarily strict about this due to all the attention on them, and so end up locking down artistic expression dramatically.
Midjourney and Stable Diffision may have equal stature amongst tech people, but in the public sphere they're unknowns. So they can get away with more risk.
>OpenAI and Google have to be extraordinarily strict
Why? Did the inventors of VHS tapes "have to be extraordinarily strict" and bake in safeguards because people might violate copyright laws, make porn, or tape something illegal?
Enforcing laws is the responsibility of the legal system. It sets a concerning precedent when companies like OAI would rather lobotomize their flagship products than risk them generating any Wrongthink.
* RDS is a managed relational database service. You got a database? We'll run it in the cloud for you. Exact same bits and bytes as you're running locally.
* Aurora is Amazon's own relational database. You can't run it yourself, only with Amazon. It can pretend to be either Postgres or MySQL. And it'll be cheaper and faster and have higher availability. But it won't be the exact same bits and bytes as your own Postgres, so there's some risks.
So far the stuff we said runs on clusters. You pick how big and how powerful and how many and in which AZ and configure how to scale it.
This has fairly pragmatic limits before at a certain point sharding and continuous horizontal scaling just gets too hard.
* Until Aurora Limitless that is which is, well probably not limitless (I don't know) but effectively.
* But you're still configuring cluster sizes and scaling policies. If you don't want to do that, Aurora has a Serverless option. It's the same Aurora but now you don't have to worry about scaling it yourself. The first version was Aurora Serverless but people said it wasn't very good.
* So they put out a 2nd one which is great and scales to zero.
* Now if you have globally distributed customers or care a lot about resiliency, you probably want your database in multiple regions. And you're going to be setting up eventual consistent updates for that. To make that easier there is Aurora Global Database, which is the same Aurora, but now with cross region replication.
* But it's not strongly consistent across regions. Aurora DSQL is. It's an even more bespoke version of Postgres. Actually it's not a relational database one at all, it just pretends to be one. But it uses atomic clocks and shit to cheat the cap theorem and give you global distribution and resiliency with strong consistency.
So in conclusion there's basically only 3 things here:
* RDS which is an unopinionated way of running a relational database in the Cloud
* Aurora, Amazon's highly opinionated relational database. That has Global, Serverless, and Limitless configuration options.
* And Aurora DSQL which is not Aurora a relational database at all but plays one on TV. But it gets to have the best of all worlds - SQL and NoSQL.
They probably should have called it something new but Aurora has good brand recognition. People know and trust Aurora.
In the long term all will probably continue to exist depending on where you are in your cloud journey. But I also expect that in the fullness of time if you're building a new Cloud native application and you don't have to worry about legacy migrations, you'll probably choose either DynamoDB or Aurora DSQL in 99.9% of cases.
>Aurora is Amazon's own relational database. You can't run it yourself, only with Amazon. It can pretend to be either Postgres or MySQL. And it'll be cheaper and faster and have higher availability. But it won't be the exact same bits and bytes as your own Postgres, so there's some risks.
That doesn't ring true to me. From my understanding rds aurora mostly replaces the storage engines of these DBs and the frontent mostly follows upstream releases and is compatible with them.
You're spot on. GP is repeating the common misconception that "Aurora" is a distinct, from-the-ground-up relational database system which can "pretend" to be either MySQL or Postgres by reimplementing their protocols/front-end from scratch. But that isn't the case at all.
In reality, Aurora MySQL is clearly heavily based on the MySQL codebase, but with major changes specifically to InnoDB's persistence and MVCC layers. Aurora MySQL implements a physical replication system using shared storage, undo, and redo logging. The rest is otherwise compatible with MySQL because it is using a lot of MySQL code.
I haven't read up on Aurora Postgres, but I'd bet it's something similar being applied to the Postgres codebase.
It's entirely possible the custom storage components may share some common code between Aurora MySQL and Aurora Postgres, but a custom storage layer alone isn't a "relational database" which "pretends" to be other DBs. (In contrast, it appears that DSQL actually might be that – a custom system with a reimplementation of Postgres wire protocol.)
Not trying to knock Aurora btw, it's really interesting technology and is quite an achievement by Amazon. But the constant confusion about what Aurora even is creates some support burdens in the DB ecosystem.
Very nice write up! It makes even more sense if you compare with Azure and see that they have the same kind of offering with the exact same breakdown, i.e. Azure Database for PostgreSQL/MySQL which is the managed relational database solution, Azure SQL databases which is the in-house one (i.e. SQL Server modified for Cloud) and promise cheaper/faster/higher HA because that's their own opinionated technology, with serverless and hyperscale (i.e. limitless) configuration options. They don't have DSQL equivalent yet, but their NoSQL offer (Cosmos DB) already have SQL query language support and with some modifications & improvements could be their "best of all worlds" equivalent.
> f you're building a new Cloud native application and you don't have to worry about legacy migrations, you'll probably choose either DynamoDB or Aurora DSQL in 99.9% of cases.
If cloud-native means spending your life in AWS, that is.
But I use "cloud native" in a way that actually makes sense, not the way that GCP dubbed Kubernetes and all of it's ecosystem of friends "cloud-native".
"Native" development on a desktop OS means using primitives specific to that OS. As opposed to using something like Java Applets or Electron which will give you cross-platform compatibility, but you're not developing "natively" for the platform. That has the obvious pros and cons to each way.
Kubernetes is very much the Electron way: Rely on a generic high-level abstraction that will then create awkward bindings of running on abstract generic servers - whether they're in your data center or in the cloud.
Calling that "cloud-native" is some Orwellian post-truthness. I hate it.
If you want to develop in the cloud and make the most of it, use Azure, use GCP, use AWS, I don't care (not true: AWS is the best; use AWS). Just use the cloud provider's NATIVE primitives.
Otherwise, you might as well go back to running your own data centers, what are you even doing.
(Note: i'm just some guy. You don't have to listen to me. It's just what I think)
> I super doubt these limitations are there because of any EC2 requirements.
I think you are misreading me, but to be fair, I was being vague about timelines.
my main thrust is that S3 is dominant because EC2 was/is lacking. S3 is optimised for uptime and consistency, which means that its brilliant at mostly static file hosting. It will work for more dynamic state type stuff, its just not really designed for it (see https://xeiaso.net/blog/anything-message-queue/)
making your own object store that is fast, durable and available and has another feature is really really hard to do at scale. Its far easier to put up with s3 than make your own.
Andy Jassy was the first product manager on AWS, and while there is lore debates whether he "came up" with the idea of AWS (probably not), he's been on it from the beginning and through every single product.
I don't know much about Selipsky or Garman, but Jassy absolutely was/is technical enough to get to AWS to where it was today (or know when not to get in the way of those more technical than him).
Watch any of his interviews from before he was Amazon CEO. The man knew his shit.
I don't dispute he is an excellent manager and an excellent CEO. But excellence is execution does not drive vision. The times demand a different profile, as AI is not there yet, but can within the next 24 months become really disruptive for many business models.
One example: Amazon generated $14.65 billion from its advertising services. And Google a gazillion more. But if I send an AI agent searching for me or shop on Amazon, they don't care about the ads.
As someone who has been involved in high level crisis management issues like this multiple times across various companies I can tell you that in a competent organization it looks nothing like your day-to-day decision making as an engineer or PM. Better yet, as few "rank and file" employees are involved as possible to avoid dangerous situations like you just described.
I don't want to debate the merits of what happened, but a prosecutor is going to open with "AWS billed people for things they never asked for or consented to." You're already fighting an uphill battle that it is not fraud.
Now what is going to save you is intent. If your defense is "yeah we identified the problem and corrected it" you're good to go. If on the other hand, someone decides to run a fucking metrics report of how much you could lose by stopping doing fraud and god forbid it is ever seen or mentioned in front of anyone in the decision making path - you now have to deal with mens rea.
If you have material knowledge that someone took "a look at the metrics", shoot me an email. I can help put you in touch with programs that offer financial rewards for whistleblowers.
If you were L8 then the responsibility for setting the culture is 100% on you.
L10s don't micromanage, and L7s take their cues from L8s.
If you want to have fewer meetings, you can set that culture.
If you want less fungible engineers, reinforce specialization in your OLR process.
If you don't like a process, kill it.
This is worse than than the "you're not stuck in traffic, you are the traffic." This is "you're not stuck in traffic, you are the accident creating the bottleneck".
In pretty much any big enough company with enough people, the issue is that you have a general sense of direction at higher levels, and if you go against the grain, it makes other people criticize you because they want to be seen as going with the grain for their career paths. If you want to see the biggest example of effect in action, go work for the US government.
So at certain point, you just stop caring to do anything because the paychecks are worth more to you.
Nah, I hired specialists and caught no end of trouble for doing so. All of them that are still in tech make $1M+ annually these days because they stand out among the lovable but mostly ineffective fungibles that are the cannon fodder for rounds of layoffs there.
But advocating for my direct reports instead of happily figuring out which one to fire next is what made the experience so dreadful and career-limiting when the L8 and L10 herd's jobs depend on continually hiring and firing to look busy.
I'm not lecturing them for leaving. I'm lecturing them for complaining about things that it WAS IN THEIR CONTROL TO ADDRESS. More than that - it was their actual JOB to ensure.
And what makes you think one L8 against an entire hivemind of L8s and L10s whose high-end compensation depends on negating everything you're doing can move the needle? 80+% of my meetings, so very many meetings, were endless discussions of who the next "unregretted attrition" would be so as to appease the need to fire. Those are your leaders, and that's how they spend their time. Or maybe you're one of them and the truth hurts?
I chose to "disagree and commit" to "The Law of Two Feet" and left. One L8 claimed he would be blacklist me and my direct reports (who also left) from ever returning. He got a harassment claim later and I asked in return for my input that HR investigate that claim. Nope, not blacklisted, he was just a lying PoS.
I know a lot of people here are writing about how this can be done for small consulting companies, but I also saw it in Big Tech.
Amazon until 2022 really genuinely exemplified this. I saw it for more than a decade leading up to this. Just an unbelievable collection of people that truly Gave A Shit. Publicly we called it "Customer Obsession" and through that lens you could move mountains around here in the pursuit of Doing The Right Thing.
The first sign of trouble was 2021. Salaries skyrocketed in the industry. Amazon didn't keep up. A lot of great people left because they got obscene offers, and you know, who could blame them? Our core of "intermediate" engineers (L5 here) got decimated - why bust your ass for a promotion when you can just get a Senior offer from one of 100 over-funded Unicorns for more money than you would've made here. Sensible.
Then in 2022 the stock price dropped in half and a bunch of folks who seems like were only putting up with the bullshit as long as the stock grew indefinitely left too.
Then 2023 brought layoffs.
There's still a lot of us around that Give A Shit, but I feel like we are outnumbered more and more by those that just want to punch in and out and no longer Make History. I get it. I can't blame anyone individually. But I miss it.
Trust me. We as the customers can tell. It's really sad.
Everything that gets released today seems cobbled together and half baked. The "this is something we bought and slapped an API and product umbrella on top of" is glaringly obvious.
Back in 2018 we used to get strong "give a shit" vibes from all the product teams we talked to. Now it's more "this is what we're going to build, isn't it going to be great?".
I skip all the product meetings we get invited to now.
> Amazon until 2022 really genuinely exemplified this.
I remember Amazon from 2002 — that felt like they really gave a shit. There are occasionally times where it feels that way now, as a customer. But most of the time the customer service experience is slow, impersonal, and painful.
I understand they're a big company, and I don't expect it to be as good as when they started out. But I wouldn't say that Amazon "genuinely exemplified" giving a shit up until last year. Apple might fit that bill, but even they have fallen off recently (apparently you now have to make appointments to buy products in-store?!).
The frupidity used to be a huge sore spot, but being honest, is no longer a part of the culture.
Sure, there are no catered buffets or massage chairs, but the old inflexible rigid frupid systems are all gone. Engineers get top of the line M1 Macbooks, nice monitors and good chairs. The desks are all adjustable sit/stand, not doors. You can get and expense whatever software you need. And yeah you gotta fly coach but nobody is going to bite your head off for expensing some peanuts from the mini bar at your hotel.
Mid 2022, my (brand new) manager received a rigorous scolding for holding a team event where the team (spread out across East Coast and West Coast) met in a single location (Seattle; where majority of the team was) for a couple days of team building (i.e., working in the same office, going out to lunch).
Frupidity may not be the correct word. Perhaps miserly is more appropriate.
Top of the line? When I started a year ago I was told to specifically not opt for the Windows laptops because they ship garbage HP units with 8 gigs of RAM as standard
Only to be then shipped a base M1 Macbook with 8 gigs of RAM anyway!
Sorry dude, you don't have credibility based on "Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management,"
Management at AWS arguably puts up with much more shit on a regular basis.
Another example of this is stuff like Bluesky. There's a lot of reasons to hate Twitter/X, but people going "Wow, Bluesky is so amazing, there's no ads and it's so much less toxic!" aren't complimenting Bluesky, they're just noting that it's smaller, has less attention, and so they don't have ads or the toxic masses YET.
GenAI image generation is an obvious vector for all sorts of problems, from copyrighted material, to real life people, to porn, and so on. OpenAI and Google have to be extraordinarily strict about this due to all the attention on them, and so end up locking down artistic expression dramatically.
Midjourney and Stable Diffision may have equal stature amongst tech people, but in the public sphere they're unknowns. So they can get away with more risk.