Everyone: PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.
As the article points out: Every other cloud does this! They all have non-production subscription types with hard spending limits.
There's a difference between "unable" and "unwilling".
The people that refuse to understand the difference are the same people that don't understand that "unsupported" isn't synonymous with "cannot be made to function".
Don't be that person.
If you have a large account and you're in regular contact with an AWS sales representative, pressure them into making this happen. Even if you work for Megacorp with a $$$ budget, keep in mind that your future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.
Don't walk down the same path as IBM's mainframes, where no student anywhere ever has been able to learn on their own, making it a dead-end for corporations who pay billions for this platform. You, or your company will eventually pay this price if AWS keeps this IBM-like behaviour up.
Think of the big picture, not just your own immediate personal situation.
Apply this pressure yourself, because the students you want to hire next year can't.
I'm also going to point out, as a former AWS engineer, that
"too hard" isn't in the AWS engineering vocabulary. If an AWS engineer were to dismiss something as "too hard", they'd be on the fast track to a performance-improvement-plan.
The problem isn't that it's too hard. It's that it isn't a business priority. As soon as it becomes prioritised, a legion of very smart AWS engineers will solve it.
As an antidote to the "nothing is too hard" flex...
Everything has trade-off's. When a responsible person says something is "too hard" it's because they've evaluated the consequences and have deemed them too costly in terms of time, resources, cost, maintenance, or strategy/lost-opportunity. Some things really are "too hard".
I agree it's not too hard technically but am surprised at your contradictory suggestion AWS would deploy "a legion of very smart engineers" to solve it if they decided to implement that ordinary feature. How smart are they?
I suspect the truth here is a bit more ugly. AWS doesn't want to do it because they see the pocket-change taken from accidental mishaps as a net positive. They don't care if it inflicts suffering on students and novices. Sure, they'll fix individual cases if they're shamed publicly or if the individual contacts them in exactly the right way, but at the end of the day, it's more money in their pocket. For every 200 dollar oopsie that gets charged back, how many go silently paid in humiliation and hardship?
I would speculate that Amazon doesn’t care about the occasional $200 mistake, and is perfectly happy to refund it should they notice.
What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.
I would guess most companies spending millions per month on AWS are just guessing what next month’s bill will be, and have been conditioned to not freak out if it’s a million over, as long as that doesn’t happen every month. That conditioning has to start early.
> What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.
Why not? It would make companies happy.
The reality is that, as much as many would like this feature, I'm not sure that a lot of large customers are actually asking for it. Does anyone who works at a large company know if they've asked for this feature before?
I can imagine why startups might want it, but also, AWS is extremely forgiving with startups and throws tons of money at them, so idk.
The problem is, what to do when your account hits the limit? The subtle point about EC2 is you can’t actually shut down servers and have them come back up later unless you have a specific configuration - which not everybody has, so you can’t simply power off systems and not destroy something. Not insurmountable, but not trivial either. You also can’t say that there’s a limit except for an incompatible subset of products, nor can you ask users to be okay with randomly destroying things - the mere rumor of that destroys user trust in something. (Who’s still shy about using EBS despite the last major incident being back in… 2016? Also that outage was limited to one region.)
> nor can you ask users to be okay with randomly destroying things
what do you think will happen to your EC2 instances if your payment method on file with Amazon fails repeatedly and for more than 30 days? Do you think they will just keep those instances intact indefinitely without payment because they promised not to 'randomly destroy things'?
You are correct, which makes it even more pernicious that Amazon chooses to not fix this ... profits at any cost is not a long term benefit ... the longer AWS fails to fix this more folks will tend to lean toward the competition especially Azure ... I have been a unix/linux server side developer since the beginning and always use a linux laptop so no fan of Microsoft however their Azure platform is lightyears ahead of AWS ... its obvious the AWS console web front end needs a wholescale rewrite from the bottom up ... again as you say its not a priority probably because all big consumers never use the AWS console as they have written their own automation layer atop the AWS SDK
I routinely tell a coworker that <random task> isn't possible so that he does the work for me. Oh I know it's possible, I just don't want to do it. Thanks Corey! You're the best.
This works in help forums too. Post a question, there's a chance you get ignored. Post something wrong, and someone will flame you and answer your question.
I realize it can be fun to be snarky but consider the “business priority” portion you didn’t quote and the difference between “it’s too hard to do” and “doing it would cost X and impact performance of service Y by Z%”.
> Every other cloud does this! They all have non-production subscription types with hard spending limits.
They don't. I've looked. They all have some ultra complex scheme for monitoring billing and programmatically shutting down services, which is a favorite recommendation by apologists, but none have a nice, solid "shut down everything" plan that's usable for learning and testing.
AWS does have billing actions which are fairly new, but they're still a bit too difficult to deal with if you also follow the advice for setting up a root account for billing plus sub-accounts / organizations for use.
Azure has been ignoring the request for over 8 years [1]. There's a warning they're moving away from User Voice to a product-by-product solution for feedback, so that request might disappear altogether. I reached out to them and politely asked for an update and they said they'd look into it, but it's been over a month and I don't think there's going to be an update without some pressure. If anyone has a good social media following and would like to see hard caps on spending in Azure, tweet @AzureSupport. Maybe if enough people do they'll actually follow up.
> PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.
I agree 100% with this because all I really want is something along the lines of the "personal learning" account that's described in the article. I want a dev / testing account where I can learn and test assumptions about billing without risking a massive overage that I wasn't expecting. They don't have to get into the complexities of dealing with production accounts.
AWS, Azure, and GCP have all had close to a decade to figure it out, so I think the only way it'll happen is if we start asking legislators to regulate those providers. It would be relatively simple in my opinion. Cloud infrastructure providers must allow users to indicate a maximum amount of spending per month and may not charge more than the indicated amount.
I think big tech is proving they can't self regulate, so it's time to quit letting them.
Annoyingly, Azure does have this capability! If your company is a Microsoft gold partner (there might be other requirements, I can't remember) then your Azure developer licenses comes with £100 free credit per month. When this money runs out, every service associated with that subscription just shuts down. It doesn't overrun, heck I don't remember even getting an email about it, they just stop until it renews the next month.
They just refuse to implement this system for non dev licenses.
That's exactly the same way that Azure for Students works, which is why I use it instead of AWS Educate (well, that and because Azure doesn't limit what services you can use your $100 on, unlike AWS).
btw. azure has a subscription type where you can put a prepaid amount onto your account. they just do not support it by default. btw. bizspark customers get this of subscription, for 3 years with an free amount of up to 300 USD per month
"Every other cloud does this! They all have non-production subscription types with hard spending limits."
This is simply not true, to the best of my knowledge. GCP has budget limits, which can send alerts, but outside of manually catching a pubsub and shutting down billing on your own, there are no configurable hard billing limits.
I'm not sure if that's what you meant by "manually catching a pubsub" -- you have to manually set up the script once, but it will run automatically when triggered.
Yes, well aware. I'm not sure if you actually looked at the code on that page but you have to catch the pub sub in the deployed cloud function in order to shut down billing on the project/account. This is precisely what I was referring to.
What we need, a function that is AWS maintained and supported, that basically is AWS responsibility - that if it did not work - failed to shut down service that was consuming all the budget, they eat the cost.
The budget limits are delayed hard billing limits. Everything I've seen/heard suggests while you can go over, they absolutely don't charge the overage.
We go over our budget "limits" literally every month. We do this because they are simply alerts, nothing more. We have separate alerts set up for 25, 50, 75, and 100% of our "limit" so that we can track variable usage and catch spikes early.
We, of course, pay our full bill every month - over the "limit".
^this. For business-critical operations you almost _never_ want to have your service shut down, especially as a surprising result of a big spike in usage.
>future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.
I've shared this before on hn, but I was about to bite the bullet on a fire base subscription to back a web app I'm building. Google announced its cancellation the week I was going to subscribe, I was hitting daily limits in my testing with my garage code and wanted the extra buffer to speed up my progress. As soon as they announced the cancellation I ripped out fire base and have been working on a swap to hasura/postgres/auth0. I can't afford, likely multiple, hundred or thousand dollar mistakes for a hobby.
Which clouds do this? I work quite a bit with AWS and GCP, and Azure to a lesser extent, and I've yet to see a spending limit on any of them. I'd love to be proven wrong though.
Azure definitely has subscription types that have payment limits. I have one subscription that has some credits on it, and if I go over the limits all services are suspended (i.e. compute, storage, etc) and I can't do anything unless I add more money or wait until the next period starts and the new credits kick in.
Not all Azure subscriptions types are like this though.
GCP has a spending limit, but it is by no means transparent[0]
Basically you add a budget to your project, create a pub/sub channel, then make a lambda that subscribes to the channel and turns off billing when it receives an overbudget message to shut everything down.
Or you could integrate it into your project directly to handle this more gracefully.
You can even send a test message, to check that everything does indeed shut down, but the whole process is extremely clunky and error-prone.
Yeah, but that's $45 USD per month which is an expensive way to learn and test things that would normally cost <$5. It does get you $50 per month of Azure credits though, so it's probably the route I'd go if it's possible to buy a month of Visual Studio and ignore everything but the Azure credits.
I had (have?) an Azure account through my university account and while I agree that Azure services such as Container Service are (in my not so humble opinion) ridiculously overpriced but the fact it I never entered payment details to use my USD 100 of free credit on Azure. I effectively have spending cap of zero dollars.
I thought I was the only one who thought pricing on Azure seemed rather high for what you get.
But I also have free credit I can use each month, which offsets the cost to "normal" levels, but can't imagine being a business and opting to pay for Azure services by choice, unless they provide something no one else does.
Had Bizspark from microsoft which gave us access to Azure and about $300 in credits I think per service or the whole thing iirc. They shut down services once we went over that limit. This was back in 2016/2017
GCP definitely has a spending limit. Of course it also (at least in the past) used to take 24 hours to update, so if you had a massive spike in real usage, your site would die for a day while you frantically tried to increase the limit.
Why are people saying this? It's not true. App Engine had a spending limit, but it's deprecated and going away in July. The rest of GCP has never had a spending limit.
The best you can do is build your own service to monitor your bill and manually cut off your billing account if it gets too high. Hope you did it correctly because there's no good way to test it! And the bill notifications can be delayed an arbitrary amount of time with no guarantees, and disabling billing can take an arbitrary amount of time too, so you can still be screwed even if you did it correctly.
I think it's a glaring hole in cloud platforms. I get it, it's hard to implement in a way that is forgiving and doesn't cause a bunch of support tickets from people who set their limits too low and made their site go down. But that's why people use cloud platforms, so they don't have to implement the hard part. It's beyond me why the cloud platforms don't do something better here.
Why should they if they can use obscurity and people's limited comprehension of their complicated systems to produce legitimate, legal bills to customers?
You're talking a 'get paid/don't get paid' decision here.
When you look at this on a high enough distributed level, across enough users, this is the profit margin. Enough people can screw up and then be able to pay the cloud provider, that it is part of the business model. You're required to run some chance of producing a legitimate megabill.
Why be forgiving when your business model requires that you set traps for fools?
> As the article points out: Every other cloud does this!
No, they don’t, the article is misleading if not technically wrong (since they use “and/or” to link billing caps and better whole-project deletion, the latter of which mitigates a subset of the problem but doesn’t address accidental unbounded spend from a project that you want active at some level) and the same general class of complaint is raised for the other clouds on this issue; the frequency of the complaints seems to vary (ordinally, but not linearly) with usage of the various clouds.
Hmm not sure where the problem is. With IAM you can create a policy to limit the user to only use certain services, deploy only the cheapest type of instance type, limit the region, the time of day etc. All of this and more is built in to IAM by default. You can even create a CloudFormation as a internal product this way you can limit even more what people do, and budget for that. And on top of that you can make a Lambda that is triggered every 1h to shut down unused resources or send a warning to shut it down, if not, it will go down the next day.
The possibilities are endless. I personally don't see the IBM comparison.
The same as it is not expected by anyone to know how to ride a bike or drive a car. Someone has to teach you or you have to go and take a course.
Same with any technology, you should read first all the documentation that there is, watch all the videos that AWS releases every year for free where they explain in detail every service that they have, and try to explain the best practices of it - this is all for free, or you could spend $1000 for a course, instead of risking to lose $10.000.
If you jump in to the water without knowing how to swim, and then get angry at the water because you drown - well...
After 18 years of age you become a grown up, because your parents are not responsible for you anymore. You become responsible for yourself. That is what distinguish a child from a grown up.
Of course a parent needs also to tech you responsibility and what it means.
Mine for example never did, and I had to learn life the hard way. At first I was blaming others, but then I realized that to really grow up, I need to stop blaming others and start owning my mistakes.
So, it is not expected for anyone to know everything, but it is expected that if you want to learn something new, you need to first research the topic.
Are you seriously suggesting that someone who wants to use AWS should go and read all the documentation and watch all the official videos available? That would take literally years. It's huge.
It also doesn't solve the problem. If the user makes a mistake they still lose a ton of money.
It's a really interesting problem. It's getting too big and too scary for a new developer to jump in and try AWS services, which means new devs will move on to newer services with better UX. I wonder if this sort of problem is one of the few existential threats AWS could actually face. Its possible it could end up as the Oracle or Salesforce of cloud services, very successful in enterprise but definitely not a good thing to have on your resume if you want to join a startup...
I would not want to be the competing cloud service, with self-imposed limits of only what people are willing and expecting to pay, going up against Amazon (or any other cloud service of that scale) where they not only have scale already, but are being paid the extra money regularly by people screwing up in small to medium ways (even IF the most massive screwups are 'forgiven' to devs who wouldn't have been able to pay anyway, but have twitter)
I don't believe there will be any new services with better billing UX through competitive pressure. You're basically asking the competing thing to take less money in order to win, as the distinguishing factor. It doesn't make sense as a way for them to win against the larger thing taking more money.
Probably the only practical solution would be arbitrary legislation forbidding the practice, as if it was anti-usury laws. Basically taking the guise of society and saying 'you may not do this, even if people are dumb'. And then you've got a problem (at scale) of people figuring out how to do hit-n-run megabilling and walking away under the legislation, having intentionally taken what they 'mistakenly' did. An 'oops, I accidentally a bitcoin' defense.
1. yes I am, studying years to be a doctor, lawyer, Engineer etc from your point of view is also to much? Should you just start cutting people to figure out how a body works end expect for everything to be ok once you are done? The point being, yes you have to study to learn something, there is no way around it.
2. If money is a concern to you, then I should focus on learning how exactly the billing works and how to monitor correctly. This way you can build a product the right way, not to mention AWS by default has limits on their services set with limits that prevent you from doing something incorrectly. For example you can only make 5000 requests a sec on the API Gateway, you can only have 1000 concurrent lambdas, you can spin only 25 ec2 instances, ecc... (true, not all services have limits like this - but then again, if you want to use one, the first thing you should do is check the pricing page, this is what I do fro every new service that I'm planing to use).
3. AWS is not for developers, it is meant for SysAdmins and DevOps (true that some marketing materials are not clear on this), they should be the one configuring it to allow developers to host their code. If you want a turn key solution, then there are better solutions, like Heroku - incredibly easy to use and understand and have a much simpler billing structure.
With AWS you can do anything you want, AWS provides lego blocks, what do you build with it is up to your imaginations, and for sure it is not meant to be use directly by developers who have no idea how networks, computers, databases, cpu, ram, policies, storage etc works. Developers should focus on coding, and SysAdmins and DevOps should focus on managing the infrastructure.
And if you want to learn AWS because you want to be a SysAdmin, then it is true, that AWS could have a plan for beginners with even smaller default limits, and limits set on everything - this way you could more safely play with what they have. This would be a nice things to have in this case for sure.
But because they don't provide such thing, you need to be the responsible one, and start learning AWS the right way, and not get in gun blazing, and expect all will be ok. My recommendation is to learn one service at the time. If you do this, over the years the acquired knowledge will be gold. Plus the more services you learn the right way the easier it gets.
There’s room for a little bit of the personal responsibility argument, but it’s ineffective when you take it way too far.
> studying years to be a doctor [...] Should you just start cutting people
Spoken like someone who’s never used AWS or been a doctor. Your analogy is horribly, badly flawed. Doctors do start cutting people, in the US nearly all med students dissect a cadaver at the start of their first year. What you’re suggesting is doing years of pure documentation reading, unguided by other people or a curriculum, before practicing AWS, which would be silly and a waste of time. People learn by practicing, which is why med students dissect cadavers, and which is why AWS offers a “free” platform to learn by practicing, tutorials to guide the learner, and advertising to attract learners.
People in this discussion are complaining that the free tire is misleading, plus not all services are covered. And it is true that if you turn on a bunch of server and you don't pay for a year, but forget about them, you will be charged the moment the year passes. Not to mention that you will be charged if you use the CPU of the free tire server to much - which probably very few people know about.
And my point is that the marketing of AWS is misleading, they try to convince you that if you don't know anything about computers, but you know code, you will be able to manage AWS. This is very misleading because AWS tries to make you think that AWS is a service like Heroku, simple to use, and there is just one button to push to make it all work. Completely false. I've seen countless AWS accounts that were completely misconfigured by developers who thought that AWS easy to manage. A basic example is the autoscaling of EC2. People will go to the autoscaling section of EC2 "enable it" and be superseded when it dose not work. Where the reality is that you have to do 8 other things to make it work, not to mention the work that needs to be done in the OS itself.
> Select a learning path for step-by-step tutorials to get you up and running in less than an hour.
Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
It is very dangerous. If you select the full-stack tutorial you get: "Time to Complete 30 minutes". It should say: "30 min to ruin your life" ;)
If you want to really learn AWS, then this page should be used as a reference of how to design a stack. If I were you I would read the tutorials to see which services are needed for a solution, but before doing anything, I would read the docs for each of those services to really understand them, then I would go back to the tutorial and actually do it, and - MOST IMPORTANTLY - I would read the pricing page for each service that you are going to use.
> Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
100% - when I started working with AWS in 2016 I had a very hard time figuring it out, because I was looking for the simplicity the the marketing team was writing about. I really don't like what the marketing team tries to tell you, because it dose not exist.
Regarding an approach to learn about AWS, I would start with all the serverless services that they have, since the pricing for most of them is ideal for beginners (WARNING - read the pricing page for each since not all have a free staring plane, like S3 and DynamoDB) and for simple weekend projects.
```
All resources deployed via this stack will potentially cost you money. But you'd have to do the following for this to happen:
- Invoke Lambdas over 1,000,000 times a month
- Send and receive over 1000 emails a month
- Perform over 10,000 Get and Put operations and over 2000 Delete operations in your S3 Bucket
- Exceed 100 build minutes on CodeBuild
- $1 per active CodePipeline (must run at least once a month to be considered active)
The only payment you'll encounter from Day One is an S3 storage fee for emails and CodePipeline artifacts.
```
So you can have a stack that is actually doing something very useful that costs not even a $1 a month.
It is possible to pay $0 to AWS, but you need to first understand AWS to be able to do it, another trivial example of a tiny project that is useful and cost $0 to run: https://github.com/0x4447/0x4447_product_secure301
The last point would be: don't listen to the marketing material - they are there to sell you AWS, marketing never cares about reality.
I also recommend this website https://awsvideocatalog.com - pick a service and watch all the keynotes AWS has on that service, if you'd spend 1h a day, in 6 months you'll know more about AWS then anyone else complaining here.
Agree with all of this. I think the reason you're getting so much pushback is that you started this conversation with "not sure where the problem is". Clearly you do see the problem - AWS encourages beginners to get going as quickly as possible with these tutorials and the free tier, but then makes it difficult for those users to avoid unexpected charges. They should either stop encouraging beginners, or start offering easy ways to protect yourself.
That statement was related to the sentiment that there is no way to protect yourself or a team of people from AWS pricing, people were implying that there is no tool to help you limit or track expenses, which is not true, since there are plenty of tools to do that.
Anyway, life gose on, and it was overall a good chat :)
> I need to stop blaming others and start owning my mistakes.
There's a big difference between owning up to your mistakes and taking on a bunch of unnecessary risk in situations where you know it's more likely you're going to make mistakes. IE: Learning, testing, etc..
In fact taking on unnecessary, possibly unlimited financial risk in a situation like that is a mistake. Want to own that one?
You can by insure for you as a driver and for your car (and in many jurisdictions have to) — this limits your responsibility. You can spend $10000 on AWS and still make a mistake and pay another $10k+.
I don’t get the point that AWS docs are free. I mean, they are free, but the services are not. Do you pay for a visit in a car dealership?
> You can by insure for you as a driver and for your car (and in many jurisdictions have to) — this limits your responsibility
Real insurance policies have maximum coverage limits, so, while it reduces your liability by a capped amount, it doesn’t, strictly speaking, limit it; you still face unbounded potential liability beyond your insurance coverage.
what's the biggest unexpected billing of AWS fees reported? I'm thinking 100,000 seems unlikely to be reached very often.
on edit: this is not to say that I believe an insurance for this kind of thing would really work (although maybe that is because I know nothing about the insurance business) just that I don't think the problem with the insurance would be that you could always end up getting billed more than you were insured for.
It seems obvious that AWS simply doesn't want you as a customer if you need to implement spending limits. We've all talked about having customers that were too much trouble for the value they bring. Customers with hard spending limits are those customers for Amazon.
So items 1 and 4 are courses which you need to sign up for and get everything including instruction. That fails to qualify as "on their own".
Item 2 is documentation. This is good! But without access to the system to try it out, how self-teachable is it?
Item 3 is the one counter point. With a third party emulator though, how comprehensive/accurate is it? Would you hire someone to work on a mainframe who'd never used a real system before?
Compare that to e.g. Java. Sure there are plenty of college courses, some even Oracle affiliated, but I can also just download a Java compiler and start plugging away. .NET is there with Visual Studio Express. Oracle cloud, for everything else wrong with Oracle, has a free tier with no time limit. Microcontrollers for embedded development can be had for €20. You can install Linux or a Linux VM on your own computer. Even stuff that falls into the more enterprisey camp like MS SQL or Oracle DB has free personal licenses.
>So items 1 and 4 are courses which you need to sign up for and get everything including instruction. That fails to qualify as "on their own".
Attending a course on your Own.
>Item 2
No it's a MVS distribution (a full blown Operating System) with all the compilers 99% the same as a "modern" z/os
>Item 3
That's the S/390 emulator.
>Would you hire someone to work on a mainframe who'd never used a real system before?
Most Mainframe-Programmer where never near the Hardware anyway...so YES i would do that absolutely (you don't do Hardware near programming in Mainframes, an emulator is perfectly fine), same as a Dev who programs in a VirtualBox, no difference.
If you want to learn how to use it, and MUCH more:
Forgive me if I am out of line, but as I understand it, your company would still be legally liable for the debt. So even if blew past your prepaid card limit, your company would still have an obligation to pay using an alternative method.
Not a lawyer, etc, and don't even know what jurisdiction you are in or what providers you use, so this is just a general caution that your strategy may not generalize to other companies.
Nah, I think not. My business needs to run. Would be very painful if an engineer put in a hard shutdown at $X and then left the company only for me to find out after all my services are shutdown when we've grown past $X.
This is a hard no from me.
How is AWS going to even know what to shutdown/remove? What if it's storage causing my bill to overextend?
I’m not sure I’d want a running business haemorrhaging $10,000/minute or whatever is possible for cloud compute.
My business is looking at cloud compute at the moment and we have absolutely no idea how to do it safely. In fact sagemaker is exactly a product we have looked at and discounted since we cannot be sure we can do it safely without getting unexpected megacharges.
We had an incident recently where a cellular router was set up to send a text message on a GPIO changing. The old firmware didn’t have pull-ups on the input. We told our end customer to update the firmware. They didn’t and also didn’t hook up the input and left it floating. We got a £20,000 bill for text messages as a result and a soured relationship with the customer.
AWS employs cost obfuscation by design otherwise the default view when you open the console would show you all of your current active services. Not only is that not the case, a single screen to show you all of your current active services doesn't exist. You need to take a deep dive into cost explorer (assuming you have access in corporate land) and try to decipher in what that all means.
They definitely need a senior executive to stand up and say, "The Customer wants us to be transparent in billing, fix that now."
Then they need to start a team dedicated to finding a good way to let customers halt spending at a given limit with minimal impact on their operations.
They already win on UX (okay, okay, it's an opinion ffs), but unlimited liability makes a lot of people very uncomfortable. Those two actions would go a long ways towards demonstrating good faith in that area.
If it would cost too much, maybe they could present it as an easy way to cut expenses at the same time that they introduce a small price increase. This is a common and long-standing complaint/feature request.
As someone that has tried and failed to get some small personal sites running on AWS a couple times, I'm going to have to tag this snippet with [citation needed].
From my admittedly limited experience with GCP and Azure (and this is obviously subjective), the UX in the most successful competition is, at best, no better than AWS. It isn't that AWS has good UX, just that all the cloud providers have bad UX.
Oh man, I have long been joking about the AWS UI. Like how it will gladly walk you through all the steps for launching a server and only at the very last step says “oh you don’t have permission to do that lol, get permission and start over”.
I compared[1] it to a bartender who walks you through an entire sale and only at the last second rejects your purchase for being underage (instead of denying you the at the initial request for something alcoholic).
Of course, that’s one of the minor things in the grand scheme.
agreed especially when compared to say Heroku or Digital ocean. I see many new comers struggling to deploy a small website. It's overwhelming. I understand that Heroku exists for such users and they are using AWS under the hood, but is there any service which takes AWS cloud APIs and simplifies it with a leaner UX?
> agreed especially when compared to say Heroku or Digital ocean.
Funnily enough, I've never ever been able to understand what Heroku is, or how to deploy anything useful on it :) But I'm old, and I've always found it easier to tinker with nginx configs.
Heroku is very simple - you just create an app and "push" code to a repository tied to the app. You need to write code of course, but they auto-detect the language and prepare the environment or platform for you (so platform as a service). They tried to appease the geeks by making everything powered by CLI - but otherwise its just point and click for deploying apps.
Tinkering with nginx is good, but what if you want to add a database, a cache, a logging service? What if you want to automatically build code when you push to github? What if you want to easy clone multiple environments (for staging, prod etc) without never doing an SSH into server? What if you want to have versioned releases that you can easily rollback to a specified version or push a branch into one environment and another branch into another environment? And you get a workable subdomain for every environment you build with SSL enabled. All of that can be scaled down and up with literally few clicks.
> but what if you want to add a database, a cache, a logging service?
I... just add it. How do I add it in heroku where everything is "just create an app and push it" and everything is separate "dynos" (whatever that is), each with different pricing, storage, bandwidth etc.
that is true. everything is a separate dyno - means separate machine/vm to which you need to connect to. You get free tier for almost all databases and you only need to know how to connect to it, not tinkering with the storage, tune the parameters, scheduled backups, automatic scalability etc - that's done for you. it will become expensive once you go past free tier or the cheap tiers though.
> not tinkering with the storage, tune the parameters, scheduled backups, automatic scalability etc - that's done for you.
Thing is: if we're looking for free, that may make sense. you still need tinkering though: you need to figure out what the whole dyno thing is about and how to connect from one dyno to another, and so on.
However, if we expand this a tiny bit to cheap and/or free, then simply running the lowest-tier server at Digital Ocean will be a better proposition. And installing a database these days is basically the same thing: run a command to install, all you need is to connect to it. No tinkering. And it will still probably scale significantly better than a free tier at Heroku :)
Once again, I'm biased and old, and can afford to spend money on side projects. I know that sometimes even $15 a month is out of reach (oh, I've been there). But yeah, this sorta prevents me from ever understanding Heroku :)
I think that's basically what Lightsail is supposed to be. I haven't used it myself, but from glancing at it they're pretty clearly targeting DO/Linode/Vultr etc.
I think there's some verticals - players that simplify/aggregate SES, or S3, or whatever - but none that handle being a layer on top of most/all of AWS.
Have to agree with the others. For example, while setting up ELB it's possible to select at least one option (un-checking the Public IP box) that causes the setup to just fail with a nonsensical error message. Turns out ELB requires a public IP to communicate with the nodes. That's just the most glaring one off the top of my head.
I also remember trying to set up SFTP whenever it was first released. It was literally impossible to do what they were advertising (S3-backed SFTP with jailed homes for each user) without writing a manual json config (I never got it to work). I had built my own solution for this exact thing on EC2, using a bash script more or less, and thought the hosted option would be less of a maintenance burden. Needless to say I quickly gave up.
The more annoying part is that the service only supports public/private key logins. If you want user/pass you have to write a lambda. The lambda is pretty simple though, it checks the credentials (so it can hit any backend you like), and if they pass it returns a 200 with a json doc of role (which is just sftp assume role), policy (the scope down from above), and home dir.
This touches on a larger issue with AWS though. It's that they are trending to leave out functionality and point to lambda as the solution. On one hand, I get it. The lambda solution is infinitely more flexible, but what if I just wanted an sftp for a couple users that uses user/pass?
To your final point, it is so much less maintenance. There is no server to manage, and since I want all the data in s3 anyway, it's already there. This solution replaced a server with chroots, EBS, scripts to move data to s3, etc...
terraform 'destroy' isn't infallible. There are certain resources that trigger the creation of other resources (for example, lambda functions will create cloudwatch log groups, and dynamodb tables create cloudwatch alarms) and when terraform destroys the resource, it doesn't necessarily clean up all the associated resources.
Terraform has it's shortcomings for sure, but I don't think it's reasonable to expect Terraform to go out and clean up second-order effects of it's resources.
I'm not doubting that the situations you describe are true, but abandoning resources like that is an AWS-lifecycle problem, not really a Terraform one.
Sure. My point is just that `terraform destroy` doesn't necessarily solve the problem at hand. And you could still end up continue paying for those second-order effects after running a terraform destroy.
I am pretty senior and can thus sometimes afford to do a new thing and try doing it the right way, at the same time. Not even always. Many people just take one learning curve at a time...
Typically Terraform takes longer to get something working than mindlessly clicking through the console. In my experience those mindless clickthrough things end up sticking around for years even when they weren't intended to.
This is why you have separate development and production accounts: a development account where you mindlessly click through so that you can learn through the UI what's available and how it works; cleaned up on a regular basis by something like aws-nuke, and a production account where you have the discipline to only create resources through Terraform / CloudFormation etc.
Even CDK sucks though because if you’re still kinda new to it all, you want to login and make sure that it’s all hooked together correctly. And you’re back using their shitty UI.
Why you can’t look at the load balancer, look at the listener and then show what’s in the target groups is beyond me.
I have a different take on it. I started out doing everything through the console, then learned the cli and boto3, and very recently CDK.
CDK is another tool that build on the cli and boto3 concepts, and also manages the orchestration and dependencies.
Having to go back to the console isn't a fault of CDK. Learning the right tool to use for a situation is part of the learning curve. I go back to the console all the time to look something up quick or to understand what config options are available. Or I repeat the same steps in the console enough times that I get bored with it and automate
Edit: also I tried and have given up on CloudFormation more than once. CDK is like a wrapper layer around it, and has been pleasant to use.
It either works great or barely depending on the service you're using - some AWS teams have dedicated dashboard teams, eg 'ec2 dashboard team' which solely focus on the dashboard experience, while others touch it as an afterthought.
I'm pretty sure something along the lines of this^ was posted on HN by a former AWS employee but I can't find it now.
Their web console UIs vary widely by service. IMO, the UX for most of their simpler services, specifically SQS, S3, Lambda, DynamoDB, are really easy to use and work nicely. If you want to start with a Docker image, send it to AWS, and have it get spun up and attached to a DNS, well that's a huge complex mess to get set up.
I don't understand why so many people want to use AWS for such small budgets. At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?
AWS is awesome when you have a large number of resources, that are created programmatically and reproducibly, with redundancy and duplicate environments.
The budgeting tools are really amazing at letting you categorize your costs and create appropriate alerts. The permissions system lets you define very specific roles.
Its complex. But if your system is complex, it gives you the tools to keep track of it all. If you have a <=$5000/month budget, it is probably too small to make sense in AWS. You can probably run your system on a couple servers.
> At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?
I have a handful of AWS Lambda functions with a DynamoDB backend serving hundreds of clients, my bill for the month of April was $0.01.
No, a VM wouldn't cut it.
But you are right that there are certain slots where AWS doesn't make sense: There's one in the lower middle range where you can save a bunch of money by using a VM or two with your own DB servers. And there's the one where you're so big it might actually be worth it to implement the whole stack yourself.
It's a common backend for chat bots (Discord, Matrix, IRC, Telegram).
Basically it takes different inputs and commands and uses different APIs to fetch data based on the input efficiently without the need for web scraping. DynamoDB is used mostly as a cache for common queries so I don't go over API quotas.
Most of the bots are made by me, but with AWS API Gateway I can easily generate API keys for anyone else and keep track of their usage.
> At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?
No. I looked into AWS vs Azure vs Cloudflare vs Digital Ocean for something I'd like to build this year and (for my thing) Cloudflare (Workers) was the best balance of cost, scalability, and maintainability.
> I don't understand why so many people want to use AWS for such small budgets.
Serverless (I hate the term) makes a lot of sense for small budgets and projects. You're investing very little, so the downside of (severe) vendor lock-in is somewhat low. The upside is HA infrastructure, horizontal scalability, zero maintenance, ignorable monitoring (at least initially), consumption based costs, etc..
The biggest problem with all of them, except Digital Ocean, is that it's difficult to "own" your data in the sense that you can download a copy of a DB and keep using it somewhere else. IIRC Digital Ocean has a fairly nice managed PostgreSQL offering, but it still scales similar to a traditional DB (ie: not automatically).
The next biggest problem with AWS and Azure is that you can't figure out what anything costs, at least not easily. For example, I know for a page I'd want to serve via Cloudflare Workers, I could do 1 hit = 1 point read from Azure CosmosDB, but I couldn't figure out if the pricing includes egress. Just look at the pricing page for CosmosDB [1]. It's ridiculously complicated and that's one service on Azure.
Linux server administration requires a certain amount of know-how and determination. I can't tell you how many times I had to rebuild a DigitalOcean server because I messed something up and wanted to start with a clean slate.
A lot of hackathons, workshops and courses ask you to use AWS these days. Whether it's to run machine learning instances, win a sponsor prize or learn how to use Lambda, students are often encouraged to learn one of the major cloud providers.
Also it's a resume boost. It's another buzzword you can add to your resume.
We give point-in-time run-rates of all active resources based off of the region and resource/service configuration.
In addition we try to simplify people's understanding of where their costs are coming from. If anyone needs help with this, they can personally reach out to me at ben@vantage.sh
Same respect, nice tool, but it is sad it is needed. But I guess it's similar to lawyers, police, army, I don't like the need for them, it, but good they exist.
Because, if you are a company that uses AWS at scale (deploying literally thousands of resources at a time), you care more about meeting demand and getting resources to spec, than you do about the cost of an individual elastic IP...
The price of each individual resource isn't something that you want to see on every screen you touch. It literally clutters the console with information that you couldn't give a shit about when your company is making millions (or billions) on the services you provide. You care more about your service's reliability/scalability/uptime/etc than anything else. This is priority numero uno.
If and when cost analysis becomes priority, you look to see where you may be overprovisioning resources - hence, the billing console.
But for the larger players on AWS (the multi-billion dollar ones that AWS cares more about making happy than you or I), an extra $100k in AWS expenses in a year isn't a worry, it's a write-off.
I agree, having set up serverless and lambda for an api/app that was to be used a few times per day, the billing made no sense at all, it would increase even though the services were used less and was difficult to find what the costs were or whether they could be reduced, eventually I had to shut it down and create a non-lambda solution somewhere else because I had little control of the cost.
Students and self-learners want to experiment with doing complicated things on AWS without getting a surprise $3,000 bill because their script didn't shut down those instances like they thought it did. They want a hard fail-safe to protect them from surprise bills. And they want a solution that's easier and more reliable than checking the AWS itemized breakdown twice a day.
I agree with that. That is why there are "Billing Alarms" in big letters on the billing page. But that wasn't the point of the reply: the person above me was complaining that there's no way to tell what services are being billed, which is bunk.
This is what keeps me from tinkering with the free tiers and never even attempt to host my own projects in the cloud.
I want a free tier with a lock on it. I'd very much appreciate advice for what tiers would fit my monthly use if I hit free tier caps, but if I got hit with $500 right now I'd be ruined.
I think my Google compute stuff is safe, but I really don't like having any doubt.
Until AWS fixes this the best thing to do is just use their service as little as possible. There are plenty of other cloud providers out there these days which don't employ this hostile practice.
I closed an AWS account for this reason just a few days ago, we hadn't used it for a while but there was no clear way to remove our credit card so it felt like a risk just to have it open, what if a developer logs in to mess around and accidentally flips some switch that smacks us with a charge of a couple grand... unlikely, sure, but the fact that it's even possible is terrible system design. Better to just nuke the account and move on to other cloud providers that don't make it harder for me to sleep at night.
Not disagreeing with you (a view with all the active services would be great) but one of the many benefits of using Terraform is that it allows you to know what you running.
Yep and is a huge SaaS market place now so mid to large size companies can figure out what they are spending money on.
I don’t use them for any projects mostly because I already don’t enjoy having to manage AWS at work all the time. I also don’t want to live/work in a world where AWS is my only option so I try and use smaller hosting providers and services like tarsnap.
I cancelled an account 2 years ago because no matter how much I explored cost manager or any regions I used, I can't find why I still get billed a small dollar or so a month.
I still get bills on the 3rd of each month, followed by a "we can't charge your card" (I removed it when they pulled this shit) followed by a few days later you get the "hey we're gonna suspend your account if you don't pay"
Yeah cool, I only closed the account 2 years ago now, go right ahead and suspend/close/do whatever you've gotta do, I stopped caring.
The cost explorer is beyond deplorable. It provides either too little information or information that is so fine-grained it becomes useless.
It infuriates me the hoops we need to jump through just to understand how much a single virtual machine costs. Or in cases like yours - to find out what resource a charge was for. Impossible.
Designed to be impossible. FFS. We would spend more money if we were able to manage costs better.
Same! I literally closed an account and created a new one after a few months because I was charged a few bucks a month and I didn't know what I was paying for.
I think it was either related to db backup or some encryption keys but I couldnt find what and where to delete to get rid of this charge.
Same here. I had a monthly billing of a few dollars and couldn't figure out where it's coming from (and given that it's only a few dollars, it wouldn't make sense to spend many hours investigating). In the end I just canceled the card and hoped for the best.
Yup. Got charged over $800 for "experimenting" with a DynamoDB database and forgetting to delete it afterwards.
Sure, I called customer support and they reversed the charges. But something the nice lady on the other end said as she chuckled: "This happens all the time".
DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
Apparently I owe AWS 1 cent for DNS. The problem is I can't login to pay it, so every month I get a email that says "your aws account is going to be suspended" and 30 days later I'm disappointed that they didn't follow through with the threat.
AWS charges me $1 or $2 every month, and I log into my account and click through every page and can't find a single active service or any clue on what it's for. It's marked "EC2 - Other", whatever the f that means
So an elastic IP is just an AWS provided IP that they hand over to you... it's free while your instance is running but if you stop your instance, you get a small bill (it's like $0.84 a month)
I don't have any elastic IPs. The only thing under EC2 is a security group that it won't let me delete, but seems like no resources. My bill this month is $2.88 /shrug
What does the bill details screen show? I manage an AWS bill and I find it overly detailed. Others mentioned leftover EIPs, but the bill shows these as their own line item.
Not saying it can't happen, but I've never seen an 'Other' listed on the bill details screen. As I said above, it's detailed to a fault.
Actually it looks like this is for an EBS snapshot. Now it seems like EBS is a Lightsail feature, and the Lightsail console says that I don't have any storage or snapshots? Hm
EBS is also the general external storage attached to an instance. Go the ec2 console and look at snapshots and you should see it there. They are region specific, so make sure to select the region referred to on the bill.
I haven't used Lightsail, so I'm not sure if those snapshots end up in the general ec2 console. But, if you already checked the LS console and didn't see anything, then they must be in the ec2 console.
I keep a penny on my desk at work. Every time a coworker comes across somebody's account that is off by a penny, I offer it. They think I'm being funny, but the point I'm trying to make is that it wastes more than pennies worth of our time to spend it worrying about a few cents.
I've seen people mail in checks worth less than the stamp it took to send them. It's a wild world out there.
The interesting thing is, accounts being off by a penny or two (excluding actions such as customers doing a mis-type in the bank transfer) can actually point to deeper issues such as improper rounding somewhere in the path.
Usually they wrote us a check for the wrong amount at some point and end up with a small balance remaining in their account. We don't want to just forgive and forget every time somebody owes us $0.16, but if it's been there months and it's going to take extra effort to get it from them, it's probably not worth it.
We do try to determine where the discrepancy started and watch for indicators of a bigger issue.
They do this to me too every month, as the card on my AWS has expired, and they complain and complain and complain before finally charging the card on my Audible account. Definitely need to get around to separating those.
> DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there.
Did you look at the PDF invoice in the Billing section? I have never seen an invoice where the line items didn't add up to the amount charged to my credit card. If there's just $2.40 on there with no explanation, I'd open a support ticket to complain.
(While looking into this, I was surprised at how minimal DO invoices are, however. For GCP, I'm used to seeing on the order of a million line items per month. Seeing only 3 on my DO invoice was surprising, and could definitely lead to a case where something isn't accounted for correctly. But, I bet support will fix it up for you.)
> DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
Oh I used to get this, for me it was snapshots of a VPS I destroyed. Completely my fault, but yeah it was annoying to figure out at the time. Probably worth checking you don't have any reserved IPs, disk backups or snapshots left over
Honestly I get lost inside of AWS. Only recently was I able to figure out why I was getting charged $.82/month which, in the long run, is really nothing. But it's amazing how hard it was to figure out why I was getting charged for something that I originally thought was just going to be free.
I used the AWS free plan many many years ago as a student. I was so paranoid about losing money that I immediately shut what I was using down after I thought I didn't need it any more (even though leaving it on was part of the free plan). I turned off the instance, but I forgot the public IP. A month later I got charted a few pennies because an associated public IP is part of the free plan, but an un-associated one isn't.
I didn't go bankrupt, but it proved to me how scummy AWS can be given that actually trying to use less is what got me charged.
I have a DO server thats been running for more rhan 5 years. It was the basic $5 bucks instance they offered at the time I felt a bit robbed a couple of days ago when I was spinning another instance and I saw that the newer $5 instance has more memory and more cpu power. I feel like they should have decreases the cost of the older less performant instance.
I have been spoiled by my Internet provider who year after year they will automatically increase my speed as they made it cheaper, as I paid the same amount for the service.
When they changed the pricing they emailed me and told me that I could effectively reprovision my machine to get the price difference, I think it was part of a rollout of some other infra as well.
The billing statements I see in DO under Billing (sidebar) > Billing history seem pretty verbose to me. If it's not there I would recommend reaching out to their support.
Yeah but it shows they are reasonable and flexible if a customer makes a mistake. The reason Google Cloud is third, people remember how they were treated when they used Google for Work or whatever its called now.
> But something the nice lady on the other end said as she chuckled: "This happens all the time".
Is better then if they said: "We reminded you about the consequences of not deleting, we have done nothing wrong."
>DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
I would not go that far to say it's fraud. Calm down.
Maybe it's something I missed. Maybe its some hidden feature that I did not turn off. But I deleted all my droplets, all my IP's all my firewalls, etc, etc and could not find anything else.
You have in good faith attempted diligently to have a zero purchase. You cannot even find any purchases. In that case they have reached into your pocket and taken money without your consent. That is just a fact. By mistake? It's being short changed in a shop at the absolute best.
This is not a mistake any company should be able to afford to make. Ever. Under any circumstances. If it really is "just an error" they need to making amends in a very public and obvious way that is clearly expensive to them to show good faith.
But we don't get that. We have to fight from a position of being in the wrong when we have money taken from us by these fraudulent practices to overturn them.
It is necessary to go HARD at the morality of "accidental theft" because it is so prevalent and because all these companies clearly don't consider it a detriment to their reputation.
You've been diligent. When does it become their responsibility to not reach into your pocket when you clearly don't want that. Or anyone's pocket. I'd bet quite a sum that where there is one there is many.
This is where I'd say that every single paid service needs to have a way to contact a human being, and any cancellations expressed to that person by an authenticated user must be respected. If services are allowed to construct Gordian knots of byzantine interacting services, then there must be a way to cut through that mess.
I'd double check that you don't have multiple orgs. You used to share a cess to your account with people, then they made some org change a while back that essentially moved that shared access setup to an org and gave you a new personal account iirc. Easy to not realize you have both.
> “It’s the student’s responsibility to know what they’re deploying.”
Anyone who seriously argues this is 100% unaware of the quagmire that is AWS billing. There are companies with entire teams built around just optimizing AWS billing - it's whole unsurprising that some AWS feature actually spun up 5 separate AWS features that end up being billed.
And even when you spin things up knowing their cost you may not be accounting for data transfer, storage, mismatched reserved instance purchases, unused EIPs, and so many more "hidden costs"
Honestly, my company only forces everyone to add tags with ‘platform’ and ‘team’ to every resource that gets deployed to AWS (otherwise your resource is automatically destroyed).
This works perfectly for going to cost explorer and seeing exactly where you are spending your dollars.
I don’t think this is a very high difficulty exercise.
The issue is that when students are "learning" "cloud", they are not really learning cloud as much as following tutorials to click together turnkey solutions. Dynamo DB, ElasicSearch, and whatever else are all business-oriented, quick to set up services. And such, these services are hiding what is going on underneath, with the expectation that the company will cover the costs of whatever those services run. This is a standard buisness practice.
If you want to learn how to put together cloud solutions with those technologies involved, you don't even need an AWS account. All you need is a decent laptop or a desktop, with docker or VM software, where you install everything yourself and learn how to configure it (since all of the software is free), do all the networking yourself. This is what actually learning the cloud involves and translating those skills to AWS is very easy.
Perhaps the blame is on the universities and/or tutorials that push the students towards creating free AWS accounts, but regardless, there should be some incentive to takes one profession seriously so you don't end up making this mistake in a company that will not have protections in place that people want for free tier. And money is as good as incentive as any. Id rather people be smarter rather than treat ignorance as a standard and put blame on others for not expecting people to be ignorant.
You are making the assumption that most people are starting these AWS accounts to "learn cloud" when that isn't the case. AWS advertises a free tier for many functions that have nothing to do with "learning cloud".
For example, in the article, the poor student was using Amazon SageMaker to likely build and train models. Blaming the student and saying "why didn't he just buy his own GPU" when Amazon offered 50 hours of free training comes across of out of touch. If the student had the money for a decent desktop he wouldn't be crying over $200.
Amazon could fix this problem easily by introducing a billing cap. While others say it's malevolence as to why Amazon doesn't build one, I think it's more likely that even Amazon couldn't even add this feature to their sprawling billing infrastructure even if they wanted dto.
Can confirm. Default ElastiCache clustering option chooses an extremely high compute node, I ended up accidentally spending $1300 in a month just for testing some Redis clustering script. The minimum option ends up only costing a few dozen dollars a month. The billing alert did not even trigger until the very end of the month, so I got a $1300 surprise. I complained about it to AWS, mentioning how misleading their shit was, and they ended up refunding me $800. Still -- a $500 mistake anyone could make at 11pm. They also made it extra clear in their response that they are not legally obligated to refund me, and that they were doing it out of the kindness of their hearts.
AWS console is some next level outdated shit that needs to be improved. GCP's console is way cleaner IMO. I really hope AWS fixed this after this happened to me, but somehow I doubt they did.
Felt like a complete idiot, I guess I'm not smart enough to use AWS's UI in a safe way. When we start trusting tools, we inevitably become complacent. Some user flows are designed to be gotcha's. A little gotcha machine that generates |cost - refund| revenue. They gotcha, and there's nothing you can do about it but beg them for a (usually partial) refund. The refund is the thing that makes you feel better about it.
Thats exactly the impression I got when I was reading the AWS docs. I am a student too and was evaluating AWS for a hobby project.
Reading the docs I got the feeling that the product is designed for big cooperations only. But not for someone who wants real price control. The free account just made no sense to me.
I ended up renting a cloud server ~3€ a month. (Not AWS) Looking at this thread this was the best decision I could make.
> Corey Quinn, your first and last stop for any question that touches AWS billing, has called for an updated free tier that treats “personal learning” AWS accounts differently from “new corporate” accounts, and sets hard billing limits that you can’t exceed.
This would be good.
I don't normally do "cloud" stuff. It's just not my skill set. But I have looked at it on occasion and one thing that turns me off is my inability to know if I'm going to fuck myself with a large bill from some of these services.
Yeah this is a huge problem. I suspect this is significant for people to choose inferior non-hosted alternatives just for peace of mind. Hard limits would probably create a large influx of users.
Even for something as simple as S3, I'm hesitant to use the real service during development because it's so easy to stack up charges. For one of my accounts, the usual bill is <$10 but last month it was $27 because there was some network flakiness that resulted in excessive transfer.
I had a class last semester where we were required to use AWS. Our professor tried his best to teach everyone how to be responsible with their tool use and how to predict their costs (As well as how to set up a free education account and get AWS credits). I was amazed when final presentations rolled around my group had a total cost of 4.30 dollars after 5 months of uptime and use, while other groups had costs ranging from 30 to 150 dollars! I use AWS for my job so I guess I just never really went through those growing pains, but no system should be that easy to rack up costs unknowingly.
I had an AWS account that charged me like $6 per month for a year after I thought I had turned everything off. I finally went on a hunt for what was causing it, and had to escalate to support to find it. This was 11/12 my fault I admit, I should have been on it after the first month.
More recently, I was looking for a cloud GPU provider, and tried a different provider that I won't name. I tried out a ~$3 per hour instance, and shut it down after maybe 20 min. A few hours later I (thankfully) started getting billing alerts that my bill would be $1500 ish at the end of the months if my usage kept up. I couldn't find anything still running, could not get anything from support (they had some kind of chat support that told me to open a ticket), first opened a ticket then after and then after ab hour shut down my account as a last ditch effort. At which point they promptly emailed me an invoice for the $16 I had incurred during the time before I cancelled. The help desk relied to my ticket like a week later, asking for more information.
Needless to say, I won't ever use that company again, but it could have been a lot worse, even more so if I was a student or someone who blew their whole experimentation budget on whatever mistake I made.
Hinting at the name like this seems unfair to any other companies that might reasonably be interpreted as "[Vague description of provider now removed]"... especially since I can't decide if you're sarcastic about [piece of description] part.
At the very least [two names] are big names that you could plausibly be referring too... and I could see only one name coming into many peoples minds and then them assuming you're referring to that provider.
Funny story: I keep receiving a $0.01 monthly invoice from AWS for a very old account (opened 5+ years ago) I have lost access to. I have no idea why (the invoice doesn't list the services being used) and the associated credit card has expired a long time ago. They requested notarized documents to prove my identity. Obviously I'd rather settle my $6 (+ interest) debt 50 years from now than give hundreds to a notary today!
Edit: as the other comment mentions, most banks where you have an account offer this for free. My local UPS store where I have a mailbox also offers the service to me for free.
GP is probably not in Germany, but just an example of why I’d be uninterested in getting something notarized here unless my continued freedom and/or good fortune depended upon it:
American notary: someone who doesn’t have a history of fraud, sat through a workshop and passed a test.
German Notar: a very specialized lawyer, bills like a very specialized lawyer.
Theoretically, you can get stuff notarized at US embassies and consulates, for $50 and an appointment, must bring your own witness(es). That of course has not been an option much of the past year.
In the USA, in Marin County, I was charged $25 per document. The guy doing it was adding a finger print, which is not a bad idea if the originator of the document is present.
If notarization isn't common in the country where you are, you should probably escalate through AWS support in that country. They probably have an alternate procedure. If you're talking to US-based support, they are used to thinking this is a trivial request. Getting something notarized is something that can be handled in 5 minutes at places as common as convenience stores. Most people have coworkers that are notaries and can do it without even getting up from their desk.
If that's not the case in your country, ask them for a different procedure. It's not something they intend to be onerous.
I'm not interested in recovering that account anyway, I'd spend more money (in terms of time spent dealing with them) on trying to get it back than what I'd owe in a few decades.
In Germany you can also get documents notarized in (Catholic) parish offices. I think they do it for free.
That said, this sounds unreasonable and if you're not in the US this may violate consumer protection regulations if they didn't require the same level of verification to sign the contract in the first place.
Then you have not provided enough information to validate your claim. For example, the EU imposes price controls as well, while in Japan it can be expensive.
However, there's a much simpler route: Amazon is a US country, so you can simply use notorize.com for $25.
My claim is that it's ridiculous to send $0.01 monthly invoices for an inactive account, not sure what you mean by "validate", maybe it isn't that funny of a story? :) But it doesn't make sense for me to spend even $1 on anything regarding this matter. I'm ok with not having access to that account anyway. If anything they should pay me to go through the hassle.
Oracle - of all businesses - got this right. I know, I'm in shock too.
In Oracle Cloud Infrastructure, after your credit is exhausted, you have to explicitly opt in to billing or else they stop all paid services for you.
Moreover, you can choose to keep billing disabled and use their free services without fear of unknown costs.
AWS have just decided to run with a policy of offering refunds when people make mistakes. Unfortunately, some people are ignorant of this, or too timid to ask for their money back.
>Never thought Oracle would have such a nice free tier.
Attracted a fair bit of attention in some corners of the internet for this reason. Then they canned a bunch of accounts using "always free" stuff for myself and others. Do not trust.
Might just be an urban legend but doesnt Amazon keep track of people who do too many refunds and cancel services with them? I've heard of and met people who dont do minor returns to Amazon because they were afraid it would put their Prime account on some kind of "naughty list"
I thought I was just to incompetent to figure out how to stop an instance…so I just gave up while watching my creditcard getting billed for about €20 each month..
Last month I went full on and spent a few hours removing everything on every screen I could find.
After many hours of failing to remove an instance because you have to stop a thingie from auto starting on another screen first…and another thingie from running there.
I finally managed to remove all the instances…
Then two months later, I still get a bill for a few €…
This will scan your entire account and list all of your resources - it's actually made for generating CloudFormation templates, but it's very useful for a use-case like yours.
I'll bet you created the environment with Elastic Beanstalk, whose job it is to (among other things) replace instances if they fail.
When you stopped the instance in EC2, EB did its job and created a new instance.
You eventually figured this out and killed the EB env, and the instances stopped reappearing.
But the Elastic IP address assigned to your EB env is still on your account, and it's no longer free of charge, because you don't have a running instance.
So you will be billed about a €/mo until you delete the Elastic IP reservation.
This is ridiculously confusing, and ridiculously common.
I used the free tier account to play around for a while. I thought I had deleted everything before the end of the year free tier. I was wrong! For 2 months I paid $10 as I could not figure it out what the heck was running.
I had to close my account in order to stop the charges. Today when I hear anyone speak of AWS free tier for a year, first thing I warn them about is to make sure they keep track of what they create so they will know what to delete, otherwise they will keep buying Starbucks every month to AWS.
Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? I don't use those platforms much but was always surprised I couldn't find anything like that right off the bat. I'll build cloud stuff if it makes sense at work, but if I'm footing the bill I'll stick to something that can provide an actual upper limit.
I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. You can burn through a lot of cloud resources in 3 days.
GCP at least has some provision to get very detailed information about usage (but not cost) that updates in less than an hour. That, to me, is the tool for building something like "shut down our account if usage is too high". It is annoying that you have to code this yourself, but ultimately, it kind of makes sense to me. Cloud providers exist to rent you some hardware (often with "value-add" software); it's the developer and operator's responsibility to account for every machine they request, every byte they send to the network, every query they make to the database, etc. and to have a good reason for that. To some extent, if you don't know where you're sending bytes, or what queries you're making, how do you know if your product is working? How do you know that you're not hacked? Reliability and cost go hand in hand here -- if you're monitoring what you need to assure reliability, costs probably aren't confusingly accumulating out of control.
> I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports.
That does not sound like a good reason, but more like a crappy implementation of usage collection.
I don’t see why a bunch of Google engineers can’t implement real-time billing properly, and see no reason to defend their inability to do their job.
IIRC the excuse is that billing is a separate department and they count all the usage dollars way after you done using it, not in realtime. You would still be able to go over your limit and then who should foot the bill?
Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.
This is my personal opinion, though I do work at AWS:
It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute resources. Now imagine AWS billing, which is arguably far more difficult in scale and complexity.
I also work at AWS (nowhere near billing), so the usual disclaimers apply, but:
I actually have no idea if billing is real-time or not? I think it's mostly batch, but the records aren't necessarily, though they may be aggregated a bit.
The general point in this thread certainly holds: our systems provide service first, bill second, and that by throwing a record over the wall to some other internal system. It's not unthinkable they could tally up your costs as you go, but the expense has fundamentally already happened, and that's the disconnect.
It would be hard to react ahead of time. Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons. Expensive big-bang on/off things, probably simpler, though I can think of a few sticking points.
It would be hard to react after the fact, too. Where does a bill come from? My own team is deeply internal, several layers removed from anything you're recognize on an invoice, but we're the one generating and measuring costs. Precise attribution would be a problem in and of itself- cutting off service means traversing those layers in reverse, then figuring out what "cut off" means in our specific context. That's new systems and new code all around, repeat for all of AWS- there's a huge organizational problem here on top of the technical one.
I could see some individual teams doing something like this for just their services, but AWS-wide would be a big undertaking.
I wish we had it- I'd sleep a little better at night, myself- but from my limited perspective, it sure looks like we're fundamentally not designed for this.
“Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons”
That’s hardly necessary. Let’s suppose you have some service that costs 1 cent every 1,000 queries. If you’re billing it then you need to be keeping track of it and incrementing some counter somewhere. If old number mod x < new number mod x them do some check, that’s very cheap on average and doesn’t add latency if done after the fact.
PS: Phone companies can pull this stuff off successfully for millions of cellphones. If you’re arguing keeping up with AT&T is to hard, you have serious organizational problems.
That counter may well not exist outside of billing for longer than it takes to batch some records together. It will need to be shared and synchronized with the rest of the fleet, the other fleets in the availability zone, the other zones in the region, the other regions, and every other service in AWS. There are strict rules about crossing these boundaries for sake of failure isolation.
As an amusing extra sticking point, your service has no idea how much it actually costs, because that's calculated by billing- the rates may vary from request to request or from customer to customer.
Without spending way too long thinking about it, the complexity in figuring out exactly when to stop is significant enough that it probably cannot practically be done in the critical path of these kinds of high-volume systems, hence the reactive approach being more plausible to me.
I don't know what kinds of problems AT&T has, but at the risk of saying dumb things about an industry I know next to nothing about, your phone is only attached to one tower at a time, and that probably helps a bit. And I'm not sure when it wouldn't be simpler and just as good for them to also react after the fact, anyway.
First arguing based on existing infrastructure ignores the fact your changing the system therefore any new system is a viable option. All the existing system changes is how much things cost. Anyway, for independent distributed systems you can use probability rather than fixed numbers.
That said, your losing the forest for the trees, the accuracy isn’t that important. You can still bill for actual usage. A 15 minute granularity is vastly better than a 30 day one. As long as you can kill processes you don’t need to check in the middle of every action. Things being asynchronous is just the cost of business at scale.
I'm hardly saying it's impossible; I'm saying that it's not easy, and may even be hard. Doing it well would likely require a wide-reaching effort the likes of which would eventually reach my ears, and the fact that I haven't heard of such a thing implies to me that it's probably not an AWS priority.
> PS: Phone companies can pull this stuff off successfully for millions of cellphones. If you’re arguing keeping up with AT&T is to hard, you have serious organizational problems.
To be fair, AT&T in particular does prepaid shutoffs on a pretty coarse granularity, I think it's like 15-minute intervals.
I know this because for a while I had to use a prepaid LTE modem as my primary internet connection. You can use as much bandwidth as you want for the remainder of the 15-minute interval in which you exceed what you've paid for -- then they shut you off.
I once managed (by accident) to get 3GB out of a 2GB plan purchase because of this.
Of course that free 1GB was only free because I consumed all of it in the 14.9 minute time period preceding NO CARRIER.
> There’s a lot of middle ground between credit limit checks within every database transaction and the current state.
But there isn’t a lot of middle ground between distributed, strongly-consistent credit limit checks every API call and billing increment (which is, IIRC, instance-seconds or smaller for some time-based services) and a hard billing cap that is actually usable on a system structured like AWS. Partial solutions reduce the risk but don’t eliminate the problem, and at AWS scale reducing the risk means you still have significant numbers of people reliant on the “call customer service” mitigation, and how much spending and system compromise to narrow this billing issue is worthwhile if you are still in that position?
> the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical
You don't have to bill in real time.
You just have to provision funding for every resource except network bandwidth.
Customer sets a monthly spend limit. Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit. If the spend limit would go negative, you refuse to create the new resource.
If the spend limit is still positive, the remaining amount is divided by the number of seconds remaining in the month times the bandwidth cost. The result becomes that customer's network throughput limit. Update that setting in your routers (qdisc in Linux) as part of the API call that allocated a resource. If you claim your routers don't have a limit like this I call shenanighans.
This should work perfectly for one region.
There's probably a way to generalize it to multiple regions, but I'm sure most small/medium customers would be happy enough to have a budget for each region. They'd probably set most regions' budget to zero and just worry about one or two.
The web UI probably would need to be updated to show the customer "here is what your bandwidth limit for the rest of the month will be if you proceed; are you sure?". JSON APIs can return this value when invoked in dry-run mode.
> Customer sets a monthly spend limit. Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit. If the spend limit would go negative, you refuse to create the new resource.
AWS systems are highly distributed; this kind of sharp billing cap would necessarily introduce a new strong consistency requirement across multiple services, many of which aren’t even strongly consistent considered one at a time (and that’s often true even if you limit to a single region.)
> Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit
For the motivating use case (avoiding a bill on the scale of even $200—possibly even $1—from a free-tier-eligible account), using monthly chunks doesn’t work; you suddenly couldn’t spin up a second simultaneous EC2 instance of ant kind after an initial t3.micro instance, which would cutoff many normal free tier usage patterns.
I mean, that’s a good way of capping if you are using AWS as a super overpriced steady-state VPS, but that’s not really the usage pattern that causes the risks that the cap idea is intended to protect against.
This is a particularly poor solution to completely the wrong problem.
Hogwash, I tried to spin up 100 of your F1 instances in us-east-1 a week or two after they first became available, and found out about this thing called "limits".
Wherever you're enforcing the limit on max number of instances per region is already a synchronization point of exactly the sort needed here.
I'm sorry, this just doesn't pass the bullshit test. Resource allocation API calls are not even remotely close to lightning-quick. There is no fundamental immutable constraint here.
> For the motivating use case (avoiding a bill on the scale of even $200—possibly even $1—from a free-tier-eligible account),
Avoiding a $1 bill is definitely not the motivating use case.
A lot of people would be happy to have a mechanism that could prevent them from being billed 5x their expected expenditure (i.e. they set their budget limit to 5x what they intend to spend). It doesn't matter that that isn't perfect. It is massively better than what you're offering right now.
> Hogwash, I tried to spin up 100 of your F1 instances
I don’t have any F1 instances. Have you mistaken me for an AWS employee rather than a user?
> in us-east-1 a week or two after they first became available, and found out about this thing called "limits".
Yes, individual services, especially in individual regions, and especially a single type of resource within a service within a region like, say, instances in EC2, are often at least enough like centralized to impose hard limits reasonably well.
Billing accounts (and individual real projects which—and this is one disadvantage AWS has vs, say, GCP—AWS has only the weakest concept of) tend to span multiple resource types in each of multiple services, and sometimes multiple regions.
> Resource allocation API calls are not even remotely close to lightning-quick.
Resource allocation API calls that have high latency aren’t the only API calls that cost money and would need coordination. Heck, API calls aren’t the only thing that costs money.
> Update that setting in your routers (qdisc in Linux) as part of the API call that allocated a resource. If you claim your routers don't have a limit like this I call shenanighans.
Eh. AWS's edge network is highly distributed. Unless you want an even split of your rate limit across every possible way out of the network, you'd be much better off settling for an even split across your EC2 instances, and there's no room for bursting in this model. Enforcing per-instance limits (on any dimension) sounds pretty feasible, though.
This wouldn't generalize straightforwardly to services that don't have obvious choke points that can impose this sort of throttling, such as, I think, DynamoDB.
> You would still be able to go over your limit and then who should foot the bill?
The provider. What they do wouldn't be accepted in any other industry. Imagine hiring an appliance repair shop who sends a repair person that can fix your stuff immediately, but can't tell you what it's going to cost until 3 days after the work is done.
Then you get a huge bill because you wanted "appliance repair" (one of them), but ended up with "appliance maintenance" (all of them).
A lot of people have thoughfully responded with reasons why this doesn't or can't exist: real-time billing is far too expensive to implement, better to get a huge bill than to lose data or shut down critical systems, etc. I guess it makes sense—ideally you are monitoring your stuff, whether you're using your own tools or built-in ones—and you know ahead of time when your usage is creeping up. Also, I suppose only the customer can really know which systems can be shut down/deallocated to save money and which ones would kill the company if shut down. It sounds like if you're a small startup strapped for resources, you can avoid these bills either by self-hosting or by being careful about how you build your cloud infrastructure. I.e. maybe you could host your app on your own box OR on an equivalent VM in Azure that's just going to fail if it runs out of disk/CPU/outgoing bandwidth instead of autoscaling to outrageous levels.
That’s extremely hard to design, at least with the current state of what AWS bills and does not bill.
Example: let’s assume you’ve set the cut-off budget too strict, spun off another shard for your stateful service (DB for example), it received and stored some data during the short window before some other service brought whole account over budget (i.e. paid egress crossed the threshold).
To bring VM and EBS charges to 0 (to implement your idea of ‘shut down everything’) AWS will have to delete production data.
While it may be OK when you’re experimenting, it’s really hard to tell in automated way.
So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.
—
There may be some other way to do this, maybe mark some services expendable or not, so they are stopped in the event of overspend.
A complicated solution is not what people are really asking for though.
What I and I expect most people want is a cap which then spins down services when they reach that cap. Nobody is going to care if the cap is set to $1000 and the final bill is $1,020. The problem being solved for is not wanting to have to ever worry about missing an alert and waking up to a bill that is a factor or two beyond expectations. I can afford my bill being 10% or even 40% above my expectation. I can't afford my bill being 500% off.
I do understand that, but there are services that are still billed for when ‘spun down’. To stop getting the bills they have to be terminated.
The solution seems to be to implement ‘emergency stop’ when whole account is put to pause but no state is purged. And then you’ll probably have a week or two to react and decide if you want larger bill, salvage the data or just terminate it all.
> So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.
Another option would be to "reserve" them in the budget. That is, when for instance you create a 10 GB EBS volume, count it in the budget as if it will be kept for the whole month (and all months after that). Yes, that would mean an EBS volume costing $X per day would count in the budget as 31*$X, but it would prevent both going over the limit and having to lose data.
We have a batch process that uses S3 as a way to temporarily store some big datasets while another streams and processes then and, once complete, removes them. This takes like an hour.
So our S3 bill could be, let’s say, $10/mo. If you went and looked at your S3 bill and saw that and thought setting a $20 cap would give you reasonable headroom you’d be sorely surprised when S3 stopped accepting writes or other services started getting shut down or failing next time the batch job triggered.
Under this system, the budget and actual bill need to be off by a factor of more than 10 ($240). And this also doesn’t stop your bill being off by a factor of 10 from what you “wanted” to set your limit to. More than the $200 under discussion.
I think there's a good argument that they could do better. But there's also probably an argument that harder circuit breakers would result in posts like "AWS destroyed my business right when I was featured in the New York Times"--including things like deleting production data. I'm sort of sympathetic to the idea that AWS should have an experimental "rip it all down" option and kill all stateless services option but that adds another layer of complexity and introduces more edge cases.
Merely being on free tier is a signal that you do not want a $27,000 bill. There is no excuse for what AWS does here, and it is clear they use this clusterfuck as a revenue stream.
It’s a lot easier for them to refund the individuals who make mistakes than deal with the fallout and bad press from the small businesses they’ve destroyed when they incorrectly enable that option.
Yes, a useful last resort safeguard would need to be more granular than just "turn everything off", at least if we're talking about protecting production systems rather than just people learning and wanting to avoid inadvertently leaving the free tier or something like that.
Still, it's not hard to imagine some simple refinements that might be practical. For example, an option to preserve data in DB or other storage services but stop all other activity might be a useful safety net for a lot of AWS customers. It wouldn't cut costs to 0, but that's not necessarily the optimal outcome for customers running in production anyway. It would probably mitigate the greatest risks of runaway costs, though.
You can get alerts. And there are some limits for young accounts. But they really need a "beginner" mode. Or a budget mode, put $x on the account can't spend more. But I guess they are making a lot of money with "mistakes" so there may not be kuch incentives
On top of the other reasons for complexity and delay, this just creates another potential mistake where people delete their entire accounts or stop production services.
It's far easier to negotiate dollar amounts than lost data or service uptime.
Azure will alert you when you exceed a budget, but it won't disable anything.
Azure MongoDB billing was insanity. I was up to $800 to host a couple of GB that wasn't doing anything.
I'm still not sure what happened, even their support kept saying "it's priced by request units" and I kept saying "How does a handful of queries a day translate to $40 in request units?"
A year later, I think that I had a lot of collections and they seem to charge per-collection, but I'm still not even sure. Thank goodness we moved off of it after only a month or two.
I work for MongoDB. The writer is referring to CosmosDB, the Microsoft clone of MongoDB. You can runa real MongoDB cluster on Azure with MongoDB Atlas. The pricing model for Atlas is per Node size + disk + memory + data transfer. It's generally easier to predict your costs using this model. Users tend to over-provision with this model so we recommend using auto-scaling. This will allow your cluster to scale up or down based on load (price will adjust as well).
> Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"?
All of them have billing APIs which should, in principal, allow you to build “Shutdown everything at some indefinite time interval (and additional spend) after I exceed $X”, though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, and actually stopping all spend is likely going to mean nuking storage, not just halting “active” services.
None of them provide anyway to more than probabilistically do “Shut everything down before I exceed, and in a way which prevents exceeding, $X.”
The whole point of a "spending kill switch" is as a backstop when you make a mistake; but if you do it as a "DIY project", what prevents you from making a mistake on it? It has to be a built-in feature.
If its done first party, someone (realistically, af AWS’ scale, a substantial number of someones) will mess up using it and nuke their account.
Customer service can correct “we screwed up and have a giant bill” easier than “we screwed up and lost all our data”.
So its not going to happen first party.
(It’s also not really possible DIY as a hard certain cutoff at or before a limit, only at an indefinite interval in time and money beyond a limit. So you still have potentially unbounded expense.)
The problem is that a billing alert at $10 won't prevent you from accidentally starting up something that will bring you a $1000 bill before you can react to that email - there needs to be a process that cuts off service at some hard limit instead of just sending an email and continuing to spend money.
As the article points out: Every other cloud does this! They all have non-production subscription types with hard spending limits.
There's a difference between "unable" and "unwilling".
The people that refuse to understand the difference are the same people that don't understand that "unsupported" isn't synonymous with "cannot be made to function".
Don't be that person.
If you have a large account and you're in regular contact with an AWS sales representative, pressure them into making this happen. Even if you work for Megacorp with a $$$ budget, keep in mind that your future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.
Don't walk down the same path as IBM's mainframes, where no student anywhere ever has been able to learn on their own, making it a dead-end for corporations who pay billions for this platform. You, or your company will eventually pay this price if AWS keeps this IBM-like behaviour up.
Think of the big picture, not just your own immediate personal situation.
Apply this pressure yourself, because the students you want to hire next year can't.