I don't understand why there isn't at least a setting that says "turn everything off if I hit $x."
Then just given people a certain grace period to reactivate or get their data out before it's removed.
It wouldn't fix production deployments where you want alarms, not a shutdown, when you hit spending caps, but it would help people on the dev stage to avoid issues like this.
So let’s say they “turn everything off”. Does that include deleting all of your objects in S3? Deleting your database? Deleting your attached disks (AMIs)? Deleting your DNS entries?
>Does that include deleting all of your objects in S3? Deleting your database? Deleting your attached disks (AMIs)? Deleting your DNS entries?
People bring this every time. You give a grace period and then yes - assuming the user opted into a "turn everything off".
Its not some impossible engineering feat & clouds are full of "out of money data gone" stuff already. e.g.
>If a paid subscription ends or is terminated, Microsoft retains customer data stored in Microsoft 365 in a limited-function account for 90 days to enable the subscriber to extract the data.
If it a personal site I run for kicks and expect to cost less than $100/year, but is suddenly running into thousands? Yes please, delete it all. It's the only way I could sleep at night.
Sure. I'd enable that on dev / testing accounts without hesitation. I don't know why so many people pretend like everyone will be forced into getting resources deleted if there's an option of a hard limit. You can have multiple accounts. They even recommend it.
It makes way more sense for me to build my stuff using a dev/testing account. After that I'll have a good enough understanding of the resources I'm using that it's practical for me to configure more complicated cost controls using a production account.
That sounds like Amazon taking one of their problems and pushing it onto the users. If the root limitation is that Amazon's billing process is so poor that it can't interact with their other processes, then that should be Amazon's problem to fix. Until and unless Amazon fixes their own problem, Amazon should be eating the cost resulting from Amazon allocating server usage beyond a user-specified maximum billing.
As a personal user, I wish they gave two options to fix this.
Option 1:
When credit balance reaches a certain level (or monthly spend reaches a certain level), initiate a resource stop on every resource that can be stopped without data loss. This would still incur charges for some things like EBS volumes, S3 data, etc, but at least it would slow the bleeding.
Option 2:
I don't care about data loss, just terminate everything when I hit the threshold. This should require a double-opt in and maybe a warning banner in the console UI.
Author here. I just updated the post with an additional idea for fixing the free tier, suggested by several readers.
It turns out there is a non-widely-known program called AWS Educate Starter Accounts [0], which give no-credit-card access to a limited but useful subset of AWS services. The problem is that you can only get access to these accounts through student affiliation with a participating educational institution like a high school or university.
It might be more feasible to expand this program, say to any applicant who demonstrates some reasonable threshold of non-bot-ness, than to re-engineer the normal free tier.
To any AWS people reading this: I believe this could be a useful step toward solving the free tier problem, and would be happy to be a sounding board.
Arguably discovering surprise bills absolutely should be part of the free tier, how else can you mentally prepare for running something in production on AWS?
If someone told me about a cool new programming language and to teach myself how to use it could either be free or maybe $5000 because infinite loops are expensive, am I going to learn it? Hell no.
I tried signing up with a prepaid credit card and they refused the card so I moved on. It's setup for massive profits on minor mistakes. The risk on a free tier is like shorting a stock, one bad day and you go bankrupt unless you have a connected twitter account.
Sounds like a new dystopian future that is best not to be part of.
When IBM ruled or Microsoft or AOL things you had one main evil corp. We're in a period where Google, Apple, Amazon, Microsoft, Facebook can appear the hero or villian depending on the day but the sum of the faangs is much greater than the worst evils of the original megacorps. Could you envision forced unlimited billing on a free tier with no ability to limit charges on the account?
When I read about the myriad predatory practices of Amazon I think about who the predator is and the saying "the fish rots from the head." I'm looking forward to the day Jeff Bezos pays the price for being the predator that he is.
He’s out of the crosshairs, and there is someone else running the show - they may end up reporting to him, but similar to Bill Gates and ballmer; spotlight isn’t on him anymore and he’s not making the day to day calls anymore.
I have a couple of $20 AWS vouchers from various things. I kind of want to give them to juniors and tell them to go and learn the product. But I won't, because if someone incurs a charge accidentally it'll be on me.
These threads always open a discussion about how hard limits would be unacceptable to some businesses, but the opposite also applies in other scenarios.
I took a grad course course 2 years ago on cloud computing and we were showed how to setup a student account. I ran a few machines, shut them down, mostly did the stuff on a local server.
Suddenly I started getting hit with these real small charges each week. I never did figure it out and I certainly didn’t willfully authorize them. I just paid and then I annoyed them to shut off the account. I never could figure out what they were for.
I know many of you would hate this but I would love an option to shutdown everything until the bill is paid option. I bet the description of what I was paying for would have been a lot more helpful if they were losing the business then when they just auto charged!
AWS was charging me 0.50¢ a month for over a year for an account that was tied to a Google Suite email address for a failed startup I was part of.
I couldn't recover access to the account. They couldn't figure how to give me access. They wouldn't just remove my credit card from the account.
One month the charge became $10 so I called my bank and had them block AWS charges. It seemed the only way to deal with the fear that someday that would creep up and up.
This appears to be by design. People have been demanding a budget capping service [0] for a decade now. AWS would continue to bill for compromised accounts [1]. AWS "free" offerings can cost a lot more than you think [2].
I once had a cloud computing class and a lot of it was based on the AWS free tier. The number of students who got dinged and needed the professor to pull strings was... too many.
The thing that always gets me: I _think_ some of this is a solved problem within AWS. I've never used it before, but the AWS Educate "offers students no-cost access to a specified, capped amount of AWS cloud resources without requiring a credit card for payment".
That sounds a lot like what I want a free tier to be. Let me play for a bit, set hard guard rails so I can't accidently spend $100, or much much worse. I mean, I can understand requiring a CC to do this, but that's mostly so I don't spin up a ton of free accounts and also so the friction to going to a paid plan is lower.
It's too late for me, my personal AWS account is well outside of the free tier, but the first time I spun on a large instance to test it out I was so insanely nervous I'd screw up and end up with a huge bill. I can totally see someone else backing out at that point out of fear, but if I knew I'd hit the guardrail before spending money, even if it meant losing an instance, I'd have totally be happier to play around.
Lots of companies also gets hacked each month for thousands of dollars because some key to S3 with too many privileges gets leaked.
The entire system is completely sinister. The fact that keys pertaining to S3 has anything do with being able to start hundreds of VM's in different parts of the AWS system or do whatever is bad.
I've seen companies be ruined by this, and it's in no way obvious how stupid their system is. You have to read huge manuals to know how to "only give access to s3" through a key.
Instead of starting with "no access" then adding atomized access you have to understand this extremely complex "json privilege system". Instead of just programming, this is the only allowed IP, the is the only allowed bucket, this is the only allowed service, and my max is 200usd, or something to that effect.
Also the fact that a key can start new services that are billable is almost criminal in my mind when people don't even gen an email when it happens - makes zero sense.
> Instead of starting with "no access" then adding atomized access you have to understand this extremely complex "json privilege system". Instead of just programming, this is the only allowed IP, the is the only allowed bucket, this is the only allowed service, ...
What would that look like if it’s not going to be a series of access permissions and filters represented as JSON?
Security is never a simple checkbox and complaints like this about it needing to be simpler need to be backed up with an alternative. I genuinely wonder what alternative there is to the current permissions model.
It’s incredibly expressive and doesn’t take that long to understand. People who cannot master it would likely leave some other side door open anyway.
> ... and my max is 200usd, or something to that effect.
This has been a valid complaint for years. Though to solve it you need to answer what happens to legit resources when your billing cap is reached. Do all your ephemeral serves turn off? Do your EBs volumes all her deleted? Do your S3 objects all disappear?
Json is great. Their implementation of it, not so much.
Way to complex only needing to give permission to a single "thing". And much of the naming makes little sense to new yes.
DigitalOceans interface is easier for 90% of people.
But to be honest it's years ago that i worked with AWS, but i remember everything being way more complex than it had to be and i had to use days undestanding their enourmous interface and "usersystem". And as long as this is a problem that pops up again and again, something is very wrong with their system i would say.
If not creating a simpler interface / user system for regular users then at least give some huge warnings that unless you know what you are doing a key with the wrong JSON means access to all funds on the credit card.
The feature I most want from AWS is a simple way to create credentials that are only allowed to read from or write to a specific S3 bucket.
The way you do this at the moment genuinely involves copying and pasting JSON policy documents around! It's horrific.
I want this for myself, but more importantly I want it for users of software that I write. I would love to be able to build something that stores a user's data in an S3 bucket that they own (and are billed for directly) - but it's currently just too difficult to talk them through setting up the bucket and creating the right credentials for it.
They simply have to be in separate AWS accounts for this to work. To that end, you can provide them with a CloudFormation template that deploys a stack with the necessary configuration.
I've stayed away from AWS for these reasons. Instead I use systems on top of AWS, like Vercel etc.
Is it ironic that Amazon's mantra is to be "Customer obsessed" yet AWS is so magnificently confusing for anyone not doing it full-time?
As a designer I've used plenty of Digital Ocean, Vercel, Cloudflare Workers and other static hosts without a problem... I've never been able to figure out how to even start on AWS, and all these horror stories constantly make sure I stay away
I'm so miserable that I finally got around to embracing open ecosystems after almost a decade of being Microsoft-poisoned and now everyone is all in on some baffling walled garden with abysmal UX and about 300 different services that you need a damn NASA PhD to understand.
Like the number of times I find myself in some random service/feature/part of AWS like "do we use one of these? how would I even tell?".
The crazy thing is, this obvious (to anyone experienced in business, at least) danger must be costing AWS money. I personally know of small businesses who have entirely avoided AWS even though it might have been a good fit for their needs in most respects, entirely because of concerns about the opacity of the billing and the inability to add safeguards in case something goes wrong. Some of those businesses are no longer small, either. AWS does have a reputation, at least among those more familiar with cloud services, for being reasonable about unexpected charges and probably putting something right if it was obviously not intended. But in that case, they aren't even pulling some sort of dark pattern scam to make more money here, and the lack of last resort safety features makes even less sense...
I am a student in the same age. When I wanted to start a hobby project I looked into the AWS free tier too. It is so unclear how the end costs come together that I decided to just leave AWS alone.
I rented a small cloud instance from HETZNER for ~ 3€/month and host everything myself. If I need a DBaaS I go with the MongoDB Atlas or ElephantSQL free tier.
I prefer constantly paying 3€, over a free account that breaks my neck if I miss something.
The exact thing happened to me one year ago, I was 17 and using my dad's credit card to test out Lambda and SageMaker, I had assured him it won't cost anything since I'd be using the free tier.
However, my application instance somehow kept running ( was a total noob to AWS ) and I got charged over $300 the next month when I got a monthly report in an email. I panicked and literally just deleted my account. Yes, I just nuked my account like the article mentions. AWS never reached out after that.
Hey I did the same when I first used it. Raked up 9000 Rupees. Cancelled the card and called them, customer care said no worries and the bill was gone.
I'm in agreement with the cloud providers over those of you wanting a hard shutdown.
Businesses, the entities that are paying the most money to AWS, will NOT want a hard shutdown. When you generate revenue off of your SaaS service maybe you'll understand.
No, I won't be pushing my TAM to enable a forced shutdown due to budget metrics.
Not to mention, how does AWS decide what to shutdown and what to delete? It's not like it's only running resources that cost money, what about all my data that's stored???
> Businesses, the entities that are paying the most money to AWS, will NOT want a hard shutdown.
I don't think anyone is arguing that any account MUST take advantage of hard shutdowns. Only that it should be an option for those who truly do want it.
> How does AWS decide what to shutdown and what to delete?
That's a much more interesting question. It tends to be services, rather than storage, that are the root cause of these outlandishly large surprise bills. But if, by chance, it's S3 that's running up the bill, what would a hard shutdown mean? Which data does Amazon delete first to get the account down to its hard limit?
I had an AWS account for one of my businesses, and decided to bring it in-house about 2 months ago.
went through the account deletion, and got yet charge/another bill today - account is deleted, so can't log in to see why I'm being charged.
hopefully their support will help out, but not holding my breath.
editing to add: billing was for db backups, I terminated the db, and no clue how to remove the automatic backups it made. of course I can't log in to look any further.
The issue here is really Billing... AWS Budgets allows you to set a budget as far down as $1 USD... however because billing is done piecemeal, it is possible to have up to 24 hour delay for charges...
You can spend $500,000 on AWS in 24 hours, it's not difficult, it's not easy but you totally can blow past your "budget" because it's not a hard cap.
A major problem is that there's no easy way to see all resources in a single view for any cloud.
Azure comes closest in the dashboard but still misses some items. GCP has an Asset Inventory page in the Security section for an organization. AWS can use the Tag Editor to browse all the regions. It's notable that none of them have a single clear page though.
Oh come on. We all know that the 'accidental revenue' from the way Free Tier is set up probably makes up a cool 2 million or more annually. Plenty to justify its continued abuse of naive students. Why would they walk away from that cash? The only people they're pissing off is people who aren't using AWS anyway.
Anything that they can get from "naive students" and developers who don't notice small recurring charges is so utterly insignificant that it can't justify any decision whatsoever. If the public relations aspects of it cause even a 0.01% change in AWS growth, that's already $5 million of lost revenue; if you can assure developers worldwide that it's not so risky to try and adopt AWS a bit more and get a 1% extra growth, that would be worth $500 million and justify walking away from all kinds of irrelevantly small cash flows.
If bad PR affected their business, they might have something to consider. But this will never affect their growth. They're not going to lose a single solitary sale that isn't already accounted for by their standard profit model. The only bad PR that would affect their business is if their actual reputation as a service provider were tarnished. Charging people money for a service they used does not tarnish AWS's reputation. It's their whole business model.
When you stay in a hotel room longer than you're supposed to and they charge you for another half day, or you eat food from the mini fridge, or buy a pay-per-view movie, etc, you could claim ignorance, and create stories about how terrible it is that people get charged for these things unknowingly. But Holiday Inn's bookings are still not going to take a noticeable hit.
Moreover, they don't even need to offer a Free Tier to get people to use AWS. They're AWS. They're the gold standard. It's like saying IBM would need a Free Tier (back when IBM wasn't trash). It wasn't a question of whether you should use IBM, it was whether you could afford it.
I’ve heard of plenty of people accidentally blasting past the free tier, or past dev credits, or other ways where they got burned by a surprise bill. 100% of those people have gotten a refund from AWS support by filing a ticket.
AWS pretty clearly isn’t raking in money on this. Even at your imaginary 2 million a year, that’s a drop in the bucket for AWS, let alone Amazon.
It's got to be Hanlon's Razor here, I doubt AWS is twirling their mustache and cackling maniacally. It's probably a difficult problem to solve and not easily attachable to a profit center or something, so nobody feels empowered to actually fix it.
I'm sure some crusader within AWS could get it done if they tried hard enough and collected the data to show the negative side effects and how they effect AWS.
An article like this one is hopefully going to get someone within AWS to do just that.
I think this is actually the failure mode for companies like AWS: they may earnestly be so large/complex that no well-intentioned crusader can push through an improvement like this.
My gut is that this is actually the cause of most of the “evil” things that “large” companies do, where eventually they get so big and/or complicated that organizing an organic positive change isn’t plausible.
One person isn’t going to come remotely close to fixing this. I work for AWS/Azure/GCP and can tell you first hand that billing is an insanely hard problem to solve. It’s all done in batch by the service itself and pushed to a centralized service once all the hard work has been done. Each service has a small, underfunded team that handles billing and their last thought is some random guy spending a few grand on his personal account. Every ounce of their energy is going to reducing their own COGs and reacting to the asks of the biggest customers.
Obfuscated billing helps Amazon with $350/hour engineers, where an engineer figuring out where $10k is going each year isn't a top priority. Been there, done that.
Obfuscated billing is definitely not there to hit free tier or poor folks. AWS has an awesome reputation for just about everything, and it isn't worth the reputation hit. Free tier is there so that:
* the student in college will pick AWS in their first job
* the random engineer will prototype something on the weekend, and Amazon gets millions of business
* the random pre-funding startup starts building on AWS, and if it goes big, so does the account
And it works. Amazon has made millions based on things I've built on free tier. AWS' problem is that my ONE YEAR free tier ran out probably around a decade ago, and I've long since moved on from what I was doing then. If AWS were to continue to provide me a free tier, the same thing would have happened a few times since.
I've gotten thousands of euro of 'free' Amazon cloud credits over the years, and would love to have used tried it out for cloud computing/GPU stuff, but the opaqueness of pricing means I won't touch it (except for S3 backups) - I just don't trust myself enough to not mess up.
Just speaking from personal anecdotes, every time I’ve accidentally been charged for something on AWS that I didn’t intend, their support team has refunded me without much hassle. Things may have changed in recent times but I’ve found them to be pretty reasonable about it.
AWS is not the cloud. There are others. Please try something else. As someone with extensive professional experience with it day in and out, it is overrated.
Also, the cloud is someone else's computer. It is either AWS or GCP or something else. Move to what works, for you.
At one point I tried using AWS trial for an app, didn't work out, cancelled the account (or so I thought) only to find 2 years later some charges to my bank of $2k and when I spoke to AWS support, there was another $4k accruing for this billing period.
Took ages to sort out, while they eventually did refund the amount, I was still out of pocket exchange fees (X2) the exchange rate also meant I was out more money. In total I think having a compromised yet closed AWS free trial cost me $350 over a 6week period some 2 years after closing it.
We were inexperienced, got picked up by a tiny VC and funded for a few thousand. We applied and got accepted into the Digital Ocean Hatch program, they threw $100,000 in credits at us to help us succeed.
We tried to exhaust the supply—one day the credits vanished. I was getting married that month, and it took that long to accrue a life-changing amount of real dollar debt. I sent a quick email asking DO to extend the credits. The last email I got from them was that "we expect you pay the full amount".
Yes, there should be easier ways for learners to experiment with AWS without risk. In general, people should be made aware of how much something could cost them before they can agree with it. We have regulations towards this end when it comes to investments. Huge unexpected bills used to be a big problem with phone companies, and it took a lot of effort to move past it. I hope the cloud industry can make progress in this direction.
What I'm going to say next should not be taken to detract from that in any way.
(And I know you don't want to hear it, and I could just save everyone some trouble and not say it. But I'm going to say it.)
When I read this:
> please help I made a ticket and called support but i really need to make sure this is dead please Im 20 i really dont have $200 for them please help
I honestly did a double take when I got to the "20". If you'd put a blank in there and asked me to fill it, I would have said "13". I'm still not 100% sure that the blogger didn't alter the age to obfuscate personal details.
I don't mean to single out this person as especially immature, quite the opposite: the interesting thing is that they assume that a 20 year old is obviously a smol bean who could not possibly be expected to figure out AWS billing or come up with $200, _and we agree_. The notion didn't stand out as surprising to the blogger, or to any of the other commenters on HN so far.
I'm not saying they're wrong! I just find this a remarkable signal of how far we, as a culture, have gone in extending childhood well into college age.
This is the reason I closed my aws account and gave up on learning their services. Intentional cost obfuscation means I have no interest in doing business with someone
I really recommend using something like Linode for cheap throwaway, might be a bit more expensive (i.e: 5 per month), but the surprise factor disappears..
You'd have to be doing something extremely small to make a $5 Linode more expensive. AWS might be cheaper for a static site, but as soon as you involve a database you're probably paying more.
AWS is not cheap. A lot of people seem get bitten by this.
I agree with most of the points brought there except for:
> I’ve personally got a dormant AWS account that’s charging me cents every month, and I bet you do too.
Hummm no? And if I have a charge line and I can try to chase it. It's not like that cent is nameless, it has a name and it has a way of figuring it out?
I mean, it can be not so obvious but it's not like it's totally opaque neither
20+ years ago I was offered a free 2 or 3 month subscription to compuserve. I was interested in it, as one could dial in as PPP account for internet access. So I configured my linux box to dial in and if the connection dropped, to redial.
After the 2nd month I get a huge multi hundred dollar bill. They claimed over the the 2 months I had used 2k-3k hours of time an their free months were limited to 750 hours a month of usage.
As I pointed out, that should be enough, as 24*31 = 744, so it should be impossible for me to use more than 750 hours a month.
They claimed that I must have been dialing in from multiple locations. I denied that and said they should have records of where I was calling in from. It took weeks for them to "forgive" the debt, without acknowledging that their billing was broken.
I always wondered how many other users got hurt because of this (my guess is not that many, as relatively few people were keeping their connection alive 24/7 via compuserve back in the 90s)
This is precisely the reason I have avoided AWS as a student and to this day I prefer providers that have clear billing like DO.
I don’t understand why anyone would use AWS when there is a risk of being charged stupid amounts of money if you screw something up. Are there advantages to AWS that I’m not seeing?
It's absolutely unacceptable that AWS hasn't fixed this, but in the meantime, would using a privacy.com temporary credit card with a ~$20 limit when signing up help? I'm unsure if those kinds of cards can be detected/blocked by AWS, or if they do.
I want to point out that I just tried to access my AWS Educate Account today to explore the panels and found:
"ALERT-1:
Session time behavior and instance types in your Starter Account will be changing on May 11th, 2021. After this date:
1. When your session ends, your resources will be “stopped.” You will be required to re-start your resources when you start a new session.
2. Updates will be made to available instance types. We recommend you to complete currently running work in your Starter Account by May 10th, 2021 as work using instance types that are no longer supported will be lost after that date."
Im still thinking about what can I do with 3 hours (duration of a session) of EC2 computing power...
I had an account with AWS about 6 years for a small prototype and I closed it after using it for 1 month. However for some reason I got an email last November about a change to some certificates for either S3 or Cloudfront because my account used one of those services in the last 6 months.
I don't know if this is a mistake on their part, but I haven't been charged in the last 6 years or gotten any emails before that. But it's still worrying because the account is closed and I have no way through standard support to know why they think my account was using their services
As a student trying out the AWS platform (I'm not even a devops/backend/anything related to it) I found it painful to use when compared to other providers. They keep building new products that build on existing products such as Elastic Beanstalk that assigns a LOT of resources you don't even need to host a nodejs app, etc. Even reaching the billing panel was painful. You have to go through at least 3 screens just to get to the "billing" screen. And then you have to browse inside of that to reach your bills. It's disgusting.
Another solution to this problem is the availability of disposable credit cards. It would be ideal that credit card numbers could just be created from scratch, as needed, and turned off whenever the use was done.
We had promises of this as ecommerce took off, to help avoid fraud. But it seems these don't get widespread adoption, probably because purchases are easy to get chargebacks for (in general).
A hard limit for cloud purchases is surely needed. But because the cloud providers aren't giving us this, I'm wondering if a solution like disposable cards could?
1. Does aws even accept prepaid cards? many providers don't dye to fraud/abuse concerns
2. Your payment method refusing to accept charges doesn't mean you're still not on the hook for them. Technically they can still send your bill to collections and wreck your credit report. They probably don't do this, but it's still a bad bandaid solution
1. Not sure, but I think that's the problem I'm asking about. Because prepaid or auto-generated cards are not mainstream, big vendors don't need to accept them.
2. Agreed, but at least this puts one more layer of control back to the owner. AWS doesn't have my birthdate or social security info (US), so it would be harder (not impossible) for them to "wreck your credit report". At least this approach puts an extra barrier in the equation.
I'm Co-Founder of http://vantage.sh/ and we allow you to connect your AWS account(s) and we'll monitor cloud spend on your behalf.
We send out regular cost report emails to try and help folks avoid situations like this and have some future plans around anomaly detection to try and help out in advance of things like this happening.
I strongly encourage folks to sign up and let us monitor things on your behalf. We have a free tier that likely covers a lot of personal users here.
This is the problem with usage based pricing. It is very lucrative pricing strategy because it works for lower end of market as well as enterprise users. One way to resolve this is to offer fixed tier based pricing for customers who may be more sensitive to prices and usage based pricing for enterprise clients.
For example.
Starter ($5/mo) - Use up to 100,000 API calls each month.
Growth ($30/mo) - Use up to 1,000,000 API calls each month.
Enterprise ($149/mo) - Use up to 10,000,000 API calls each month. After that, $10 for every 1 million calls.
AWS represents over 60% of the net profit of the entire Amazon empire across all its divisions and acquisitions. If they truly helped the customer to waste less money, it would have a massive impact on everyone who has stock, from sovereign funds to junior engineers. They have a vested interest in dinging your sandbox account $1/month for storing secrets you haven’t used in 2 years. I don’t see this changing anytime soon, especially when the competition is no better.
We would definitely spend more money on AWS services if our company could feel more secure in understanding which project those costs were associated with and were able to track the costs without spending multiple hours.
I don't get why AWS doesn't offer preloading credit. Surely charging your card 1 cent a month isn't worth the time.
Whereas if they allowed customers to load on an amount, even a direct debit, their card processing fees would go down and customers would just get cut off when they run out of credit.
I suppose a counter argument to this is that it's hard for AWS to keep a constant eye on how much a service is costing. In which case we're back to the same argument as spending limits.
When you realise that their whole business model is around extracting as much $$$ from users as they can, you should be entirely unsurprised by how AWS and other cloud providers behave. My cynical side is inclined to say "see, this is what you get for using cloud computing". Cloud providers are the very definition of nickel-and-diming. I still remember the amusement of finding out that AWS has Cost Management services, which themselves have a cost.
I use aws relatively often but for fairly small scale stuff, selfhosting/homelab. I'm very scared of all these horror stories.
I've set a budget and an alert for the budget. Is that enough?
I can see the budget being overdrawn for the time it takes me to react to the alert but I see no way to actually shut down servies.
It would probably be possible to script by monitoring the API and then issuing aws commands to shut things down. But it would be a huge project, someone should get on that.
Spending limits should be a legal requirement and if company cannot or doesn't want to implement it, then they shouldn't be in business. Simple as that.
This doesn't help those that have already been hurt, but when you're using any cloud provider for personal use make sure to set budgets and alerts for them. It's quite easy to set these up and they can save you a headache later on. Truth be told they should all be asking you what you expect to spend when you set up an account and alert you by default but just remember these cloud providers are not your friends.
I have had issues with students doing the same thing, and I was being charged for dormant projects that I thought I'd deleted. AWS is a nightmare of an ecosystem to navigate.
The AWS Educate Starter Account is almost useless - especially if the student has to submit their project for external review - because it doesn't allow publicly accessible S3 buckets and has rotating credentials.
> ...an updated free tier that treats “personal learning” AWS accounts differently from “new corporate” accounts, and sets hard billing limits that you can’t exceed.
Honestly, this is needed for corporate accounts as well. Not all companies are FAANG scale behemoths who can shrug off an unexpected charge.
For a scrappy startup in India, an unexpected $5000 bill would be an existential threat.
Not sure why, but they courted our company right from the beginning: got in touch with us within a month of us forming up, gave us a generous free startup credit, even invited us over to their Bangalore office and helped us with how we can optimize our infra.
Thankfully, the company reached profitability so we pay them rather hefty bills now. But I guess I agree with you in that that's not a typical startup's trajectory, and their ROI from the aforementioned outreach is likely relatively low on average so am wondering now why they went out of the way to help us.
Does anyone else use the AWS free tier for a year, and then when the free year ends, signup for a new account and start another free tier account? It's a good technique if you live frugally.
The only caveat being you have to migrate everything to new servers, which can get messy if you don't practice how to do that efficiently.
It's the AOL free trial business model. Give 'em a freebie, but take down their credit card info so you can start charging them the moment they exceed the limits of "free". And make it really hard to cancel.
As for whether Amazon will listen, Upton Sinclair blah blah not understanding it.
One hack I've learned once I got burned by similar service is to always use virtual debit card with a spending limit if your bank supports one. I have a virtual card on Revolut dedicated to small subscriptions that freezes at 100$/month to prevent this sort of fraud.
is there an IAC template somewhere that resets an aws account to 0 and deletes everything? I know this articles point is this shouldn't be possible, but as long as it is, there should be a simple measure, that could be linked in tutorials for stopping ongoing charges.
unfortunately for most cloud providers, you have to read through pricing very carefully and be even more careful with what you deploy. an infinite loop could lead to $$$$ in charges and billing alerts just alert you, they don't actually shut anything off
I am very happy this thread came up, because I'm looking at cloud providers to experiement with.
So many comments about charges that show up despite resources removed or even deleted accounts, tells me that AWS can not be trusted to present me the full picture of costs.
I had a similar experience, with costs of $100 incurred over a four-month period. All it took was on email to AWS support, and they refunded the entire thing.
I was impressed, but still have been really trying to keep off AWS just to not have a mistaken repeat.
That quote is what I think about when I see cloud hosting pricing models. Rulan was the late co-founder of Clark's Pine Factory in Northern Utah and my first boss in the 1990's.
Part of the problem is that AWS billing is soo slow to update.
This doesn't only hit budgetting... trying to get all the costs to go the proper billing tags (and verify that you found them all) also requires test cycles of up to 2 days.
Privacy.com doesn't believe anyone would want their service if they have a land-line, and not a mobile. "Give us your phone number", okay, "we sent a confirmation SMS" ... hahahahaha.
Yes you can probably dodge that by going with a SMS enabled VoIP provider.
AWS offers throw away accounts during immersion days, jam sessions, etc (especially at re:invent). It would be great if these were extended to the general public, even if at a small fee.
There’s a service like this called Qwicklabs that I have been using for GCP training. You load a time limited lab and get a new set of credentials only for that session. After the timer is up, poof everything is deleted.
Me too! I use AWS all day at work, so I should be able to find the service that charges my personal account 64 cents per month... :shrug: I'm glad that student got help.
The AWS billing dashboard has a "budgets" feature. I just added a daily account-level budget of $1 to my dev sandbox account, and setup an alert at 50% of the budget or $0.50. It took about a minute to setup.
You can choose to include or exclude refunds and credits. If you exclude them, I am guessing but it should show your true cost without the free tier pricing. If they are included it would tell you the impact right after any free tier offers expired.
Perhaps setting a budget like this should be part of any organization's new account setup process.
aws free tier is a joke anyway. surprise billing even after minimal usage and they shutdown your account if you forget to pay or the credit card billing info does not work . If it is truly free, why do they need your credit card.
The author starts eith comparing a $200 buck charge that AWS support roller back to somebody losing or imagining to lose $100K in trading. Wild juxtapositions there.
After seeing how recklessly people use AWS when it's not their own wallet on the line, I'll wager to say that this 20-year old learned a valuablr lesson.
This is something I think about a lot. For work, I run a cloud service, and being able to accomodate students (and other cost-sensitive users) like the one in the post is important to me. I've always thought that letting people experiment over the weekend is what leads to people wanting to use something at work (where the real budget is), and I think that supporting those users is how you build a userbase of advocates. Maybe they can't afford (or justify) the enterprise plan quite yet, but they can still be a happy user and cheer on those that can justify the expense.
I've personally found that it can be hard to get the approval to type in your company credit card until you've done enough research to prove that something is going to be worth the money. That leads to a chicken and the egg problem; people won't be confident until they've paid, but they can't gain that confidence until they pay! So you have to get the small "testing" use case perfect, or you'll never have any real customers. (My corollary to this is because AWS already has your company credit card, and you already have "root" or similar because of your role, it sure is easy to build whatever you want there. If AWS provides some service, you can start using it and the cost will be lost in the noise. But if it's not on AWS, then you'll have to produce some justification to use the service before you can start paying for it. I ran into that a lot at my last job; I wanted to buy $5/month services like Sentry but was told no, whereas other people could just create a m4.4xlarge RDS instance for $1000/month and nobody even noticed or cared. People really like being approvers at the time of entering the credit card, rather than at the time of actual cost accrual, and cloud providers really facilitate that. Not sure that's really helping those approvers -- it almost feels a little bit like embezzlement.)
Anyway, ranting aside, here's what we do for the cloud service I work on:
1) We have a free tier. It's really free; you don't even get to enter a credit card. Sign in and start doing your work. (We delete your stuff after 4 hours, though.)
2) For the paid tier, all costs are pay as you go. The instant you click "delete workspace", no more costs accrue. Merely having an account open doesn't accrue any costs for you, and there is no way to create a phantom resource that you can't see in the UI and delete. If you delete everything, billing is over.
One weakness that I'd like to fix is the latency between resource use and when we tell you about it. That takes a few days, so if you are playing with aggressive autoscaling, you don't really know what it's going to cost until your experiment is over. I'd like to collect real-time usage data and just bill based off of that, so that the UI can update you within seconds of your job starting. If it's too much, you can just pull the plug and not be surprised.
The next step is letting people pre-pay, and do what the vast majority of comments on this thread want: kill the compute resources when the budget is exceeded. My thought is that it's hard to ask your customers for money upfront, which is probably where the post-pay model originated. I personally always have reservations about buying 3 year reserved instances from cloud providers, even if it saves a ton of money. "What if we stop using it tomorrow!?" But there is probably a good compromise here: type into the UI what you'd like to pay for autoscaling per month, and once that budget it exceeded, run at the bare minimum "keep the lights on" level. More difficult than the alternative, but certainly possible. And very good for users -- no total outage, no unexpected bill they have no hope of being able to pay. Things are just slower for a while.
Anyway, I don't know what the perfect formula for cloud pricing is -- but it's clear that what AWS has is not quite right, and that we can probably do better. To paraphrase Jeff Bezos ("your margin is my opportunity"), what AWS has consistently done wrong for years is your chance to make it better and get paid for doing so.
Goodness - reading these statements makes me laugh - "sinister" "evil" "by design".
Reality - very few folks want a hard billing limit.
To stop charging you AWS would need to delete all your EBS and S3 volumes, stop all EC2 instances, release all public IP's, delete all AWS directories and the list goes on. The idea that AWS would build this giant data loss footgun into their system is ridiculous.
Somewhere in AWS someone said, how could this blow up, and they came up with 100 ways, including misconfigured cost accounting etc.
That said, the GCP project based model makes more sense to me, give you more control etc.
That said, if there is such demand for hard billing limit playgrounds (I'm sure there is but not by folks giving AWS a lot of money), someone should be able to do a hosted solution for AWS that bills into their corp account and gives you a playground for learning (with a real hard billing limit). That type of approach is used in a lot of other contexts already.
AWS employs a whole lot of brilliant people, and they're an insanely profitable business unit.
I'm sure those people, given the appropriate motivation and opportunity, could provide a solution (or several) that avoids foot-gun data loss at every turn.
There's all sort of solutions they could go with.
As an idea, perhaps some kind of functionality within the billing side of things:
They could, perhaps, decide to just apply a basic cap functionality - when you reach that cap, it stops/disables all chargeable resources above the free-tier and gives you 72 hours to go raise the cap or they'll delete everything except things in their AWS Backup.
AWS can eat the storage cost for the stopped things without a noticable impact on revenue.
Perhaps a more advanced workflow-type setup:
When monthly spend reaches $x across all regions:
- stop all running instances that are not below the free-tier limit
- apply deny policies for per-request charged services
- send an email alert to the billing, tech and admin contacts.
When monthly spend reaches $y across all regions:
- Delete all chargable resources not tagged with 'Foo: bar'
- send an email alert to the billing, tech and admin contacts.
One of the other things that would be great, is to be able to apply limits to AWS Accounts.
For instance, the AWS Account that we give to developers to experiment on, it would be great to forbid them from starting baremetal instances, or GPU instances, etc.
Some of it can be controlled through deny rules, but not enough.
Billing alerts are also currently nowhere near responsive enough - it can be hours or days before you get told that some resource is running up your bill.
A comprehensive dashboard of 'here is every chargeable resource running in this account, in all regions, right now' wouldn't go astray, either.
It's a good lesson in cloud computing to know how much you're spending on services.
Usually if you're a single person and you leave an expensive product running unknowingly, AWS support have been kind and refund you the money and take the loss as 'customer made an error' type transaction.
It's not like they hold the debt over your head for life.
I think the article is a bit extreme to highlight the person's reaction, it's pretty easy bait for most people to swallow. Realistically though, if you can't afford $20, how on earth did you afford a credit card to sign up for AWS in the first place?
There are people comparing this person's response with the Robinhood investor that lost money and took their life. The real issue here is a mental health problem and better awareness around that - AWS is a cheap scapegoat we can all blame to ignore the real issue at hand...
The problem is only partially "know how much you're spending on services".
There's a big problem of: There's a huge amount of AWS that even if you configure it "right" - still leaves you with potentially unlimited liability.
Take a simple example of using Cloudfront and S3 together to host a basic website. There's a good well documented way of securing this all "right".
However that doesn't protect you from "slashdotting" - perhaps you put an image on your site and it gets linked from the frontpage of Reddit.
You go from paying a few $/month, to suddenly overnight having potentially hundreds or thousands of dollars of spend because tens of millions of people have viewed your image.
AWS Billing alerts are not going to save you, because once they (eventually, maybe a day or so late) fire, and assuming you immediately see that notification, figure out what it is being accessed, and remove that file (or just turn off the bucket/cloudfront entirely) - the bills could have gone way up.
It gets worse when you're talking about other AWS services.
The whole point being that billing caps, along with billing alerts that are actually timely, are actually necessary.
You shouldn't have to depend on the grace of AWS Customer Service to not force you to decide whether to pay rent, eat, or your massive credit card bill.
> Realistically though, if you can't afford $20, how on earth did you afford a credit card to sign up for AWS in the first place?
The example was $200. And for a lot of people, particularly students and other people new to development - $200 can be a large chunk of money.
Part of the problem is that you go to AWS Conferences and talks, and they talk about how there's all this amazing stuff, and encourage people to sign up and try it out.
What they never do is tell you "Oh, by the way, this demo costs $x/hour". I'd love to see some kind of taxi-meter style box behind the presenter/in the corner of webcasts showing how much the AWS Bill would be for the demo.
Taking your cloudfront example, the free tier covers 50GB data out at 2 million requests - so your image would need to be less that 25KB to be under the free tier limit.
Taking a 4MB image, that would equate to around 8TB which is priced at under $1.
Make that 20 million views and you hit $10 - oof!
---
$20 or $200 doesn't matter - how on earth could you have a credit card if you can't afford to pay off $200?
---
AWS is great and people are free to try it out. They should also be aware that things cost money and not to put their credit card down for services without being fully aware of the costs associated.
It's like buying a property for investment then complaining to the real estate agent later down the line for a number of issues the property has that require maintenance down the line.
What's wrong with buyer beware? Why can't people/big companies offer services without people crying foul about 'getting hurt from bill shock'? Where's the personal responsibility and accountability for your own actions and decisions gone?
AWS aren't scheming to scape little bits of money from small time developers. They're interested in catching big fish companies who are looking to choose their platform / service offering in the cloud computing space. If you think these small time developers with 'bill caps so i don't overspend' is at the top of their feature priority list you've got the wrong idea about the business.
> $20 or $200 doesn't matter - how on earth could you have a credit card if you can't afford to pay off $200?
Who cares how they got the card, maybe it's a debit card, maybe it's their parent's or partners. Maybe they've got one that only allows a very low limit of charges.
The point is that there's a whole ton of people for whom $200 isn't a "Oh, well, that's a learning exercise - better be more careful next time".
> What's wrong with buyer beware?
Buyer beware is fine where all parties are fully aware of all repercussions.
However, you'll note that this doesn't apply to certain kinds of transactions, particularly financial transactions.
In some cases, you're either required to go through an expert who can advise you on the downsides and risks, or the seller is required to ensure that the buyer is fully informed of the risks.
It's the same reason we have warning signs on a whole bunch of things about how they could injure themselves.
Without the ability for someone to limit the damage, someone could be up for thousands of dollars in spend without being aware of how they even got there.
> AWS aren't scheming to scape little bits of money from small time developers
It's not even students and other inexperienced people that end up in bill-shock.
I've dealt with well-experienced senior developers who've gone off to an AWS Conference and spent a day or three being lulled by the story about how X is the new hotness, and would solve all the problems we have.
The AWS reps have all confirmed the story that X is the new hotness, and it's perfect for some new solution - and they've got some big name customer who's implemented it and saved millions Vs their legacy on-site solution. The AWS reps also hand out tens of thousands in account credits like they're mentos, with the message "Go play, see if it works for you".
Those senior developers have taken a look at the pricing on the thing, thought they understood it, and then fired up a demo to prove it out. When the billing starts to come in, there's been shock from them that they spent so much in so little time.
The pricing on things is quite opaque with one headline price for something, but doesn't include that you'll also need to consider the pricing for other bits. The billing is often delayed quite signfiicantly, and alerting is only useful as an indicator of how much you've fucked up, rather than a way to prevent it.
So, yeah, with AWS marketing all their solutions as being much lower cost, while at the same time not making it clear on what things do cost, and not making tools that you can prevent major fuckups -- they definitely do need to shoulder some blame.
Then just given people a certain grace period to reactivate or get their data out before it's removed.
It wouldn't fix production deployments where you want alarms, not a shutdown, when you hit spending caps, but it would help people on the dev stage to avoid issues like this.