This is a great writeup and is completely on target for realistic deployments on AWS.
We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post:
* I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens.
* We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only.
* If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs.
* RDS is still just a dream for us, since we still don't have a comfort level with performance.
* Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances.
* We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS.
* We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.
* If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.
I don't get the 100tb.com model. Checking out their website, I see:
Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15
1Gbit Dedicated Port
So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server.
250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month.
So either:
A) Their connectivity isn't hot.
B) If you actually use that 100 Terabytes per server,
they start to curtail/rate limit/traffic shape you.
C) Some other option I haven't considered? Maybe they
just rely on the vast majority of their customers not
using that bandwidth, and subsidizing (significantly)
those that do?
blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.
So - that pretty much answers my question. If you actually start to use that 100TB, they drop you as a customer. TANSSAAFL, so, if it looks to good to be true ($0.33/mbit/second on Softlayer network) - then it probably is.
I'd be very interested though, in what people have been able to sustain per server at 100TB without getting dropped.
My guess is that it's somewhere around 32 Terabytes/month per server before 100TB gets a little angsty (100 mbits/second sustained), particularly if you have more than a dozen or so servers with them (filling out a GigE sustained)
They recently shut down all the servers of a Chinese reseller with 45 servers after a lot of their servers were the target of a DDOS. It's of course understandable to disable those servers in case of a big DDOS, but the problem was that they gave the reseller 24 hours (during the weekend) to backup the data.
It's pretty much impossible to do this in the weekend without even the passwords of the customers. Imagine receiving this email when you wake up (probably some time after it was sent) and having to get new servers immediately somewhere, emailing/phoning customers for their passwords and then also having to transfer potential TB's per server in the small remaining time frame. After posting on WHT (meaning after a PR disaster), 100TB did extend the deadline.
I haven't done a ton of work with 100tb.com, just a couple servers with them a few years back. It's great bandwidth. They're a SoftLayer reseller, so the nextwork blend is pretty good. I never got close to the 100tb cap, so am not sure if they'd impose some metering if you're doing it on a recurring basis. I feel like most of the people I've talked to about 100tb.com have gone in with the mentality that it's nearly unlimited, but end up using less than 1tb. If you aggregate across a ton of resold servers, I bet it's possible to make the model work pretty well.
blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.
I suspect it is C) - the simple reality is that most Web hosting use cases don't consume bandwidth at rates that saturate an unmetered port, other than those that violate the TOS from 100TB.com (i.e. running a CDN reseller)
We've never, ever had a single issue with 100TB.com. The servers we have with them are solely in place to provide MP3 streaming capabilities, and we certainly do get our money's worth.
Current bandwidth pricing: Level3 (quite good) about $3/Mbps, Cogent (not as good), 70 cents/Mbps. So $700 to $3000 for a 1Gbit pipe depending on quality, though usually they will charge you a bit more and you will have 1Gbps on a burstable 10Gbe connection.
However - most people don't use anywhere near this amount - really, they are selling you a server connected to a gigabit switch.
As soon as over-subscription kicks in (it does immediately, since you don't start using 250Mbps the day you have your server installed) their bandwidth costs go to a fraction of what you are paying monthly.
What do you see as the benefits of using Elasticache vs your own memcached servers? It seems to me that memcached is so simple to configure and manage that paying a premium to have someone else do it isn't that great of a win.
We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers.
Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!
My ops guy has reminded me that we do, like you, have one EBS drive for snapshots of one particularly hefty DB in addition to our other backup methods.
> We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.
Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?
The idea that SNS isn't first-in, first-out is scary enough to me - but the fact that it's also "at least one copy will be delivered" (rather than "only one copy") makes it kind of insane.
Not sure I follow. Could you elaborate how SNS would require an SSH tunnel to work? I've never encountered this issue in environments I have worked in.
In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over.
RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.
You might try using the SendHub API (https://www.sendhub.com) for your SMS alerts. Way easier to get started with than Twilio's, and the groups feature makes it very easy for people to subscribe to groups of alerts.
Disclaimer: I work at SendHub, but I've done it both ways and I definitely prefer ours ;)
The elasticity, no capex and api management are great. But bending over backwards to deal with their terrible SAN, saturated layer 2 and low reliability easily wastes at least as much engineer time as it saves in ops. If the OP's 100 boxes are similar to most of the other AWS deployments I'm familiar with it's likely that'd be 10 or less actual machines to manage.
Don't get me wrong - the idea is great and it's great for really low end workloads where the gotchas don't matter or really big systems where you'd be managing against most of those issues anyway.
But in the middle of the two there is a dead space that I bet a lot of shops are stuck in - double digit instance numbers running a workload that a 2-4 servers could handle with plenty of leg room.
If you're deciding on a cloud provider in 2012 I think it makes a lot of sense to shop around. There are lots of people doing the on demand api deployment thing now with different trade offs. I like joyent a lot (local reliable io) or providers with cloud and a colo area even if it's exorbitant - as paying five hundred dollars a month for instnaces one ssd could replace sucks.
I second this. If you are in the position to commit either way I'd suggest to first run a simple benchmark with your app on EC2 versus some cheap dedicated server.
As parent says: EC2 does have its place, but for deployments in the 10-20 hosts range (or an app with "special needs") you tend to pay through the nose. Keep in mind that the amount that you pay for a fraction of a server on EC2 will rent you the entire box elsewhere.
Pretty much exactly this (I've posted this statement nearly word for word several times in response to articles like this). Basically EC2 or Rackspace Cloud, etc. are very good for small startups running a handful or two of instances, or really large companies that need hundreds/thousands of instances under which scenario hardware might very well be the more expensive option.
> The failure mode of EBS on Ubuntu is extremely severe: because EBS volumes are network drives masquerading as block devices http://joyent.com/blog/magical-block-store-when-abstractions... , they break abstractions in the Linux operating system. This has led to really terrible failure scenarios for us, where a failing EBS volume causes an entire box to lock up, leaving it inaccessible and affecting even operations that don’t have any direct requirement of disk activity.
Assuming that I/O is reliable and fast-ish was a bad OS design decision in the 1970s, as the Unix guys themelves soon realised, though it was an understandable mistake at the time. Continuing to develop and use OSes with an I/O interface that assumes I/O is reliable and fast-ish, in 2012 - it's pretty bad, isn't it?
Hardware and software have co-evolved, so that disks provide an illusion of error free operation until they throw in the towel and die. And they have consistent performance. This has worked OK so far.
With network filesystems (eg. NFS) you can choose to return an I/O error to the application when you hit a timeout or a network error (the -o intr mount option). This is rarely used since applications aren't used to dealing with them. So can't really blame the OS here either.
On one hand you're right: when block devices fail, they're pretty much gone--if you've ever tried to read from a bad sector you'll know this exactly, or worse still on SSDs, where the disk will fail to show up to the bus.
That said, I'm not sure I agree with the idea that we got away with a lack of error handling because disks had consistent performance. Magnetic disks have always had incredibly inconsistent random IO performance, and even inconsistent performance between different parts of the platter(s). And in the spirit of co-evolution, we engineered around it: OS disk caches are critical to decent performance on HDDs.
I think it's not that we found disks to be consistent, as much as our solution to their suboptimal behaviors was caching, rather than error reporting/timeouts. I believe this is because caching was the most transparent approach; a good cache makes a variable-speed disk look just like an ideal disk, so an application can be written assuming the disk is perfect.
Interestingly, as distributed systems have evolved, we've ended up having to engineer the error-handling constructs that might have been used for block devices; we see them in network filesystems (as mentioned) as well as most other network services. Applications have been designed to deal with errors. We just haven't propagated those constructs down to the disk devices in the recent past.
> Magnetic disks have always had incredibly inconsistent random IO performance
From a non-realtime app POV disk seek performance is pretty consistent: you get 5-25ms seek cies that center around 10ms.
Especially In contrast to network backed where you get to contend with hiccups and contention with other users.
OS disk caching came about for a different reason.
We here at PipelineDeals also abandoned EBS-backed instances after their 2nd outage.
Instead we rely on instances that use an instance-store root device. During the EBS outage, our instance store servers did not have any issues, while our EBS-backed servers really struggled throughout the day, with crazy high loads.
We still use EBS for database backups via ec2-consistent-snapshot. The slave DB that performs this is not production-facing.
We have 2 separate chef recipes for our DBs:
One is for a full-time slave. This recipe will set up the db to use the EBS volume.
The other is for a slave that will be promoted to a master. In this case, we do a little extra legwork to do a bit-by-bit copy of a recent EBS snapshot, onto the ephemeral disk.
Great post! It's fascinating to see that other people have come up with the exact some solution -- we also run a "skeleton crew" server setup in us-west.
Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS?
Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn't hire someone.
Frankly, if your bills are greater than $3K per month with AWS I question whether you are truly saving anything.
(I figured that midsized instances vs. dedicated servers are about 5:1 in terms of performance)
You can find essays from AWS users who swear it would be prohibitive--financially and mentally--to ever switch.
And you can find essays of people who run their own rack who swear the same thing about moving to AWS.
I worked at an all-AWS bay area startup that ran a $100k/mo Amazon bill, and now I work at a bay area startup that runs all its own hardware.
It's about tradeoffs. It's not just about cost. (Though there are many cases where Amazon is cheaper and it has little to do with your monthly spend, and certainly not an arbitrary threshold like $3k/mo.)
Writing this at the close of my employer's Open Enrollment period, I'd say this: Comparing AWS with running your own hardware is like trying to compare two health insurance plans. Each brings its own seemingly impossible tradeoffs. All you can do is bring your experience and judgment to identify core issues, make the best call you can, and suppress the urge to dream of the road not taken when you're knee deep in whatever hell you're experiencing that would be a non-issue if only you had went the other way.
Though if you loathe the AWS "fanboyism" so much then maybe it's the company you're keeping: Few currencies in a startup are as valuable as flexibility. The younger the startup, the truer that is.
If I knew how to setup OpenVZ and create < 10ms latency VPNs across multiple datacenters, then I would be a lot better at ops than I am. While I'm much better at ops now than I was when I co-founded awe.sm, I am still primarily a developer.
Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month.
If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger, I think.
I am not trying to be sarcastic, I am just tired of some of the AWS fanboi's (not you) who act as if AWS solves all of your problems immediately for $50 a month.
There are real costs no matter which way you go.
PS, would be very surprised if you had even 45ms latency between AWS-east in Virginia and any of their facilities on the west coast...
Much of the attraction that I see from people comes from the flexibility. Once you get to any significant size in an organization, it's not unusual for people to have to wait for _months_ to get servers rolled out.
The ability to whip out your credit card, and fire up new servers at AWS in minutes, becomes very attractive in that scenario.
I wasn't taking you as sarcastic, nor was I attempting snark in my reply: there are real costs either way, and it's our current judgement that learning how to do a virtualized, distributed stack would be more expensive than getting AWS to do it for us. (But, if I'm being honest, we haven't done any math)
And the < 10ms latency I'm talking about is between zones within us-east; latency to the west coast is a lot worse, but we only have emergency failover capacity in west.
Thank you so much for sharing this. It's about time people started pointing out that EBS is a disaster. If we all do, maybe Amazon will finally fix it.
Nice read, but I wish they had included the location (and AZs) that are in use. I've used Oregon, California and Virginia with different results.
The comment around Ubuntu is interesting and I wish there was more detail there.
We use mdadm to run RAID across multiple EBSes. mdadm is great, but has a kink that it will boot to a recovery console if there the volume is "degraded" (i.e. any failure). This is even if the volume is still completely viable due to redundancy. This is obviously very bad, as you've got no way of accessing the console. It's an unfortunate way to completely hose an instance.
It's an easy one to miss, as you rarely test a boot process with a degraded volume. When it happens though - hurts a lot.
(If you'd like to check on this, make sure you have "BOOT_DEGRADED=yes" in /etc/initramfs-tools/conf.d/mdadm).
We are primarily in us-east-1, as I mentioned in the post, with a skeleton set of DB slaves sitting in us-west as an emergency recovery if all of east-1 goes down.
In terms of AZs, were are distributed roughly evenly across all AZs in east-1.
> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime.
So what do you use now for your persistent storage? This might be the most interesting part.
All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).
This is a good approach if you can get enough redundancy with your instances and if instance failure is uncorrelated.
However, what do you do if there's a massive outage that affects most or all of your instances simultaneously? Diversification that fails when you need it the most isn't very helpful.
As I mention in the post, we go across AWS availability zones: these systems have physically separate power and networking. When they fail together, it's because they are connected at the software level -- usually EBS. If you're not using EBS, then Amazon's multi-AZ pattern is sufficient: just make sure your instances are spread across AZs.
Of course, east-1 has fallen off the map entirely on at least one occasion -- for that reason we keep a "seed" set of warm databases in us-west; if east-1 were to have an extended outage we have a disaster recovery plan that involves transferring DNS to IPs in west and spinning up new app instances there.
How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about?
Even the smallest instances have 160GB of storage, all the way up to 3TB on the cc2.8xlarge. New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage. (just tested it last week)
New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage.
Just a tongue-in-cheek comment: Those instances also cost over $2000 USD per month. For the same money you could buy a new Dell Server with 2-4 SSDs, every month. ;)
Each EC2 instance type has a different amount of ephemeral storage, from 160 GB to 3.3 TB. You need to arrange for it to be mounted when you launch the instance.
Excellent post, and it reflects my experience with AWS, too. I really like the concept, and love S3 (and Glacier), EBS is just too unreliable for us to rely on for our entire business. Nice read, thanks!
We use Percona's XtraDB streaming backup to take backups of our smaller databases, and incremental backups of the larger DBs. We store them in a series of places: on a backup instance within east-1, on a second dedicated backup instance in west-1 (in case east-1 ever bites the dust completely, such as during hurricane Sandy), and then long-term archival on S3. S3 is good for smaller databases but for our biggest ones it takes multiple hours to download a full archive, hence the backups in multiple places.
However, our primary strategy for uptime is redundancy -- every db has at least one slave, and we are spread across multiple availability zones.
With postgresql, you can do streaming replication to a few machines. I have one large machine on standby, and then a couple really small ones, some on a different provider that just receive the updated db data using pg_receivexlog.
http://alestic.com/ is the best thing I've come across (Ubuntu focused, but many general solutions as well).
FWIW, we (at DuckDuckGo) ended up in much the same place: ditched EBS, avoid anything that relies on EBS, and multi-zone and multi-region redundancy (also for latency purposes). For ephemeral storage purposes, we end up mainly using xlarge machines since they have the greatest stability and speed (with 4 drives in RAID-0).
As a bit of background: a lot of the performance pain around EBS is the inconsistency in performance between IOs. Many times you'll get a nice fast op, but sometimes IOs will get slow suddenly, or even stuck. You can imagine that internally, this is EBS seeking on magnetic disks or getting blocked by a clogged network. Regardless, inconsistent performance is nearly as bad (or perhaps worse) than consistently bad performance in certain cases:
Thus, most of the time, performance issues with EBS bite you when you have application data on an EBS volume. If you're constantly hitting the volume to serve data to customers, you'll be seeing the hiccups in EBS performance and passing them right on to customers. Clearly this is a suboptimal experience, and can lead to other failure modes; a slow/stuck bunch of ops can get an application to completely stall.
It's important to note here that EBS has no error reporting system or timeouts; common operating systems aren't very good at handling disk IO errors, so EBS will never produce them, even when its having issues. This lack of transparency can make handling/working around stuck EBS operations nigh impossible.
So, having experienced these pains, people generally say "Don't constantly use EBS if you care about your application performance/stability," and they aren't wrong.
That said, if you use EBS as a boot volume only, you are likely not hitting that volume enough to feel the pain of slow-IOs; once booted you won't touch it much. At the same time, using EBS as a boot volume has all kinds of conveniences: the ability to start and stop the instance without loosing data, snapshots and AMI creation, persistent machine-specific configuration, etc.
I hope that's useful. EBS is not a perfect service, but it definitely has its uses. EBS as a boot volume has made using AWS quite a bit friendlier.
Note: I worked on the EBS team at AWS a few years ago. I currently use EBS in my startup's infrastructure. If you have further questions, feel free to reach out to me.
As you say, the real trouble with EBS is when serving application data. We run a lot of databases, and EBS was just not good enough for them even when there were no system-wide issues.
Our problem with EBS as a boot device was that occasional system logs, etc. would go it, and when EBS was down the inability to reach the disk (the block-store abstraction problem) would lock up the whole OS.
I've run a few projects on AWS and agree with how much it simplifies your life, but like the OP the big sticking point is EBS. I wasn't aware that ELB relies on EBS. Good to know!
I've also looked into ephemeral storage, but ultimately I decided to just rent a dedicated machine from elsewhere. Building a B2B site, I'm not as worried about massively scaling on a dime. The project has still used transient EC2 instances for a few odd things, though. It's nice to have that option when you need it!
Here is actually a real issue with many of these startup type companies using EC2 or similar services... one of the very benefits that are brought up (not needing to have someone building and maintaining servers) is also the person whom generally would be taking care of those services (or abstracting them away with CM solutions).
Many times what I've seen is simply that position is simply 'filled' by developers whom have ideas about designing and securing scalable systems, but when it comes right down to it... they get hacked within some period of time because it turns out, they don't know these things well enough to take the place of someone whom dedicates their time to it.
> I assume it runs a standard linux distro, therefor patches , firewalls, dependencies etc are still an issue surely?
AWS's security groups pretty well cover you on the firewalls front (although some folks like to still run software ones on their instances to be doubly sure), but you're otherwise correct.
For a single instance, you're not going to see significant differences. It's running large clusters of instances you're going to see things being better than the average VPS provider, with stuff like Virtual Private Cloud etc.
I mean assuming that you want to open SSH you are going to need to secure that either using keys or fail2ban. If you have a web server you might need mod_security (or does AWS reverse proxy everything?). If you want remote database access you need to configure that too.
Does AWS provide easier tools for that stuff? Because you can't really cover it with a general firewall.
The only things exposed on our VPC are an Amazon ELB and a well-secured SSH bastion host (keys only, EC2 instances have password login disabled by default) for getting into the private cloud.
No reason you can't run stuff like mod_security, but that's not strictly a firewall, just Apache setup.
from reading the "bad" parts in the article, man, what a pain in the butt. I guess i'm glad to be on Google app engine and not have to worry about this stuff.
We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post:
* I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens.
* We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only.
* If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs.
* RDS is still just a dream for us, since we still don't have a comfort level with performance.
* Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances.
* We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS.
* We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.
* If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.