This is a great writeup and is completely on target for realistic deployments on AWS.
We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post:
* I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens.
* We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only.
* If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs.
* RDS is still just a dream for us, since we still don't have a comfort level with performance.
* Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances.
* We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS.
* We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.
* If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.
I don't get the 100tb.com model. Checking out their website, I see:
Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15
1Gbit Dedicated Port
So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server.
250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month.
So either:
A) Their connectivity isn't hot.
B) If you actually use that 100 Terabytes per server,
they start to curtail/rate limit/traffic shape you.
C) Some other option I haven't considered? Maybe they
just rely on the vast majority of their customers not
using that bandwidth, and subsidizing (significantly)
those that do?
blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.
So - that pretty much answers my question. If you actually start to use that 100TB, they drop you as a customer. TANSSAAFL, so, if it looks to good to be true ($0.33/mbit/second on Softlayer network) - then it probably is.
I'd be very interested though, in what people have been able to sustain per server at 100TB without getting dropped.
My guess is that it's somewhere around 32 Terabytes/month per server before 100TB gets a little angsty (100 mbits/second sustained), particularly if you have more than a dozen or so servers with them (filling out a GigE sustained)
They recently shut down all the servers of a Chinese reseller with 45 servers after a lot of their servers were the target of a DDOS. It's of course understandable to disable those servers in case of a big DDOS, but the problem was that they gave the reseller 24 hours (during the weekend) to backup the data.
It's pretty much impossible to do this in the weekend without even the passwords of the customers. Imagine receiving this email when you wake up (probably some time after it was sent) and having to get new servers immediately somewhere, emailing/phoning customers for their passwords and then also having to transfer potential TB's per server in the small remaining time frame. After posting on WHT (meaning after a PR disaster), 100TB did extend the deadline.
I haven't done a ton of work with 100tb.com, just a couple servers with them a few years back. It's great bandwidth. They're a SoftLayer reseller, so the nextwork blend is pretty good. I never got close to the 100tb cap, so am not sure if they'd impose some metering if you're doing it on a recurring basis. I feel like most of the people I've talked to about 100tb.com have gone in with the mentality that it's nearly unlimited, but end up using less than 1tb. If you aggregate across a ton of resold servers, I bet it's possible to make the model work pretty well.
blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.
I suspect it is C) - the simple reality is that most Web hosting use cases don't consume bandwidth at rates that saturate an unmetered port, other than those that violate the TOS from 100TB.com (i.e. running a CDN reseller)
We've never, ever had a single issue with 100TB.com. The servers we have with them are solely in place to provide MP3 streaming capabilities, and we certainly do get our money's worth.
Current bandwidth pricing: Level3 (quite good) about $3/Mbps, Cogent (not as good), 70 cents/Mbps. So $700 to $3000 for a 1Gbit pipe depending on quality, though usually they will charge you a bit more and you will have 1Gbps on a burstable 10Gbe connection.
However - most people don't use anywhere near this amount - really, they are selling you a server connected to a gigabit switch.
As soon as over-subscription kicks in (it does immediately, since you don't start using 250Mbps the day you have your server installed) their bandwidth costs go to a fraction of what you are paying monthly.
What do you see as the benefits of using Elasticache vs your own memcached servers? It seems to me that memcached is so simple to configure and manage that paying a premium to have someone else do it isn't that great of a win.
We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers.
Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!
My ops guy has reminded me that we do, like you, have one EBS drive for snapshots of one particularly hefty DB in addition to our other backup methods.
> We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.
Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?
The idea that SNS isn't first-in, first-out is scary enough to me - but the fact that it's also "at least one copy will be delivered" (rather than "only one copy") makes it kind of insane.
Not sure I follow. Could you elaborate how SNS would require an SSH tunnel to work? I've never encountered this issue in environments I have worked in.
In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over.
RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.
You might try using the SendHub API (https://www.sendhub.com) for your SMS alerts. Way easier to get started with than Twilio's, and the groups feature makes it very easy for people to subscribe to groups of alerts.
Disclaimer: I work at SendHub, but I've done it both ways and I definitely prefer ours ;)
We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post:
* I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens.
* We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only.
* If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs.
* RDS is still just a dream for us, since we still don't have a comfort level with performance.
* Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances.
* We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS.
* We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.
* If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.