To me, there is a very interesting contrast to be had between this announcement and Microsoft's announcement: it feels like Microsoft is discovering the business value of being open at the same time that Amazon is living in the time warp of proprietary everything. Has Microsoft internalized that open source is (or can be) a differentiator in the cloud? Amazon is clearly still oblivious to it -- and it will be very interesting to see if this service generates fear of vendor lock-in...
Amazon is way ahead of everyone else in the cloud space. Their closest rivals (Microsoft & Google) are pretty much copying AWS. Most of Azure's services are equally proprietary and I doubt that will ever change.
This just leaves the other smaller players (DigitalOcean, Joyent, Rackspace, etc) who are mostly offering something akin to EC2 and then partnering with other vendors to offer the missing pieces on top (frankly what other choice do they have? They can't get into a race with Amazon/Google on who can build the most no. of services - they will never win that race).
i agree with this comment completely. As an aws user, they have just done a great job in the cloud space. For anyone to really over take them, they'll have to come up with something radically different because the product is that good.
I worked on and around Openstack for 18+ months. I think ignoring it is pretty safe to ignore at this point.
<rant>
There might come along something that is actually good, but Openstack isn't it. The architecture is just bad, it will never be as reliable as something like AWS. (not because of scale, purely because of lack of error handling capabilities)
Reality is these sorts of orchestration systems need to be written by specialists. The vast majority of Openstack was written by people that admit they have no clue about systems level programming. This is what hype does, it forces a whole bunch of bad programming and architecture down everyones throats.
The marketecture and hype machine have done horrible things to Openstack. Not only have vendors riddled the thing with lockin and crap code that needs to be supported, but they have pushed entire projects that should have been shot in the head. cough Ceilometer cough
There is security, performance and plain availability problems everywhere, mostly embedded deep in the architecture.
Dumb decisions like "All systems must be Python" when Python is clearly not suited to a large number of the things they want to do is painful. As is the general "not invented here" syndrome and boys club that leads to certain libraries or patterns being pushed over others, usually to the peril of the project due to the 'blessed' thing being incomplete and unproven.
</rant>
I don't intend on returning to Openstack if I can avoid it, unfortunately my skill-set does tend towards that sort of thing so we will see how I go.
OpenStack's model seems to be "clone what Amazon does, 18 months later then hand over to vendors who don't have the scale to match Amazon's prices"
It's kind of nice in theory for internal clouds.
But I'm increasingly seeing tooling targeting Docker instead of OpenStack at the service level for doing the same kinds of things OpenStack is supposed to offer (ie, the service utilities Docker to offer automatic deployment/availability/loadbalancing instead of doing it using the OpenStack APIs).
At the cloud vendor layer, the differences between vendors can be abstracted using libraries, which removes an alleged attraction of OpenStack.
Given that, I see a few vendors challenging Amazon by building unique selling points (Google has some innovative things as does Microsoft, and Digital Ocean pushes the price/performance thing).
I see Docker taking away a lot of the "cross cloud deployment" attraction that OpenStack had, and doing it better.
So what does OpenStack offer end users? (I understand it's attractions to vendors, and maybe the internal cloud use case).
When you are actually using it as a private cloud for multiple internal clients I can kind of see the point (as I mentioned in my previous comment).
If not, then what utility is OpenStack providing?
Bare metal works well. I see many people using either Docker on bare metal or maybe a VM layer as an additional security layer, but using Docker as the deployment target.
The utility it is providing is deployment/management of the underlying resources. Docker does not do that. Some of the stuff people are building on top of/around Docker will provide it, but it's not there.
But I agree with you that OpenStack is not good enough to be worth it for that with perhaps the exception of very large multi-tenant deployments (in which case, my first goal would be to start rewriting large chunks of it). And parts of it are just so horribly over-engineered it is scary, because it makes me wonder what it is they're missing to think it was necessary to make things that convoluted, and what they're missing because they've made it so convoluted.
> And parts of it are just so horribly over-engineered it is scary, because it makes me wonder what it is they're missing to think it was necessary to make things that convoluted, and what they're missing because they've made it so convoluted.
So the source code for Amazon Web Services is clearly architected, free of cruft and under-/over-engineering? I assume you've worked with AWS's source code in order to be able to make such a comparison. In my experience, the design and architecture of closed-source internal applications delivered to customers only as a front-end is usually a nightmare.
I mildly degree with your assertion that the Docker tools aren't there yet. Some are pretty usable, and additionally if you include the category of DevOps tools then they are pretty good.
I agree entirely with you on the rest of your comment.
I've still to meet someone using it. Everyone uses aws; you hear people complaining about google compute's problems; I even hear people using eucalyptus. I've never met anyone using openstack (and I've been to aws conferences).
I have to agree with this. Amazon sees the competitive landscape and their move is to attempt to create some form of vendor lock-in by providing the services that their customers have asked for or are probably hosting themselves. This is by no means a bad move and will result in holding on to a lot more customers as well as creating new revenue streams from those services. For someone like myself however, it completely turns me off the AWS path. I will use AWS as a compute resource, as well as any other cloud provider. They will fight a battle for profit while compute costs continue to come down allowing me to reap the benefits while building a cloud agnostic stack.
Of course Amazon wants vendor lock-in. Their whole business model is about being a marketplace. Do you want them to open-source their shopping cart software?
And in Amazon Web Services, the biggest thing Amazon has to fear is the OpenStack initiative producing a platform that lots of little hosts can host.
At the same time, Amazon is stock market time bomb that will either start extracting monopoly rents and draw regulatory ire, or whose stock will drop, because of the amount of revenue multiples.
That said ... this is the way it nearly always works. Microsoft invented .NET over 10 years ago, and has kept it proprietary, leveraging the dominance of Windows, extracting rents. Now that Mobile is the new hotness and Microsoft needs to catch up, it goes open source. Google released Android for the same reason.
In short ... the companies that devote a lot of resources to R&D usually like to recoup the costs, but eventually PLATFORMS get opened. Frankly, the world benefits more when this happens sooner (Netscape open sources its codebase, the Web is designed, etc.) but the engine of progress in capitalism is still speculative investment, and that needs to pay off for years before the public at large can benefit, if investors are to continue investing.
Once we switch to crowdfunding, however, this model of capitalism and risktaking may be disrupted as well.
> And in Amazon Web Services, the biggest thing Amazon has to fear is the OpenStack initiative producing a platform that lots of little hosts can host.
I highly doubt OpenStack alone can make it possible for smaller players to replicate AWS's economics of scale, or even come particularly close. From what I understand Amazon doesn't just plop their software on commodity hardware, they use highly specialized gear optimized for running specific services. And that's not to mention buying power, the ability to run on razor-thin margins and other business factors that impact pricing.
that's almost all wrong. it's mostly commodity hardware, and their margins are thin only because they're a gigantic company. smaller companies can and do offer much better pricing, they just don't have the marketing budget to convince you of it.
It's commodity in the sense that Amazon buys it from commodity vendors, but they use custom designs built to address the specific parameters of their data centers. Very few other companies can afford to employ an army of engineers to tweak every last component for optimal price-performance, nor for that matter do they have the benefit of accounting for a noticeable share of a supplier's revenue when entering the negotiations room.
> Smaller companies can and do offer much better pricing, they just don't have the marketing budget to convince you of it.
I am aware of at least one player (DigitalOcean) offering lower pricing than AWS, but that's only in a fairly limited niche. So while they may have the "economies" part checked they don't have the scale and scope to compete with Amazon on a particularly substantial level. I'd also be curious how they match up to Amazon in terms of operational efficiency since they lack many of the the advantages I listed here and in my previous post, which give AWS an edge in the race to zero as time passes.
The only reason that Amazon isn't immensely profitable right now is that their investors permit them to plow almost all of that profit into expanding their business.
How is this a case of creating vendor lock-in at all? The service itself is API compatible with MySQL... you can migrate to... MySQL, MariaDB, or a host of other services that are exposed in a MySQL compatible interface.
It is vendor lockin because it means you avoid thinking about how to deal with scaling beyond the single box.
So when the moment comes and you want to move away from AWS and you have scaled well beyond the capacity of a single box, what then? Suddenly you have to figure out sharding, replication and HA setups from scratch. While that may be the right choice for you if you're an early stage startup and it lets you defer engineering costs until a point where you're hopefully better funded, it's also dangerous:
It's far easier to get things like sharding and caching and use of read-replicas and multi-master replication right if you think about it from the beginning (if you've ever tried to retrofit any of this onto an application not designed with it in mind you know what I'm am talking about). It'd be awesome not to have to worry about this, but if you don't, you tie yourself into a platform you'll find it harder and harder to move off.
As a product, this solution would be a killer - it sounds fantastic. As an AWS-only service, I for one will treat it as if it doesn't exist, because getting tied to AWS for the long term is not an option for most of the projects I do.
You can also do sharding on mysql, or use foundation, or any number of other platforms that are mysql compatible as I said... lock-in means that you don't have the ability to migrate relatively easily. In this case, your application layer likely wouldn't need to change at all.
They've even mentioned it's pretty much a click-through to migrate from mysql... to me that seems like the opposite of lock-in.
As to only being able to use their infrastructure to scale this way, that's a value-add.. that's part of why you pay for SaaS instead of developing your own infrastructure. This is why you're renting instead of buying.
There's nothing stopping you from using a scale-out architecture with Aurora, and nothing stopping you from running MySQL on your own hardware with loads of SSDs to get the throughput you are after.
"Baseline storage performance is rapid, reliable and predictable—it scales linearly as you store more data, and allows you to burst to higher rates on occasion."
You mean to say that I can scale my single MySQL instance linearly with data storage by just throwing bigger SSDs at it? That has not been my experience, please share how you have been able to accomplish this?
Fair point, I don't know what voodoo they use, but you can purchase monstrously big servers (you'll need more than just one fit availability in any event) and storage arrays that would exceed the performance requirements of the vast majority of plausible use cases. The cost of such equipment is likely more expensive than AWS though.
The interface is MySQL, no shortage of alternative implementations... Did Microsoft open source SQL Server?? How about the source code for ANY of their cloud services? They are ALL proprietary.
Once you have scaled to beyond the performance available on a standard setup on a single box without preparing to scale out, you face increasing costs in a migration because you potentially have to re-engineer your app to be able to run on a very different stack.
Compatibility at the connection level is just one part of a whole lot of issues to take into account when considering whether or not you can migrate elsewhere easily.
Given how expensive AWS is, that's something to seriously consider.
Well if only AWS provides the scaling you need then what would you migrate off of AWS to? You could re-arch your app to scale out for some use cases as you suggest, but that's also an indicator that using AWS allowed you to get to market faster with a much simpler system that didn't require a far more complex design, many more engineering hours and more admin.
Actually, the azure SDK bits are pretty much all on Github as well.. Azure's CLI interfaces are either via node.js modules (npm install -g azure) or, iirc powershell extensions.
I will agree on a lot of the tech.. I think, by comparison DocumentDB is horribly locked-in without an API-compatible version you can self-host (even if it doesn't use the same tech, or perform as well) is a big warning flag imo.
In this case, being MySQL compatible is anything but a lock-in.
Look more closely. MS may be opening the client side, but that is a honey trap to get people to use their server side stuff. And that is as proprietary as always.
It is the same old play book as they used for Exchange and IIS.
I think the problem AWS is seeing is how they are being commoditized (e.g. you can just run your database on the cheapest hosting provider), so profits will move towards 0. They will need to add more services like this (Aurora, DynamoDB, etc.) to ensure AWS isn't a commodity (and you can't just easily switch to DigitalOcean).
I think that the AWS vendor lock-in won't come from single features such as databases from specific vendors, but rather the ecosystem of Route53, elastic load balancers, S3, EBS, AMIs, Glacier, RDS, CloudFormation templates, SQS, security groups, libraries like boto, auto scaling groups, and everything else besides instance provisioning. Of course you can implement most of these yourself, e.g., HAProxy for ELBs, rabbitmq for SQS, but all that requires application code changes and significantly more configuration management.
So far I've been able to avoid vendor lock-in on the code level (besides S3, but the S3 API is simple and has become pretty much a standard for object storage).
On the infrastructure level, the "lock-in" part is not so much the technology, all of that is relatively easy to replace. But it would take me two additional FTE's to configure and manage everything AWS offers as a service ourselves.
But that applies to a lot of things these days, AWS just happens to be a one stop shop for a growing range of commodity services.
> but all that requires application code changes and significantly more configuration management.
And not to talk about making them highly available, scalable, having to continuously monitor and so on takes time, effort and involves significant opportunity costs.
Imagine business folks telling engineers about an upcoming promotion campaign that bring in 3X more load. Outside of AWS (or cloud provided service) one has to sit and plan for scaling up all the individual components (rabbitmq, haproxy etc.,) which really becomes pain after a few iterations.
Ouch. This pricing is pretty rough for SaaS sold on the premise of cloud scalability.
At $200/month for the entry level, their lowest price is many times what the cheapest geo-replicated "SQL engine as a service" from Google or Microsoft is. I'm not sure how the performance differs, but I am guessing theirs are no slouches.
Microsoft offers "SQL Database" geo-replicated for as low as $15/mo., and it scales up from there. Not sure about performance, but it would be apples to oranges (MySQL versus SQL Server) and difficult to compare. I wonder what the TPC numbers are, but apparently the TPC organization doesn't allow publishing that yet.
Google offers "Google Cloud SQL", also geo-replicated, and their cheapest pricing is between $10 and $18 dollars a month.
I don't think this is targeted towards those who only need to store a small amount of data and have very low performance needs. If you are are already spending $500/mo on an RDS instance, then it sounds like this would be a great solution. If you are spending $10/mo on a micro instance for your database server, I don't think this is targeted towards you.
It's not SQL Server, it's SQL Database. There are many large differences between the two.
A 50GB database with 10GB of RAM usage and 2 cores costs 700USD/Mo with SQL Database. This is replicated twice within DC. Double that price if you want georedundancy. Their entry level costs less but is also unusable for most real applications
You can see notable changes in performance if your database gets fragmented, and tweaking the .my.cnf script seems like a massive distraction from my actual work. Having somebody handle these and improve performance for long queries would be wonderful.
With a database that small, performance shouldn't ever be noticeable. Are you running the DB on EBS?
1) If you are having trouble with tweaking my.cnf, give the perl script `mysqltuner` a go. It is pretty good.
2) The biggest improvement would be moving the storage to SSDs. I personally prefer Linode to DO, but both are cheap, and good.
My linode servers can do full text non-blocking backups at about 100MB per second, so depending on your outage recovery requirement, you may be able to get away with a simple cron job.
I you manage to have "long queries" with a 300MB dataset, you're doing something seriously wrong unless your definition of "long" is very different from mine. Or you're running on harware from the 90's.
Run "explain" on all your queries and review your indexes to begin with.
That dataset is so small that there should also be no need to be continuously tweaking the config file. 10 minutes looking at a performance guide, and ensuring you have enough ram to load everything into cache should be enough to get your configuration to a "good enough" shape - there's just nothing reasonable you'll be doing with a 300MB dataset that should need anything but getting the very basics of the config right to perform decently.
You can just use RDS m1.smalls then. We have databases that are 20-40GB running off smalls with relatively high throughput (more than I expected when builiding it for sure).
$200/mo minimum pricetag is extremely high for 300mb of data, or even 10GB of data.
We're looking at migrating a product of ours to Aurora and it has an operational dataset on the order of terabytes.
(inb4 nosql: Why not NoSQL? It needs transactions that don't suck (I'm looking at you Cassandra). We use C* for other large datasets, I do wish we could just use that.)
It must introduce a millisecond or two of latency b/c cross-AZ quorums aren't free. That will likely be trumped by whatever differences in implementation quality exist between mysql and aurora.
Is a relational workload that requires four-nines-plus availability like Amazon is touting really something an organization would be willing to run in the cloud? Especially since that category includes a lot of personally identifiable data that legally can't be trusted to a third party. Wonder what kind of use cases they're aiming for.
Amazon's environments meet PCI standards and there are folks with PCI Level 1 companies running their infrastructure in AWS and storing credit card data there.
Name-clashing with an Apache project really surprises me. Also, consider the fact that Aurora stems from Mesos, which supposedly Amazon (cloud; although DB people are doing this one) folks have to know.
Also, the first thing I thought of when I saw the word Aurora was the 2012 shooting in Colorado. That's sad that the word is associated with that, but also a facet of the naming consideration.
No one is saying that Aurora is built on MySQL, I'd personally like to know :) It's the protocol (drivers) compatibility what has been claimed so far.
I'd agree with you from a technical perspective it would have been better to offer Postgres compatibility, rather than MySQL. But I believe the intent is to target both a bigger market (as of today) and challenge Oracle.
There are a lot of apps with complete ecosystems that aren't going to port to PostgreSQL anytime soon. Wordpress is the poster-child for this pattern. Yes, you can bring it up on PostgreSQL, and some modules support it, but from what I've heard, it's pretty spotty and most of the popular modules do not support it.
"Amazon Aurora delivers significant increases over MySQL performance by tightly integrating the database engine with an SSD-based virtualized storage layer purpose-built for database workloads, reducing writes to the storage system, minimizing lock contention and eliminating delays created by database process threads. Our tests with SysBench show that Amazon Aurora delivers over 500,000 SELECTs/sec and 100,000 updates/sec, five times higher than MySQL running the same benchmark on the same hardware."
I'd love to see more details (active benchmarking). At the very least, what is the CPU and disk utilization during the benchmark? It'll shed some light on how this was done: a 5x improvement in cycles per query? Or better caching?
People will benchmark this themselves ASAP. If you do, try to include some basic system metrics. The output of "vmstat 10", "iostat -x 10", and some "pidstat -t 1" would be a great start. This may only be possible for the MySQL benchmark, if Aurora is only visible via an API, and the database system can't be accessed directly (?).
To get the most comparable hardware setup for the comparison, I think you would want to run both Aurora and MySQL via RDS. RDS does not give you access to the underling EC2 instance, just to metrics via Cloudwatch.
Has anyone been able to find specific pricing on this? All of the language I've seen about it has been vague. One of the things that's kept me from using RDS for toy projects is that it basically amounts to running another instance on top of what I already have. For a "real" project, that's a drop in the bucket, but for toys and prototypes, it can be a significant chunk of the cost (so I usually end up just running my own db on the same instance as the web server for those projects).
Yes but "Storage is automatically replicated across three AWS Availability Zones (AZs) for durability and high availability, with two copies of the data in each Availability Zone."
Is that included in the server cost? Or will you actually be paying ~3x the server + storage costs?
The pricing needs to be more clear. For example we run MYSQL RDS in production with two AZ enabled, we pay x2 the prices.... so if its only costs a bit more than MYSQL RDS and you get 3 AZ for the same price -- this is good. Moreover, there are costs that come with storage (database size), this thing seems to be smart about being more efficient with allocating space.
As i understood this your database storage is replicated to multiple places, but it still runs on one instance. It does not have failover to those replicated places. You'll have to launch multiple instances for that just like RDS. So cheaper, no.
"Storage consumed by your database, up to 64 TB, is billed at $0.10 a GB per month and IOs consumed by your database are billed at $0.20 per million requests"
Sounds like the replication is included in this, but Amazon really should clarify that.
Based on how Amazon's pricing tables are, it should be included.
If you go to the normal RDS pricing, there's "Single AZ" and "Multi AZ" pricing. The Multi is 2x Single because it transparently runs the 2nd instance.
http://aws.amazon.com/rds/pricing/
Right, but you can also get much smaller MySQL instances. It's not necessarily an "ouch" given the capacity, but it is in combination with the fact that there are no smaller instance types.
I'd think for most toy projects, running more than an EC2 instance or two is overkill. At least in my experiences, running the full stack on a single server works well enough for most hobby projects.
I certainly don't want to spend $7 a day playing around with RDS for fun!
Definitely. But one of the nice things about a lot of the AWS stuff is that you can build on the same infrastructure you'd use if you wanted to scale up. So if you have a toy project that ends up being more successful than you expect, it's pretty easy to just add resources. For example, DynamoDB is ultra cheap for the lowest usage levels, but simply by reserving more capacity you can scale up with absolutely no changes to your app. Makes it easy to prototype or play on the real system and not have to re-do your work later if the project becomes more than a toy.
I guess since this is MySQL-compatible it wouldn't be that hard to migrate from MySQL to Aurora, other than moving the data...
Moving data from one instance of MySQL to another is usually straightforward as a one-liner, piping a SQL dump from the old database into the new one over the network...
I'm interested in transaction support for e-commerce. Magento users can experience significant performance issues on MySQL, and the upper tier are always interested in more RDBMS performance, eg via NewSQL solutions.
So what can Aurora do for that workload? Do the support multi-table transactions and referential integrity across all 3 Availability Zones? Similarly, they mentioned Durability targets; what's their targets for Consistency (ie ACID).
This is beginning to feel like how it was when Intel was competing with AMD -- evolutionary improvements at first, which just kept on coming, increasing in size and impact, until AMD just faded out of the high-perf server market.
So, this is a Xen VPS (Amazon uses Xen as far as I remeber) with a MySQL 5.6 database, with a custom storage engine that stores data in some Dynamo-based storage. Any code could access it via MySQL protocol (bindings to libmysqlclient.so). Fine.
I appreciate your technical insight, but glueing those technologies together is far from trivial. Making it easily to manage and reliable is even harder.
I trust your best intention, but even Dropbox was dismissed on HN as user-friendly rsync.
OK, perhaps they wrote a MySQL-Dynamo (or whatever it is) wire-protocol translation engine, and do not use MySQL server. But custom storage engine looks like most natural way.
I certainly do. On cloud no less. The reality is that Amazon got to market several years before Google, so GCP had some catching up to do. That trend is changing, and this is one indication.
The subjective opinion was about "following" when Amazon has had relational database services for years, and the latest engine they say was under development for 3 years, not exactly something in response to goog.
Irrespective of that you shouldn't be posting as joe public when in fact you are a Google employee playing cheerleader for Google products on threads about your competitors products. Keep it classy.
We're not talking about a relational database, we're talking about a mysql-wire-compatible database. Google BigQuery has been around for a while too.
Adding "disclaimer: googler" to every post I make on HN seems pretty obnoxious to both me and anyone reading. I just didn't feel that "neat" + a fact qualified. Clearly opinions on that differ, and I'll probably just post less in the future.
Which (as a document store) has plenty of use cases... and even when you read the article it's mostly about how they didn't understand what they were doing going into things... It's not a compelling argument.
"But this stuff wasn’t obvious at all. The MongoDB docs tell you what it’s good at, without emphasizing what it’s not good at. That’s natural. All projects do that. But as a result, it took us about six months, a lot of user complaints, and a lot of investigation to figure out that we were using MongoDB the wrong way."
The film/actor/career example they wanted to do could have been solved pretty easily with a document that had an _id for an actors collection and a name... that's just the document paradigm... I'm a huge fan of both postgres and mongodb and I know that's not a popular opinion around HN. I just get tired of seeing clickbait headlines being cited as the criticism within is correct.
To me, there is a very interesting contrast to be had between this announcement and Microsoft's announcement: it feels like Microsoft is discovering the business value of being open at the same time that Amazon is living in the time warp of proprietary everything. Has Microsoft internalized that open source is (or can be) a differentiator in the cloud? Amazon is clearly still oblivious to it -- and it will be very interesting to see if this service generates fear of vendor lock-in...