AWS CloudFormation: BeanStalk with custom stacks

[justinsb]

http://aws.amazon.com/cloudformation/

[cagenut]

This is cool and all, but its part of an overall trend about aws/ec2 thats starting to get on my nerves. They're spending their time and resources working their way up the stack instead of improving the lower layers of the stack for which you use them in the first place. Right now ~2+ms inter-node latencies, arbitrary 200+ms hangups on EBS, and the general lack of an SSD option make AWS a non-starter for things that need to scale without being architecture-astronaut computer science projects. For many workloads I have to colo instead of use AWS because of the age old "spend 10K on hardware or hire an FTE developer" tradeoff.

[nethergoat → cagenut]

Designing for fault tolerance and scaling out instead of individual node durability and scaling up is, in most cases, a sounder approach. Since the vast majority of applications map well to this case, it makes sense for Amazon to focus on features that assume/assist scale-out.

It's easier to design for scale-out than one might think; a great resource to get started is http://highscalability.com.

Regarding EBS, I haven't seen the hangup issue you've described. Any data?

As for low inter-node latencies, Amazon has an offering that specifically addresses that need: http://aws.amazon.com/ec2/hpc-applications/

Overall, I think Amazon is making a lot of inroads in areas with specific hardware demands. They just launched GPU compute options, and I expect we'll even see SSDs soon.

[cagenut → nethergoat]

Scaling out vs. scaling up is the false dilemma, the answer is both at the same time solving the bottlenecks you have using the time and resources you have. The "vast majority" of applications do not map well to an n-wide map-reduce-esque design, the "vast majority" of applications bottleneck on an authoritative datastore's I/O capacity.

I'm the sysadmin (not the dba) for a website that does 30K http req/s at the edge, and our mysql cluster does about 3000 req/s during mid-day with most queries in the 1 - 5ms range.

Highscalability is a terrible place to get started, its a great place to share notes but if you get started there you will waste insane amounts of time on architecture astronaut nonsense.

[toddh → cagenut]

What do you consider architecture astronaut nonsense?

[travisglines → nethergoat]

On the spectrum of scaling an application AWS has definitely leaned hard towards scaling out. (where the other end is scaling up) This means it caters a little more to denormalized databases and load balanced web servers than it does to one giant mysql database server or a cluster of a few large database servers.

I agree with nethergoat in that if you need low latency communication between ec2 instances that you should be using their hpc instances with faster/optomized ethernet.

I however disagree with nethergoat and agree with cagenut and think that the vast-majority of applications still currently depend more on a scale up paradigm. That is slowly moving towards scale out as people/businesses see its benefits and AWS is catering to this trend.

[gfodor → cagenut]

Your claim is a false dilemma unless you can prove otherwise.

[cagenut → gfodor]

http://www.mysqlperformanceblog.com/2011/02/21/death-match-e...

tl;dr - "price is in the same order of magnitude, but performance is two to three orders of magnitude different."

[gfodor → cagenut]

Your claim was that Amazon, by working on features higher on the stack, is unable to work on improving lower performance. You need to prove that there is a lack of engineering resources at Amazon to work on both tasks at once.

[smoody]

The AWS team never ceases to amaze me. Seriously. They release new, great, useful services at an amazing rate. It's difficult to believe that other cloud services will be compete over time simply because they cannot innovate as fast (I know others will survive and that's a healthy thing, but if I were making a bet today, I'd bet on AWS in the blink of an eye).

[petervandijck → smoody]

Same here. They just keep on going and going and going.

[aristidb]

They even embedded a JSON-based programming language! From the example at https://s3.amazonaws.com/cloudformation-templates/CloudForma...:

"UserData" : { "Fn::Base64" : { "Fn::Join" : [ "\n", [ { "Fn::Join" : [ "=", [ "dbName", { "Ref" : "WordPressDBName" } ] ] }, { "Fn::Join" : [ "=", [ "dbUser", { "Ref" : "WordPressUser" } ] ] }, { "Fn::Join" : [ "=", [ "dbPassword", { "Ref" : "WordPressPwd" } ] ] }, { "Fn::Join" : [ "=", [ "dbHost", { "Fn::GetAtt" : [ "WordPressDB" , "Endpoint.Address" ] } ] ] }, { "Fn::Join" : [ "=", [ "dnsName", { "Fn::GetAtt" : [ "WordPressELB" , "DNSName" ] } ] ] } ] ] } }

Certainly not the nicest syntax, but I guess it gets the job done.

[wrs → aristidb]

Ugh. The first thing I'd do is write a preprocessor so I never have to see anything like that. Why not

   UserData: Base64("dbName=$WordPressDBName\ndbUser=$WordPressUser\n...")

I'll give Amazon the benefit of the doubt and assume they just didn't want to impose a syntax, not that they think humans should be writing JSON like that.

[nivertech]

Try to understand this monster, taken from:

https://s3.amazonaws.com/cloudformation-templates/CloudForma...

Amazon guys trying to reinvent JS in JSON:

    "WordPressLaunchConfig" : {
      "Type" : "AWS::AutoScaling::LaunchConfiguration",
      "Properties" : {
        "ImageId" : "ami-16668c7f",
        "SecurityGroups" : [ { "Ref" : "WordPressEC2Security" } ],
        "UserData" : { "Fn::Base64" : { "Fn::Join" : [ "\n", [ { "Fn::Join" : [ "=", [ "dbName", { "Ref" : "WordPressDBName" } ] ] }, { "Fn::Join" : [ "=", [ "dbUser", { "Ref" : "WordPressUser" } ] ] }, { "Fn::Join" : [ "=", [ "dbPassword", { "Ref" : "WordPressPwd" } ] ] }, { "Fn::Join" : [ "=", [ "dbHost", { "Fn::GetAtt" : [ "WordPressDB" , "Endpoint.Address" ] } ] ] }, { "Fn::Join" : [ "=", [ "dnsName", { "Fn::GetAtt" : [ "WordPressELB" , "DNSName" ] } ] ] } ] ] } },
        "InstanceType" : { "Ref" : "InstanceType" }
      }
    },

After some identation cleanup:

  { "Fn::Base64" : 
        { "Fn::Join" : 
            [ "\n", [ 
                { "Fn::Join" : [ "=", [ "dbName", { "Ref" : "WordPressDBName" } ] ] }, 
                { "Fn::Join" : [ "=", [ "dbUser", { "Ref" : "WordPressUser" } ] ] }, 
                { "Fn::Join" : [ "=", [ "dbPassword", { "Ref" : "WordPressPwd" } ] ] }, 
                { "Fn::Join" : [ "=", [ "dbHost", { "Fn::GetAtt" : [ "WordPressDB" , "Endpoint.Address" ] } ] ] }, 
                { "Fn::Join" : [ "=", [ "dnsName", { "Fn::GetAtt" : [ "WordPressELB" , "DNSName" ] } ] ] } 
            ] ] 
        } 
    }

DSL like this would be much more readable and maintainable:

    base64(<<<USERDATA
    dbName=$WordPressDBName
    dbUser=$WordPressUser
    dbPassword=$WordPressPwd
    dbHost=${GetAtt("WordPressDB","Endpoint.Address")}
    dnsName=${GetAtt("WordPressELB","DNSName")}
    USERDATA)

And this just a small example, I wander how complex this JSON will be for real-life cases. Instead of reinventing Javascrict in JSON, they should have added some simple shell-like parameter substitution DSL (simple sigils) or allow some sand-boxed sanitized Javascript.

Now I understand why tools like Chef/OpsCode using Ruby for this.

[kaerast]

Surely we're better off using and developing Fog (and similar tools) so that we're not locked into the Amazon infrastructure?

[justinsb]

Jeff Barr's post: http://aws.typepad.com/aws/2011/02/cloudformation-create-you...

[g123g]

So it was good that I didn't spend time learning chef/puppet or going with RightScale till now?

[noelwelsh → g123g]

I'm not so sure (given I spent quite some time learning Chef). With Chef I can deploy to a Vagrant VM running on my laptop, to a local server, to our VMs hosted by another company, or to EC2. This is great for testing, and it gives some useful diversity. For example, the disk IO throughput on EC2 isn't great so we're considering deploying our data stores to some dedicated servers. We can use the same deployment process to manage these and our EC2 servers. This seems like a win.

[nethergoat → g123g]

Nope, CloudFormation is for infrastructure provisioning. Configuration management is still valuable post-boot.

I've been using CloudFormation in conjunction with Chef - they're complementary tools, really.

[g123g → nethergoat]

Can you share some example of how you are using them together. I am currently using Python scripts with boto to bring up the instances and doing all kinds of configuration stuff on them. Just wanted to see if there is any better way of doing this.

[nethergoat → g123g]

Basically, you just need to pass a Chef bootstrap script in through the instance's user-data.

Knife, a Chef management and interaction tool, provides several bootstrap scripts you could use as a starting point: https://github.com/opscode/chef/tree/master/chef/lib/chef/kn...

Let's use the Ubuntu 10.04 gem bootstrap script as an example (https://github.com/opscode/chef/blob/master/chef/lib/chef/kn...). The basic steps to use this with Chef would be:

(Note that this assumes you have a Chef server or are using the Opscode Platform. There's also a server-less option called "chef-solo" that you can use)

1. Create an appropriate first-boot.json to be used by the node(s). This will probably just be a "run_list" with entries like "role[app_server]" (function) and "role[production]" (environment).

2. Replace line 42 of the bootstrap script with your first-boot.json. Since this bootstrap script is actually a template, make sure to replace all other .erb stuff ("<%= %>") as appropriate.

3. JSON-encode the bootstrap script (handy tool: http://artur.ejsmont.org/blog/content/ultimate-web-encoder-d...)

4. Build your CF template, placing the JSON-encoded script in the "user data" section of your instance/auto-scaling group(s).

5. Instantiate your CF stack

If you'd like to use Chef for the entire lifecycle of your instances, just include a Chef recipe for running chef-client via cron every, say, 5 minutes. This allows changes to be propagated automatically in a pull style.

Done right, the end result is a complete, Chef-managed environment provisioned from bare metal using just one command.

[g123g → nethergoat]

Cool, thanks a lot for sharing. Your post has given me the motivation to start learning Chef. Maybe you can blog about it somewhere to help wanna be cloud programmers like me.

[xaque208 → g123g]

You can also bootstrap using Puppet. Basically you generate a template that you can use to upload to CF and it will just run the bootstrap script from the UserData section when the instance is started. See the wiki here: http://projects.puppetlabs.com/projects/puppet/wiki/Bootstra...

[jjm → g123g]

I'm sure the service will grow with more features. One thing is for sure, Rightscale can be a bit costly. I welcome anything in the AWS management space that saves money.

[bretpiatt]

Werner's post: http://www.allthingsdistributed.com/2011/02/aws_cloudformati...

[Corrado]

We are just getting into cloud computing in a big way but the main problem has been spooling new machines up in a clean and repeatable way. Our current thought is to boot up a "raw" AMI and use Puppet to provision it.

CloudFormation might be a better way. Instead of having Puppet maintain the system, why not just update the CF template and spool up a new instance to replace the old, outdated one? I guess it just depends on how much you can stick in a template. If you can specify specific versions of packages (Ruby 1.9.2) it might just work.

[sofuture → Corrado]

I'm using Judo (https://github.com/heroku/judo) to do 'launch-to-running-services-on-devices'. Great tool for what I'm doing, though it looks like CloudFormation has a whole lot of overlap.

[Smrchy]

Wonder what this means for companies like http://jumpbox.com/ We used Jumpbox in the in the past to setup some quick examples for testing.

[thebootstrapper]

RIP Rightscale server templates.

[buro9]

This looks wonderfully simple to configure considering what happens under the hood, and the conversation on Convore is already about how long it will be before a Django template is written by the community.

[fierarul]

So this is like an AWS-specific Puppet implementation? It doesn't seem to go all the way down to installing linux packages for example which would be really nice.

[sankara]

This is big. Makes AWS a wholesome offering. Kudos to Amazon!