That sound you hear is thousands of Ops Engineers' future prospects falling further and further into question.
It's becoming hard for me to justify staying in the 'DevOps' space with amazing solutions like this coming out regularly from AWS. I can hitch myself to the AWS wagon for a while, but eventually, it seems, dedicated operators just won't be needed for most small to medium deployments. It's tough, because I can see that this is genuinely moving the industry forward, but it's also negating the need for skills I've worked for over a decade to build. I guess this is what carriage builders felt like 100 years ago.
I doubt that. I'm somewhat familiar with AWS offerings and have used S3 for years. Actually creating a scalable SAAS offering correctly (emphasis on "correctly") is still a daunting task.
Sure, I just created a CodeStar project which I guess hooked me into 5 or more of AWS services but now what? What if I want to create a DynamoDB or move off of CodeCommit? How do I know if it scales or is redundant? How much does this beast I just created in a few clicks cost / month - can it be cheaper? etc. etc.
It may make DevOps jobs easier or help people get into DevOps but it certainly doesn't replace a trained AWS architect.
Genuine question: For how many small and medium-sized businesses does any of this stuff matter? Not that I necessarily disagree with you, but it seems like having one engineer who knows AWS decently well is Good Enough™ for most companies, whereas in the pre-AWS world, you needed ops people who could build most of that stuff from scratch. It just doesn't feel like you need nearly the same amount of people to scale something up with AWS as opposed to doing it in-house. But I could be wrong as I don't follow the devops world closely.
I'd look at it almost like Wix and other WYSIWYG editors: it's perfectly fine for smaller stuff. Small to medium doesn't really need to flesh this stuff out. I could use the hell out of CodeStar currently. I'm doing multiple small projects and this shop is doing the "ops people" thing, so there's a ton of friction between code, test; deploy.
But on my last project (much bigger) Elastic Beanstalk started to really show its rough edges, and I was pretty close to tearing it down and setting up by hand, which gets in to becoming the guy who knows AWS Good Enough™.
But I could've easily seen that project scaling way up and being tempted to go the Kubernetes route, which would've required a true DevOps person to maintain.
Just based on my anecdotal experience, I don't think either of the previous projects were paying for the DevOps person in the first place, and I don't think CodeStar is going to replace them now either. This is just low-hanging fruit.
Not sure you'll see this and I'm kind of stretching my memory here but...
It was a node.js project and the db was separate, and keep in mind my knowledge of AWS is/was limited. But then, this seems to be the target audience for EB, yes? I'm sure a lot of this could be fixed but...
1) Logging
It somehow took me significant time to figure out how I could send meaningful logs anywhere but poorly organized flat text files in S3 without adding some kind of logging sink to the code itself. Sure, I could've just plugged in Winston and spat out everything to MongoDB but the team couldn't understand why we couldn't just get logs out of EB. And yeah, we could could grep through the S3 bucket but that meant syncing the whole bucket and trawling through an absolute ton of log files with no more precision than your knowledge of grep and regex. Not the best thing.
I eventually dumped all the logs in to CloudWatch, but the documentation to do so was severely lacking (from my perspective) that dealing with CloudFormation template files made me consider ditching EB entirely -- If I'm dealing with this garbage anyway why don't I just go whole hog and get more control anyway?
Even after dumping the logs to CloudWatch, I was still "the guy" for doing any good searching of these logs. Maybe this was just a team quirk, but what they really wanted was logs in MongoDB or something they know, rather than something AWS specific. Again, a logging sink would've fixed this, but there were concerns about uncaught exceptions and attempting to send to MongoDB in an unknown state.
2) Deployment Time & "Health Checks"
The team was used to pulling the latest changes manually and restarting the node.js process. Introducing CI ironically increased friction as our deploys were taking an upwards of 10 minutes (20 if immutable). This doesn't sound like much of a problem in a traditional environment, but going from a "cowboy coder" environment to "why the hell does it take 20 minutes to deploy!?" was a stretch. My initial answer was "health checks!" but then we discovered that the health checks weren't actually doing much for us... I had erroneously assumed that part of the health check meant monitoring the responses of the web service and aborting deployment upon massive failure. We had a deploy that crashed and returned 500 on every request, yet somehow didn't abort. Part of this may have been because the developer involved in the bug did everything possible to prevent a "crash", instead of letting the process die upon unrecoverable error (part of a frontend JavaScrtip mindset I assume), but trying to sort out how to make the health check look for 100% 500's brought me even further in to "why not set it up myself?"
3) Deployment Failure
We hit a hard limit of deployments and had to clean out a list of artifacts we didn't even know existed before being allowed to deploy again. WTF? This is when the team started to really question the value EB was providing.
4) I ended up "the guy" anyway, so what does it matter?
The documentation was too dense for my team, and the quirks were plentiful enough that I ended up "the guy" anyway. One day I came in late and a deployment had gone wrong. EB had been rolling out bad code for 20 minutes and they didn't know how to fix it. I had to clean up the mess. That was really the "fuckit" moment for me. Hand managed EC2 instances wouldn't have caused this problem. Regardless of knowledge problems within the team itself, the whole point of adopting EB was to avoid needing infrastructure specific knowledge to scale & deploy. Now EB was doing more harm than good.
---
These are just the things I remember off-hand. I'm sure there's solutions to these problems, but it just wasn't worth it for me to investigate when really my purpose was just to write and ship code, not futz with the infrastructure. We would've run in to less problems with a naive, manual setup in the end.
You can ping me at rtannerf dot com if you want to know more. I'm sure I can dredge up a few more annoyances and clarification from an old coworker. Hope that helps.
For the small businesses that I know of, one engineer who knows the operational side decently well has been the norm for a long time. In the '90s it was one of the programmers who happened to know enough Solaris to double as the Unix Sysadmin. Maybe one genuine full-time sysadmin if the company grew a little larger. Now you have someone who knows enough AWS to fill that role, which doesn't seem all that different in staffing levels.
It depends on what do you call small or medium-sized business. Besides the same question can be asked about "developers". How many mediocre / junior / just developers you need "in-sourced" ?
More and more work can either be outsourced to services like AWS is building or cheaper IT professionals anywhere in the world.
The only question here is when taking the lead outweighs the costs.
In my niche commercial industry almost all software is server-side software. Only now is everyone moving to the cloud - mostly as a way to collect data and generate some recurring revenue. So small businesses with a few devs now need to either spin someone up on AWS or hire someone.
I could definitely see less ops engineers needed in some companies that move to AWS from some in-house platform but overall "devops" is rapidly increasing in demand.
Agree with watty here that this, as with most new Legos from AWS, makes some of the repetitive and frequent devops things easier and consistent. But the devops-person's true value is in stringing all the Legos into complex creations. And sure, while each new AWS offering, simplifies or automates some segment of work, it also feeds the number of cross-tool permutations. And that in turn creates new opportunities for the people like us that can string it all together when needed.
Operators are probably done-for, sure, with the exception of some deep-magic systems. But DevOps? Nah. It's just that "operation" is a subset of development now (as in many ways it was before I was born, when "sysadmin" was a dirty word for "Perl hacker").
Much of my value as a consultant is that I can drop into a shop, understand the tooling and the stack, and develop a solution that makes sense, using AWS tools where appropriate or general-purpose ones where the AWS ones don't fly, and I can make it approachable for developers (not operators, but developers) with less experience and make it easy for them to hammer on it and expand it as needed.
That's not going away--if anything, the steadfast drive towards understanding less of how anything you do works makes people who do understand it more valuable when the shit hits the fan. And it always does.
I completely agree with your point. I've been investigating options for moving a SaaS application away from Heroku, over to a self-hosted PaaS stack (e.g. Kubernetes-based Deis). While I think I have a modest to solid understanding of Docker, Kubernetes, and so on, shit will hit the fan and I'll be the one to frantically dig through docs, logs and other stuff once it happens. I've toyed around with the modern DevOps stacks(tm), and while the out-of-the-box experience has mostly been really great, I'm certain the dollars we'd save are not worth the sweat. Once it's out of the box, it will break, it's only a matter of time. Someone will have to fix it.
I should mention it's a very small team and the option to split the DevOps work does not exist. I'm still evaluating GCE and friends, because the amount we pay for Heroku is ridiculous.
Edit: Also I feel like a need an AWS <-> English dictionary
Ehhh. This is an unpopular viewpoint, but to be frank: I've rolled up k8s at scale, and to be honest? I think you might as well just use ECS if you're going to already be into AWS. And it plugs into pretty much everything AWS does without having to either reinvent every wheel or having to build translation layers (or use ones you don't fully understand). Failing that, use EC2 and run singleton Docker containers on each instance (easy enough to do with a short cloud-init script). You already have resource segmentation. You shouldn't have enough headroom in your cluster as-is to not be using all of it (otherwise it's just deadweight loss).
The set of shops that actually benefit from k8s or the like are a rounding error and so many person-hours are burned building stuff like this that it hurts. Simplicity is better than complexity until your needs require complexity.
If you'd like to chat further, feel free to reach out - email's in my profile.
I admit, it wasn't the easiest thing to setup. But its nice. When I commit my changes to my mainline, it deploys the code to the host and configures everything for me, and keeps all my hosts up to date.
Coworker uses it all to kick off lambda functions.
CodeCommit is just a basic Git setup, so nothing special. But working with CodeDeploy and Pipelines is ok. I wouldn't say its much better or worst than other products, and you can use them to deploy to non-aws environments also.
I don't know, I can't really sell it well. It does what it does, and it works well enough for me.
I use them too, and it works well. However, I do miss the days we used Heroku to handle those deploys _in seconds_ rather than 10-15 minutes AWS uses to roll out an EBS update.
Eh. The further AWS moves up the stack, the less dominating their products are.
Is this because the further they move up the stack, the more UX matters? That's Amazon's perennial weakness.
Remember that Bezos says, "Your margins are our opportunity." They will get around to gobbling your market, unless it requires something that doesn't scale for them.
The more tooling you get around the higher level products, the more whole industries you will capture, at least in part. So many startups are just services adding a bit in the middle of AWS and clients, value that is slowly chipped away by Amazon. Instead of the "full team" you now have an automated lightweight SaaS.
AWS's strategy is not looking for the one true workflow, its to envelop enough common usecases that it dominates nearly the whole field, giving you as custom a service as you want.
This does seem to be the case. CodeCommit, CodeBuild, CodePipeline and CodeDeploy feel like (and most likely were) developed by entirely different teams, and the actual experience of using any of them is not as good as your average 3rd party provider that fully focuses on just one of the pieces. The main thing the AWS equivalents have going for them at this time is the ultra-low pricing (by selling the underlying AWS services, and mostly not charging for the Code* tools).
AWS needs these to prevent other players encroach their ecosystem. The higher the upper-stack offerings in the tech stack, the lower the quality is requires to be dominant. But the ecosystem will secure the customers.
It's not that you cannot beat AWS in one or two offerings, but you cannot beat them as an ecosystem.
CodeCommit is about 10% where it needs to be in regards to other competitor features. They only support a single archaic code review tool (ReviewBoard). Once they manage a better UI and add additional features around their hosted git, should be a better sell.
As is, I wouldn't use it without an extreme organizational requirement.
Come on... do you mean like how all sysadmins lost their jobs when PaaS became mainstream? Operations isn't going away, it's becoming higher-level. Demand is only increasing.
Tools like this will just force Ops engineers to further up-skill. I live by a simple motto that anything which can be automated will be automated. That includes setting up most generic toolchains.
Ops engineers have two choices: adopt a symbiotic relationship with the tooling, where they can be even more productive than before while wielding the tools, or eventually lose their jobs.
AWS is the Salesforce of computing. Salesforce "consultants" routinely charge $10K+ for setting up an account and configuring a few triggers.
We will not loose your job, it might just look a little different. Perhaps instead of writing code, we will be pulling levers and turning knobs, but someone will have to have an in-depth knowledge of a the arbitrary limitations and the magic lever combinations. There is enough chaos in this space for everyone to make money...
CodeStar (and its competitors) doesn't eliminate the future prospects of ops engineers. Instead, they reduce the grunge necessary to get something working. (After all, how much fun is it to actually set up and configure Jenkins ... for the hundredth time.)
So it's possible that over time, ops engineers get more time to worry about things that are harder (or impossible) to automate. For example, organizations run similar, but not exactly the same dev workflows. So understanding your existing dev workflow, and how to tune it is an ongoing thing. And with something like microservices, maybe you have dev workflows that vary by team. Or how do you tune your monitoring infrastructure to let you root cause faster, instead of just taking stuff out of the box. Etc.
I could be wrong, of course. Curious what others think.
I agree, however, shifting from editing /etc config to "how to build this on aws etc", actually never really "shifted" me from not editing /etc config.
Could be argued that it has shifted to editing an attribute/variable/template/whatever in your config management tool rather than directly editing /etc config.
Exactly. Nobody wants to use JIRA. At one of my employers we had a dedicated employee to "manage" our JIRA install. They also managed a team of consultants that took care of developing custom features and maintenance of the JIRA server(s), which went down often.
Those future prospects have looked different for a while, though obviously prophecy is infamously not a strength for us time-bound creatures. Change is inevitable, and especially so in the accelerated, automated world of IT. There's going to be a long tail of people unwilling to make changes (and that tail may never reach zero), but Ops as it used to exist is going to have to change, just as it did when VM usage supplanted bare metal, or when we realized that mainframes couldn't offer the performance necessary for today's scale.
Really, if you're not okay with your job description evolving substantially every 5-10 years, then a career IT is going to be a source of a lot of stress for you.
Oh, you're preaching to the choir, friend. I very intentionally try to stay with modern trends, and evangelize to my fellow Ops people to do the same in order to continue providing value. The job description changes almost year to year anymore, and I'm fine with that. The source of my concern, which is maybe the wrong word, is that tooling is becoming so great that it's starting to look like I may not be needed on the bleeding edge at all!
I love this stuff. I deeply enjoy watching technology develop and become more efficient. This feeling is something akin to watching your kid drive off to college: I'm probably not needed as much anymore, but I'm pretty excited to see how far it has all come. That doesn't mean there isn't a selfish pang of nostalgia for how things once were (and a little bit of existential fear for my future financial wellbeing).
Yeah, I didn't necessarily mean you you; just general advice to anybody in IT. :-)
> it's starting to look like I may not be needed on the bleeding edge at all.
If it's any consolation I'm pretty sure this has been a theme of worry for hundreds of years, if not since the beginning of technology. Thankfully humans are extraordinarily adaptable (and bad at future prediction) so a combination of evolving our skills and failed promises means there's probably going to be a lot of work for us to do for a very long time.
> I'm probably not needed as much anymore.
I think this too is just part of successfully doing your job. And getting old. Hopefully you get to move on to new things as you hand off or automate the old. And then someday you realize that the twenty/thirty-somethings are really sharp and maybe woodworking ain't so bad after all. :-p
Kudos for honestly acknowledging your insecurities, and not angrily lashing out at Amazon. If you really feel that DevOps is a dead-end career choice, I hope you consider a lateral move to a different ladder, such as the SW-Dev ladder. As someone who made a similar transition a few years ago, I know how scary and difficult it must seem, and how much it hurts to go from a senior role to a junior role. But speaking as someone who's now a few years into my new role, I cannot recommend it highly enough. The long term benefits easily outweigh any short term switching costs.
I just migrated from IT to DevOps and was sitting in the front row for this mornings keynote announcement. There were no collective gasps.
From first pass, saying CodeStar is going to kill DevOps is like saying that Lightsale (the VPS thing) is going to kill DevOps. At the end of the day it gets you in the door with AWS until your company can be adult enough to build something for real.
If anything, I think its good for DevOps. CodeStar could mean less hacked together junk at the get go. So when the real DevOps gets hired, less to cleanup and unf'k.
When thinking about how higher-level stuff can go(without AGI etc), programmers meet the limit at understanding the customer and explaining it to the machine, which leaves them quite a lot of work.
But DevOps ? wasn't/ins't it always just a matter of time, for most small/medium sites, until someone hides all of the complexity behind a nice, smart interface and leave very little for people to do?
That regular feeling of obsolescence is pretty germane to being in tech. I remember feeling that way about Quantum Computing... in 1999.
The nice thing is the technology moves. In fact I imagine that's what attracted you to the industry in the first place. As these things roll forward we start doing more and the pie just grows.
Maybe the future of DevOps is in managing, deploying and optimizing AI stacks? Or maybe just managing the -cost- of AI? Or how do we manage and exchange personal health data on these platforms? Or biotech experiments? Or global IoT stacks? Maybe it's managing APIs, or solutions that are federations of third-party services. So much stuff in the technology pipeline it blows my mind.
Some of the harder skills may become less relevant (they'll never go away, in actuality). The principles and soft skills you have still have a long way to go.
You're just being a bit hyperbolic, silly even. Managin any environment, on-prem through hybrid through cloud is a non trivial task. Don't let this or anything else make you think otherwise. Take a look at all the devops containerization, scheduling and automation software. You can do a lot of stuff. But still it takes at least one person or a team of people to actually do it and be responsible for it in production.
This stuff is not as easy as vendors or aws would have you believe.
The sound you hear is millions of devops engineers counting their money, courtesy the proceeds from migration to aws, then migration to higher and higher clouds, then, one day, once the pendulum has reached its extreme swing on one end, migration back to on-prem :)
mainframe, timeshare, client server, cloud, it goes around and around
They should also try to make sure all of their descriptions are actually correct. They describe SNS as "AWS Messenger" and say to "Use this to
Send mobile notifications, emails and/or SMS messages". That's only one specific use-case though. Really it should be called "AWS Callbacks", and you can use it to create callbacks from other services and webhooks which can trigger notifications to mobile devices, emails, SMS, or other AWS services.
I don't think you're weird. Instructional or informational videos force their pacing on you. Text and images allow you to learn at your own pace, be it faster or slower than a video.
Few things are more frustrating than having to skip back 15 seconds multiple times in a video because I didn't quite get something, or trying to hunt around ahead in the timeline because the current stuff being presented is trivial introductory fluff.
I am the same. Give me text! I skip around all over the place, and can (occasionally) read (short) paragraphs in a glance. Videos and their enforced pacing :/
Amazon's design language is very meh. Even if you aren't in a cubicle, as soon as you are on their website, you feel like you are in a cubicle. With bad lighting. Heroku is way ahead in this regard. (not saying that is the most important consideration)
This looks amazing. When it matures, this is probably going to kill Heroku.
This may also mean that I will never have to touch devops again. I may never have to use Terraform, Kubernetes, Docker, CoreOS, and all that other stuff. At least, not for small to medium-sized deployments. It also means that I'll never need to touch Dokku or Flynn. I was about to say that some company just wasted a lot of money on Deis, but then I remembered that it was Microsoft. So maybe that was a good move, although I feel like Azure is the Bing of cloud providers.
I'm actually angry that this took so long. This should have been available 5 years ago.
> I may never have to use Terraform, Kubernetes, Docker, CoreOS, and all that other stuff.
True, you may never have to as long as you're okay putting 100% of your faith in the benevolence of a single company. Remember, a good portion of AWS services are completely proprietary and serve as a pretty fancy set of golden, jewel-encrusted handcuffs.
Fully open source projects like those you mentioned, on the other hand, are positioning developers who know them to treat the cloud providers of today as just another source of CPU cycles, storage, and networking. For the services I run on Kubernetes in AWS, I can lift and shift those in a day if/when the day comes that Amazon is a little less benevolent with their pricing and service levels. If we were asked to do the same with our services built expecting AWS functionality, it would takes months to years. Guaranteed.
Yes, there's a trade-off that I don't get 100% managed services and I have to know a little about the workings of the platform, but I would make that same commitment to AWS the second it didn't do everything for me. In my experience, that moment usually occurs somewhere around the 10-minute mark.
Amazon is great today, but Bezos won't live forever and who knows what happens when his successor takes over. It could be great, or it could be a mass exodus of people dumping years of investment. Personally, I'd rather hedge my bets and use their low-cost VMs as my source of compute today, but leave myself plenty of dead-simple exit options and sleep a little more easily at night. That's what open source software is all about.
The problem is, most people won't work for a company long enough to see your hypothetical doomsday scenario (if it even does). The people who will be screwed are not likely the ones making the decisions today.
It's not just the doomsday scenario. It's smart to avoid vendor lock-in because eventually a competing cloud vendor could offer better pricing or more reliable service. Or you could run redundant infrastructure on AWS and Google Cloud Platform at the same time to insulate yourself from major vendor outages.
Also, not all doomsday scenarios are terribly far fetched. AWS might one day have to discontinue a service after losing an intellectual property lawsuit, for example.
What Amazon has done is created the first DevOps free experience. I am sure Microsoft/Google and others will be on the bandwagon very soon. This product is VERY VERY good for someone like me who hates DevOps
You are way underestimating how many of us really, really love Heroku for our small clients and projects that run just fine on one or two little dynos. Oh and their add-ons marketplace is amazing--I can't see AWS replicating anything like it, since they have their own competing product offerings.
I doubt it will kill Heroku. For some developers and some types of projects, the minimal effort to deploy and push new versions of apps to Heroku is worth the $7/month for the smallest paid dyno.
Years ago I wrote a blog on how to set up git commit hooks, a sample shell script to update an app, etc. I did that myself for a year or two, then went back to using Heroku and other services.
There are a lot of small projects that live inexpensively on Heroku. That said, I would personally not do a large multi "server" deployment there because of cost, but for the small stuff I think that they are the 'best of breed.'
The small projects are a loss leader for the whales that spin up thousands of dynos. I would guess a lot of the whales get there without thinking too much about it - each step along the way its way easier to spin up a few more than to actually pay for ops.
What this has me thinking now is that a micro beanstalk doesn't cost more than Heroku initially and will scale much more economically if my product takes off; and I can actually achieve `git push` deployment without a lot of yak shaving.
I highly doubt that, Heroku is a great product. Very reliable in my experience and requires zero management. Under the hood of CodeStar, EC2 services in particular, still need management, patching, etc. So it feels like a quick way to spin up your app but after that there's still overhead to keep it going.
I hadn't heard of skyliner before. You guys may be a bit scared but at the same time, I haven't seen AWS create a nice api interface for their products. For so many teams, it is still just to complicated to set-up and get going.
I had a similar scare yesterday when Google launched Google Earth in the browser with a bunch of technology which, on the surface, competes with my start-up, but digging deeper, there is often still an opportunity, and they've justified your market.
AWS has a lot of resource to build something "easy to work with" like heroku, but it doesn't. Maybe CodeStar is an attempt ?
CodeStar is more like AWS Elastic Beanstalk, the one that I still don't know how to take it into my toolchain and development flow.
Now AWS CodeStar, that requires bunch of clicks, wizards, permissions configuration, choosing a template and then another changes...
To get started with they needed to show bunch of screenshots, while something like Heroku would tell how to use the whole thing in 2-3 commands in terminal. That's all.
However, I hope something like CodeStar get more polished and mature so I can move have one place to bill and one thing to worry about and not docker, kubernetes, mesos, ansible, ssh into machine, database backups, etc...
edit: Just found out for web applications they use Elastic Beanstalk as deploy step.
I don't care about lock in, and I just want something secure and easy to deploy an application. Pricing is important, but heroku gets expensive fast, and I'd eventually like something that I can manage myself once released and I can dedicate more time to.
As a small point of comparison, redis on Heroku is $200/month for a gig of memory. 1.55G (cache.t2.small) on elasticache is $24/month. For redis at least, the difference is an order of magnitude.
The t2.small will kick you in the pants in two spots, FWIW: compute and bandwidth. The first probably won't matter for a redis cluster, but the second could definitely hurt.
Amazon refuses to be pinned down on how small (it's "low to moderate" bandwidth on their instance matrix), but you will feel the pinch with extended usage.
If you've only got a gig of RAM available seems unlikely you could make any decent use of high bandwidth situations (or if you are, your cache is going to pretty much constantly evicted).
But still, you can hop on an m4.xlarge for only 150 a month. Get 14GB of available cache on 4 cores with high networking.
FWIW, redis is maybe an outlier here, but it's pretty stark. The prices are just so far out of reasonable range it feels like. Their postgres pricing seems more in line (assuming heroku's support is good, I have no experience with such).
Depends on market factors. Supply and demand and all that. AWS has created a situation where, from the consumer POV, they have infinite supply to meet whatever demand. Or maybe there is just so much slack in a data center system that it feels that way.
The real question is what does the cloud platform future look like when that slack goes away, but I think we're pretty far from that reality.
Edit: Looking into the CloudFormation templates created, there are CodeStar resources, so, I assume, the new resource types are just not documented yet. Not that it needs much documentation, but I will try creating a template with those to test if it works.
Edit 2: Here are the new resource types:
CodeStarProject:
Description: Starting project creation
Properties:
ProjectDescription: AWS CodeStar created project
ProjectId: !Ref 'ProjectId'
ProjectName: !Ref 'AppName'
ProjectTemplateId: arn:aws:codestar:us-east-1::project-template/webapp-nodeweb-lambda
StackId: !Ref 'AWS::StackId'
Type: AWS::CodeStar::Project
Version: 1.0
SeedRepo:
DeletionPolicy: Retain
DependsOn: [CodeCommitRepo]
Description: Adding application source code to the AWS CodeCommit repository for the project
Properties:
CodeCommitRepositoryURL: !GetAtt [CodeCommitRepo, CloneUrlHttp]
DefaultBranchName: master
ProjectTemplateId: arn:aws:codestar:us-east-1::project-template/webapp-nodeweb-lambda
Type: AWS::CodeStar::SeedRepository
SyncInitialResources:
DependsOn: [SeedRepo]
Description: Adding the AWS CodeCommit repository to your AWS CodeStar project.
Properties:
ProjectId: !Ref 'ProjectId'
Type: AWS::CodeStar::SyncResources
Version: 1.0
SyncResources:
DependsOn: [SeedRepo, CodeBuildProject, ProjectPipeline, SyncInitialResources]
Description: Adding all created resources to your AWS CodeStar project
Properties:
ProjectId: !Ref 'ProjectId'
Type: AWS::CodeStar::SyncResources
Version: 1.0
At a glance, this seems very similar to functionality that has existed in Azure for a very long time (templates in multiple languages, automated build and deployment, monitoring). Not sure why Azure gets so little love here on HN!
I don't mean to knock it, it's great to see this functionality make it to AWS, I was just wondering if there were any significant differences to Azure's offering?
Apparently Azure has been sneaking these features in over the past year or so... I do .NET dev on a daily basis on self-hosted infrastructure and was aware of VSTS and such, but after looking at it now I can safely say that back in early 2016, Azure was nothing like what I'm seeing now.
Back then the Dashboard was a steaming pile of Javascript and the TFS / VSTS split had just started. The new Portal appears to be much faster and it looks like they have much better docs:
From what I'm reading, this sounds quite a bit like Heroku-style functionality for AWS. Git push is your entire process to deploying a new build, without needing to set up a deployment pipeline, while having the usual escape hatches for deployment that the AWS toolchain has.
Definitely a cool thing, a good way to have something that's easy to start with, but as you need a more complex infrastructure would let you add it as necessary without incurring the usual "learn everything at once" cost.
Wow, this does look awesome. Btw, it doesn't look like they’re promoting this in a huge way, but it just showed up in my feed that Atlassian is offering a free new JIRA Software license if you sign up from within CodeStar. I've been using Atlassian stuff since 2007 and I don't think I’ve ever seen them do this before. I don’t think there’s a free version of JIRA? Has anyone ever seen that before? Pretty sweet! https://twitter.com/JIRA/status/854805603738365953
They keep climbing up the stack. The final step would be to provide ready-to-go outsourced development teams. Maybe they should acquire Gigster to plug the last gap. ;)
I would say it's the complete opposite of lock-in. If more cloud providers follow suit, then one day you might be able to simply `git push` to a different provider, then wipe away all your old infrastructure. I'm overstating things, but 12 factor apps should be very easy to migrate, excluding things like databases, DNS records, user roles, etc.
But I think this might already be true if you're comparing Heroku with AWS.
Correct me if I'm wrong, but all the build and deployment steps are being managed via Amazon-specific tools directly, which want you only to use Amazon-specific services, no? That sounds like lock-in to me.
It's pretty trivial to do this kind of thing with Jenkins or Gitlab CI and those don't have that issue. And then you can deploy however you want to any provider.
Sure, but Heroku also manages all your build and deployment steps. Migrating away from Heroku is as easy as pushing your code somewhere else, then deleting the app on Heroku. Ideally all of these build and deployment steps are automated for you, because let's face it, most web applications and APIs have very, very similar hosting requirements.
Maybe I don't really understand the concept of lock-in. I've never felt "locked in" to anything on AWS, unless you're talking about contracts for reserved instances.
Lambda is a good example. You can't pick up and move a lambda solution to another provider without significant overhead, the lock in. Architecturely, migrating a lambda to another compute platform shouldn't be too difficult if its structured well and functionally, but it's still work that would need to be done.
From what I can tell, this is a web interface wrapper around a suite of Amazon tools – but nothing is forcing you to exclusively use them once you've created your project.
> It's pretty trivial to do this kind of thing with Jenkins or Gitlab CI
I think you might be trivializing what's being offered here. This is bigger than a CI server.
Can you help me understand what is being offered here? Is it basically a wrapper for, in the Rails world, spin up EC2 instance -> install make / git -> clone repo -> bundle install -> install nginx -> configure nginx to whatever app server they have decided you will be using -> set up CI server? I wonder if you can customize how the recipe gets built?
It basically just seems like an AWS specific deployment recipe as a service for people who might not be familiar with a tool like Ansible. Is that accurate?
Not using their code hosting, use GitHub for code + Wiki.
My question is CodeStar only worth using when you need to deploy a basic "template" app out the door? What happens when our requirements mean changing web server config, firewall rules and all the actual things that happen that are customizations? If you start on CodeStar are you "stuck" with it or can you easily get off it but keep the underlying services? I know its just EC2 horsepower under the hood. The last thing I need is a service that isn't customizable, if that's the case maybe I'd be better sticking with straight Elastic Beanstalk?
I'm excited to see Amazon entering this space since I think far too much time is spent on setting up and managing devops tools, especially for small and medium-sized projects. [0] I've personally wasted lots of time on repetitively setting up CI/CD, autoscaling, and other basic infrastructure for different companies and projects. Anything Amazon can do to reclaim that time (and encourage more people to adopt best practices) is awesome.
That being said, after poking around I have a few criticisms:
1. It doesn't seem like the example templates include Docker. At this point, I think Docker should be considered a must-have for any new ops tool. It makes your application much more portable and eases the learning curve for new developers.
2. Getting things set up and working is still too complex. It took me 20 minutes to get my first project working fully (even just using their standard tools).
3. There doesn't seem to be any ability to bring in an existing project. It'd be awesome if I could just provide a GitHub URL and Amazon would automatically set up all the ops tools I need to have a production-ready deployment of that project.
I welcome the competition though. Let humans focus on unique work while computers automate things.
I haven't tried CodeStar yet (planning to do it on this weekend.), but from the description it seems to have many moving parts already.
>>each project using AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy.<<
As a developer I don't need to know all of that. All I need to know is one command to deploy my deployment package for the project. That way I am productive quickly. Then give me the ability to configure the VM/container size and autoscaling. Now that my app is up and running, give me something like Azure WebJobs to run my worker tasks.
Amazon needs to have a better PaaS story than what BeanStalk is. BeanStalk is very flexible but in return, it ends up being too complicated to set up.
Compared to Heroku and Azure App Service (which I am using in a .Net project and was surprised to find very easy to get started with.), AWS Elastic BeanStalk is very complicated. With Heroku and Azure App Service, the developer friendly PaaS UX is more or less a cracked problem. AWS just needs to copy it well.
Amazon is the leading hosting provider that continuously tries to hide the infrastructure cost to his clients aka developers so they paid more. Be careful of this "things" pricing should be taken seriously when considering AWS.
Why wouldn't Amazon use its recently acquired c9.io as the development tool for this? Did they buy it just to drive it into the ground because activity from the staff has dropped quite a lot.
Nice - though a lot of these Code* products still seem fairly half baked compared to their dedicated competitors. I kinda wish AWS would launch fewer better products than so many mediocre ones.
Does this take away the need for customers to develop their own tooling to interact with various AWS services? It's hard to understand what this is doing.
My understanding is that this is like Heroku, except there's no additional charge, and you have full access to all of the services underneath in case you need to customize something.
It seems like this is a tool that simplify the process of creating and managing services to help speed up the development process, while still leaving the typical services that AWS provides available.
I don't know how everybody feels about this but I really like to control my whole environment from the OS to what servers I use, how I secure them, what kernel features do I allow what packages do I install and how I protect my services. If something goes wrong, I fix it. If something goes wrong with CodeStart you wait for Amazon to fixit. I don't like giving up control.
There really needs to be a tradeoff here. Control is great, but it can't be so black-on-white as to implying that nothing at all would convince you to give up control all the way down to kernel features.
Plus, it's a template, not a closed solution - after it runs, you can observe what was created and tweak to your liking. If you do so much configuration as you're implying, you probably have some templates of your own.
This doesn't exactly stop you from doing that though. Companies that need to turn over products quickly and efficiently would benefit a lot from this but I wouldn't see myself using it for everything either.
Would this make for a good way to get into aws/devops since it narrows the perspective of their services?
I have no experience with devops and want to learn to manage my applications directly (mostly use services like now or heroku currently), but looking at learning aws always seemed intimidating with the 600 different services they run.
>Would this make for a good way to get into aws/devops since it narrows the perspective of their services?
It depends what you mean by that. In a big way, this negates the need for a dedicated Ops engineer for small to medium deployments on AWS. If you're a developer looks to operate your own small application, this is a great entry point.
I'm very curious to see how abstract aws is going to end up getting, would they build a browser (call an aws app directly-ish or something..?)? IMO There is a paradigm shift a-brewing and it doesn't look good for most of the clouds/vps providers.
I like how they make it look so simple, but then you go to do it, you'll keep cursing at this thing...I think debugging problems might be difficult...I wish in something like this they show a real video making this work and how long it really takes...
"AWS CodeStar is a cloud service designed to make it easier to develop, build, and deploy applications on AWS by simplifying the setup of your entire development project."
So amazon moved from letting you develop from the cli to a gui-code editor.
I was surprised by that. Since Microsoft has been pushing Azure hard the last few years, you'd think Amazon would battle back and come out swinging with a C# template.
Did you take a look at the CloudFormation errors? There's probably a reason why its failing. This happened to me and it was due to me having a CodeCommit repo the same as my project name.
I don't think it is a cloud IDE since it asks you to choose between Visual Studio/Eclipse/Command-line to edit code?
It seems to just be something that makes it easier to get started with AWS. A bit like how you can choose Azure when creating new projects in Visual Studio, which will take care of Azure-related boilerplate
DevOps is a set of practices we should all be using, not a job title. Operations Engineer would be a better title, and they aren't ever going anywhere. DevOps Engineer is a job title -- one that I'm not a big fan of.
"Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not."
I would like to know as well. I agree it's not "sexy" or "wow" but it communicates what it's meant to. This is like any graph my college professors made - they did their job.
The fonts are all off. Looks as though antialiasing is turned off, and the kerning is all wonky. It's also inconsistent -- look at the spacing between the top of the boxes and the headers.
CPanel has a terrible UX. TBH, I don't understand the point of the product. I see that it makes it easy to "edit" files without having to SSH but seriously server administration goes away beyond editing files without understanding syntax. I can only take comfort that most of CPanel deploys are in intranets (or so I hope).
Don't forget that installing CPanel also basically hijacks your system. Want to remove CPanel and manage everything properly? Just reimage and build from scratch, otherwise you'll end up going bald from scratching your head, and blind from hatred.
I understand the urge for some people to use a control panel, but CPanel and it's like seem specifically designed for people that want to be webhosts without knowing anything required to be an actual webhost. I think that's reflected in the UI/UX.
Not only actively harmful, but I honestly believe it's designed to be unnecessarily complex to make nonadmins feel like they're tackling admin challenges. I'll stop short of saying it's a dark pattern, but the case could probably be easily made.
And yet, I like the current look. It's clean and functional, and doesn't have the modern/flat/whatever style that makes one lose context easily. Do you prefer something like the S3 redesign? I'm scared they will do that to the whole site.
(Not to mention, most tasks you should script/do from the CLI, once you got the hang of things.)
Can you script 'reading the docs' for me, because that's the part where their design most grates ;) ? Por ejemplo, for tables of methods/functions on objects, the cell sizes are ridiculously huge, so I have to zoom out to like 25% on the page to see more than 2 funcs at a time.
I tried deploying a django app on EBS and node app through code star. The whole experience was really simple. It adds a whole new level of abstraction on deploying and maintaining apps on production level. Though similar solutions exist in market such as pythonanywhere and heroku, but all of this from AWS is just simply phenomenal. I 100% agree that code commit is just amazon trying to push through one of its products. But i don't think that really is the intent of codestar. They want to provide one opinionated workflow for your entire development lifecycle, which works incredibly well for teams of all size. And plus I wouldn't really worry about the lock in situation, thats a good problem to have if you reach a stage where you need to think about these issues and given aws's and overall amazon's stance on pricing I personally don't worry about it at all. Had this service been provided by microsoft I wouldnt have gone for it because of their history of over pricing and creating artificial lock ins.
I had tried configuring EBS prior with one of my apps, but the quirks of setting it up had always left me quite frustrated. But this is actually a game changer. I am sure aws will add more templates for different stacks and add more options of extensions - right now only extension is jira. Overall I see codestar evolving into the central core of developing, maintaining and deploying web apps for companies of all sizes.
I think this coupled with their acquisition of c9.io is beginning to look like a real "cloud killer app".
My vision for AWS is being able to login to AWS and see c9 with one click deploy and configuration.
I'm patiently awaiting for this. It was one thing that really woke me up to Azure with their wizard form directly from Visual Studio. After AWS reinvent 2016, the Azurphoria wore off and I was an AWS fanboy once more. Having the ability to edit/deploy inside my browser would be a huge tide turning moment.
It's becoming hard for me to justify staying in the 'DevOps' space with amazing solutions like this coming out regularly from AWS. I can hitch myself to the AWS wagon for a while, but eventually, it seems, dedicated operators just won't be needed for most small to medium deployments. It's tough, because I can see that this is genuinely moving the industry forward, but it's also negating the need for skills I've worked for over a decade to build. I guess this is what carriage builders felt like 100 years ago.