Hacker News new | past | comments | ask | show | jobs | submit login
DevOps Engineering: Who does your automation?
40 points by emcooke on July 27, 2010 | hide | past | favorite | 20 comments
Here a brief intro for those that aren't familiar with DevOps: http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/

Before being introduced to the concept of DevOps, we at Twilio struggled with what to call the role. It requires a different kind of thinking. To us, DevOps is about automation to improve productivity, reduce risk, and achieve scale. DevOps engineers should always be striving to deprecate themselves. It has been more then 2 year since we founded Twilio and we've never had a sysadmin or an ops person. This has forced us to write better software automation from day one. However, having someone that can provide leadership on DevOps issues (and fundamentally understands distributed systems/CAP/etc) to the whole team is extremely valuable.

Configuration management tools like Chef and Puppet are the beginning but a more complete view of infrastructure and service automation (e.g., http://redeyemon.sourceforge.net/) seems inevitable in the long term.

We are looking to hire a Lead DevOps engineer. Here is the detailed job description if you know someone that might be interested: http://twilio.jobscore.com/jobs/twilio/devops-engineer/bSHjYWq2Sr37U6eJe4aGWH




Linky - http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing...

Call it what you will, 'DevOps' is just what you'd expect a good sysadmin to do in a sufficiently large org. That there is on going friction between developers and sysadmin ("it works on my box" is a valid excuse?) is perhaps a larger sign of the culture and people involved.


I expect developers and sysadmins to work together on these things. I'm always confused when they don't, and saddened when people think that's okay.

For instance: I wanted to do a few projects in Rails. I made sure that my admins had links, guides, and all the info they needed when I started the project. By the time I was ready to launch, they'd already had Ruby, Passenger, and all my dependent gems installed. Deploy was a breeze.

Later, a different coworker did a project in Django, but didn't breathe a word to our sysadmins. When he and the client agreed that it was launch time, our admins were scrambling to figure out what version of Python we needed, how to set up the environment, and what the deployment best practices were. Everyone was angry and the deploy was late, but the developer didn't see any consequences.

Kills me.


I feel the same way.

In the old-world way of things, the Architects of the system would take all of that into account - the sysadmins were just there to keep the systems up and running and install the software per the developers instructions.

What's new, or seemingly new, because of the rapid advance of technology and the internet in general, is the rapid develop/deploy cycle, which ends up requiring a much tighter coupling between development and operations... the two almost blend together. The lines are getting fuzzy.


Whenever I hear that term I cringe a little. I think this was a word invented by developers (please correct me if I'm wrong) to bring sysadmins into the fold. It is definitely good to have an understanding of the application layer but developers also need to understand the systems running their applications.

I work in a large organization (5000+ employees) as a unix/linux sysadmin. You have to automate deployment, monitoring, configuration management because there just isn't time to do anything else. We have a group (about 4) out of 30+ that have an overall view of what you might call devops. The rest specilize in development or sysadmins. We meet once a week and review all progress on the dev/ops issues.

I like the term OpSec (http://en.wikipedia.org/wiki/Operations_security) someone who looks after Systems/Security ;) Since a good system admin should also know a great deal about security. Maybe Dev/Op/Sec.. haha, but really ;)

p.s. I found that job posting to be full of PR fluff. It would be nice to see job posting actually post what their looking for in a distilled form.


Thanks for the pointer to OpSec term. DevOpsSec... I sense a tshirt in the making?


I first heard about OpSec from Roland Dobbins with Arbor. Really smart fellow who has many suggestions on network design / defence.

I just think people need to specialize in their field (+ a general knowledge of how the overall systems works).

Almost how a carpenter, plumber, and electrician each specialize but probably have a general idea how everything works. Do you really want a plumber building the foundation of your house? Just seems a little silly to me. Each domain is so large that I would rather have deep expert knowledge in a area than a general handyman who might cut corners or waste time with solutions he doesn't know about.


We have a group (about 4) out of 30+ that have an overall view of what you might call devops

This is called "second line support".


Devops may be what a good sysadmin does. I'm not sure that job is. Looks far more like they're looking for a developer to write sysadmin tools.


This is exactly true. Not to mention "DevOps" is not a job position, it's a culture and philosophy. In the past few months as DevOps has taken off, people listing for "DevOps" job titles usually mean highly proficient sysadmins. In this case, Twilio just wants a developer with a cloud management background. There's nothing DevOpsy about it.

I'm sad that people are not taking the time to learn exactly what DevOps means and thusly commandeering the name for whatever they think they need.


people are not taking the time to learn exactly what DevOps means

Now I know that "DevOps" means "those who forget history are doomed to repeat it". Whenever you come across something in IT you think is new, take the time to find out if mainframe shops were doing it in the 1970s... Back then sysadmins wrote assembly language to automate systems tasks and it was just an everyday part of the job!


Ouch. See, DevOps is not just "sysadmins writing automation tools". Your post illustrates my point perfectly.


Can't someone whose role embodies a certain culture and philosophy not have a title that fits that goal? How would you define DevOps?


In a 100% ideal scenario an organization would only need developers as every possible failure/problem would be covered and handled by automation.

Obviously the real world is different. New code is deployed that has bugs, there are unpredicted events, there are complex failure scenarios that are difficult to automate etc.

A DevOps engineer may write automation software in conjunction with developers to automate the operational aspects of business logic. Thus, it's a partnership between the DevOps engineer whose metrics are driven by availability/reliability/scalability/security and the developer who is trying to attain some business objective.


You say "DevOps engineer" when you really mean SysAdmin, but I upvoted you anyway because you get it.


Sometimes I wonder if DevOps isn't really just those who would truly prefer to be developers but don't mind sysadmin work and, for whatever reason, started out doing the latter.

I, too, cringe a little when I hear the term, perhaps because I've always viewed myself as a "pure" sysadmin. That is, this is the kind of work which continues to bring me joy after 2 decades. Application development, on the other hand, despite an ability to code, does not.

CM tools (current and their predecessors like cfengine) as they tend to be implemented strike me as a way of avoiding administering the whole system, by focusing on a single component (servers) and reducing them to a lowest common denominator.

It also fits the pattern of trying to solve problems entirely in a custom software solution, a trait I associate with developers, not sysadmins. This is the source of my inference of DevOps' true leanings.

The reality is that the system consists of considerably more than fungible servers[1]. There's the underlying hardware, the peripheral hardware like disks, database software[2], network hardware and configuration, and service providers.

These are all traditional sysadmin areas, with plenty of room for professional growth and overall benefit. So much so, that I have difficulty imagining why any SA (not a developer in SA's clothing) would wish to do more dev at the expense of increaaed mastery of the infrastructure.

By way of example, something I wrote on LinkedIn, in response to why I'm not a fan of puppet:

  I'm uncomfortable with any system that attempts to maintain state in place
  on a running system, as well as possibly override the native packaging ethos. 
  That the client takes its own local file inventory on a fresh system, rather 
  than trusting the one the package manager has, is wasteful at best. What's 
  worse, to me, is that it creates yet another system to administer. I haven't yet 
  found anything similarly suitable for the Ubuntu/Debian world, but I very 
  much liked Cobbler a couple years ago, as it was primarily a wrapper for 
  what could otherwise be standard, standalone services (kickstart, package 
  repo). Even without such a wrapper, yum/rpm or apt/dpkg can do nearly all 
  the work"
[1] Virtualization and "cloud" notwithstanding, as anyone who's experienced an outage with a hosting provider or an i/o perofrmance issue with a cloud server can attest.

[2] Which can be like an encapsulated OS itself


Yea I think like some the term DevOps makes me cringe as well since people use it as a job title when it should be more thought of like agile development, just another term to describe a process of working.

I use chef because I needed a way to consistently deploy software in exactly the same way every time to multiple servers. A repeatable deployment process that scales. Why did I pick chef over puppet or cfengine? Because I also wanted to learn ruby and the entire thing is in ruby, win. Plus we have a ruby on rails team in the office, if ever I needed a hand, they were there or if they wanted to write recipes to deploy software thats not an issue either.

DevOps is just a new buzz word. My title is System Administrator. I have a depth of knowledge in the infrastructure of all our technologies and how they are interconnected. I know how to deploy, maintain, scale and do it all over again. My knowledge in programming in second to that but is necessary to automate repetitive processes. DevOps I guess is being classed as a way to bring the IT team together, making sys admins and developers aware of each others "agendas" when it comes to running apps on servers. I as a sys admin have to understand what each of our applications does throughout its life cycle, from the browser right down to the database. Because if something goes wrong, I'll be the first to figure out where that happened and relay this back to the developer. By understanding core functionality of the code I can make it clear to the developer, "look you are executing a query on the database thats scanning millions of rows because its not using an index, its occurring in this controller in your code". We are seeing this term DevOps be classified now as more and more startups are emerging, because its essentially alot easier to execute in a startup environment where everyone is talking to each other to roll things out, everyone is trying to solve problems together. In large companies you'll see a massive separation and segregation along with processes to keep people in their place and slow results.

I hope that all makes sense. DevOps as a process is not new, the classification of it as this term is.


40 years ago IBM called these guys "systems programmers". That this is a new concept to some people is symptomatic of a wider problem in the industry: everything needs to be shiny and new so every 10 years we reinvent the wheel with new jargon.


DevOps is not just "writing automation tools". If that's all you're getting out of this, you're missing 90% of what's going on.


What else is it, then?


There is a bit of the same movement in CCP Games where I work, it's not called DevOps here, but there are groups forming in both the EVE Online software team and the Core technology group with an engineering focus on QA, which in practice translates pretty well to the DevOps style of thinking about the needs for this type of development and quality work. Here's one of the advertised positions for example:

http://www.ccpgames.com/en/jobs/job-details.aspx?jobid=101




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: