First, let me say I totally agree with the premise here--that all parts of a software engineering organization need to be on board with DevOps, buying in to the vision, learning the full stack, taking responsibility.
However, this article does serve as a good example of one of the big problems with most DevOps/SRE evangelism: It's usually coming from the Dev- perspective, rather than the -Ops perspective (or better, a balanced perspective). In particular, this focuses mostly on getting code to users, and speeding up that process. While that's certainly important, it leaves out any discussion of:
Successful DevOps cultures must address these things just as much as they address code feedback loops, time-to-production, rollback strategies, code pipeline automation, etc.
A company I worked for started a devops team. 3 years later and there's still not a single ops person on it. There were so many security holes in their group we had to ban them from releasing to our servers, so now they have their own cloud setup that is bleeding money.
The frustrating thing is: I'd have loved to work closer with them. But every time we tried, the response was "You have no place here, your slowing the process down". We just wanted to automate the security testing, they decided that throwing the security tests out altogether was a better idea. I do admit that it was indeed faster....
Just because someone has an Ops title doesn't mean they know anything about security. I've worked places where the Ops team caused every security incident.
Its a business leadership issue. When teams accept requests from the business and prioritize those requests over security then we have problems. Ops teams can be just as guilty of this as any other team.
Agreed, it seems like a lot of people think about devops as a way to make dev's lives better without actively harming ops. The article does have a section on getting Ops on board with devops, but it's much weaker than the previous section on getting Dev on board. Devops done right should really improve both sides, as it explicitly recognizes the tradeoffs involved with the added automation / processes / policies. I've been reading Google's Site Reliability Engineering book ([1], free online, highly recommend so far) and it lays out pretty clearly how to do devops in a more balanced way. Key points are:
- Define service level indicators (SLIs), metrics you can track to see how you're doing on the things your users care about: uptime, latency, security, etc. For instance, measure uptime as successful responses / total requests.
- Define service level objectives (SLOs) based on your SLIs, e.g. 99.9% successful responses / total requests (aka 99.9% uptime)
- Establish an error budget which controls your deployments. Based on your SLOs you can accept a certain number of errors (failed requests, outages, slow response times, etc) for a given period. If you spend your entire error budget, no new code can be deployed. The deployment shutoff has to be agreed upon in advance and it has to be a hard and fast rule, so it's not ops' fault that dev can't deploy code.
- If you reach your error budget, devs need to work on things to reduce error rate going forward instead of developing new features. It can be tests, monitoring and alerting, rollback process, or pretty much anything, but devs should know that they're doing it so they can deploy more code faster in the future.
I think this makes a lot of sense and helps to align dev and ops on uptime goals. Specifically, dev's ability to push code is now directly tied to service level, and ops is not shooting for 100% reliability. Dev can choose how to spend the error budget, so ops isn't dictating how they do things; and ops knows and expects a certain number of problems, so they don't resent dev every time they push buggy code.
My experience is that even customers who are very enthusiastic about these new processes and methodologies quickly turn back to the old-school way of doing things once someone in upper management raises these bullets. And they weren't considered before because the customer Ops people were not invited to any discussions.
So in order to truly get the whole organization on board you do need to include your solutions to the above points in your evangelism.. and start really early.
In 2017, there's literally _no_ difference between the outcomes expected from a "Systems Engineer" and a "DevOps Engineer" - both are expected to produce HA, Automated, Well Documented Infrastructure, to allow developers to run their code.
Anyone not doing that is just a _bad_ "Systems Engineer"
Developers don't call themselves "Agile Developers" because they started using agile methodologies or "CI Developers" with the boom of CI, but for some reason Ops guys want to call themselves "DevOps Engineers" because they read The Phoenix Project.
People talk about automation as if automation = DevOps, but I believe automation is a side-effect of the shared empathy which DevOps fosters. Because you have that empathy with your devs, you realize that their main goal is to be able to get their code in front of users _as fast as possible_. Automation is the answer to that.
> In 2017, there's literally _no_ difference between the outcomes expected from a "Systems Engineer" and a "DevOps Engineer"
I don't find the title 'Systems Engineer' to be in widespread usage at tech startups in SV, so I'm not sure what to expect from one. To be honest, I would expect Systems Engineers to be working on hardware.
The infrastructure automation work done at modern tech startups is best described by the title 'DevOps Engineering'. If you're posting a job where the requirements include using automation code to set up and configure CI/CD, monitoring, alerting, Databases, Analytics, AWS, GCE, Heroku or other 3rd-party SAAS tools, the job title in widespread usage that encompasses that is DevOps. The Webmaster, Ops, DBA, SRE, IT, Systems Engineer, and SysAdmin titles describe different skillsets and are therefore less useful for describing a role with these kinds of requirements.
"Systems Engineer" seems to be an overloaded term. In the area I work in (applied R&D, East coast) the role is a mix of the software architect and QA, but encompassing both hardware and software.
Some science background (whichever science is relevant to the project) but mostly a lot of experience in general engineering practices (what generally works, how things break, what is sufficient to test to feel good that a unit will still work when moved from point A to point B, etc.). Many have software skills, but some do not. It is generally a high level role so not all projects have one even part time, but those that do tend to have it scoped roughly the same.
Projects that deal with field work / tests, especially bringing stuff to faraway / hard to get to places tend to value systems engineers (defined as above) highly and almost always have one at least part time. Just my experience.
Again systems engineer can have more meanings still, when I hear systems engineering I think of interaction specification of mechanical / electrical systems particularly in automotive, rail, and aerospace.
managing actual physical systems, not software, although the skills are similar besides domain specific knowledge.
Whilst I 100% agree with you and wish it wasn't the case, title inflation is kinda necessary when the pay and authority reflects the BS titles.
I'm not sure what it's like over the pond, but here in London I sit round tables with a bunch of "(security, system, application, cloud) architects" with less than a couple of years experience in the industry (or sometimes 10 years of one years experience) who think they have a remit to define how things work. In a silly bureaucracy, you have to play the game or you won't have a seat at the table.
I do wish it wasn't so, but also having the title doesn't prohibit you from actually doing devops either.
On the flipside, there are people who valiantly call themselves "DevOps engineers" but are nothing more than glorified remote hands.
Thanks to the recruiters and the all-too-common association that "devops = the new ops", the very term has suffered a massive devaluation. I learned this the hard way - I was foolish enough to try to hire DevOps Engineers. That's a two-month span of my life I will never get back. That's why I came up with the idea of renaming it as a search for Infrastructure Engineers.
... only to find out that I wasn't the first one. The signal/noise ratio had obviously been a problem because other companies have used the term before I did, but on the other and, even more companies have joined the crowd since then.
(It was interesting to see this blog post, because it has almost the exact title I have had in mind for my WIP longer post on the same topic.)
That is because DevOps is basically what 'a good sysadmin' always did, only now it has a title, and those in the sysadmin role now 'don't have relevant experience' because 'devops' is not on their resume, and 'systems administrator' has been re-associated with 'server tech' as a result..
"the shared empathy which DevOps fosters". I haven't heard it described like this before but I like it a lot. The way you phrased it emphasizes the "put yourself in other people's shoes" philosophy. An attitude that I would argue is the key to a successful DevOps process.
And if it's the same job, to me it seems like a better title, less vague, more descriptive & accurate. Referring to "operations" seems to more clearly describe what it is that a DevOps or Systems engineer is doing, to me anyway, than the word "systems".
I'd speculate wildly that DevOps engineer is a title that web companies use, and system engineer is a title that non-web companies use for roughly the same job. But to me, for some reason, the two terms invoke images of pretty different day to day activities.
> for some reason Ops guys want to call themselves "DevOps Engineers" because they read The Phoenix Project
Are you saying that they don't deserve the title Engineer, and should be called "Ops guys" on the job application?
While I agree there is some title tweaking going on, IMO the empathy you speak of needs to go in both directions, especially if you're a smaller organization. It really needs to be, as you say, shared.
If shipping shit yesterday is the only goal your developers have in mind, they really can't expect quality Systems Engineering. They need to recognize that Systems Engineering is very much Engineering in the same sense as development and sometimes it can be just as if not more difficult and intellectually demanding especially in the face of such social pressures. Everyone needs to be on the same page.
Hence the need for such a culture and probably for an elevated title. I speak as someone who's worked in both roles and who's never read The Phoenix Project (tbqh).
To be fair it does look like that's what it's supposed to mean. It looks like a combination of Dev and Ops, at a glance it should mean it describes a combination of roles too.
I tut at management as much as the next guy but whoever came up with DevOps didn't really help themselves there :)
Well, the full term is Development Operations, meaning the branch of Operations concerned with supporting Development. I recoiled from the definition you offered because I've actually witnessed an organization use that as a justification for not hiring any Operations staff, which is a terrible idea.
That is NoOps, but somewhat depends. You could have a developer who has a dedicated job of managing the infrastructure, but still doing some development work. Also don't forget the build engineer part for automating deploys. But if the responsibility is spread across the team, that is NoOps. It also tends to lead to a tragedy of the commons situation, because no one steps up to do the things that aren't fun.
> "automation is a side-effect of the shared empathy which DevOps fosters"
I'd upvote you twice if I could. That is a an excellent phrasing.
I've long felt there's nothing more indicative of a high-functioning devops culture than a viable continuous delivery pipeline, but making the pipelines the goal would be to miss the point entirely.
DevOps, is a developer, that do admin stuff, because the development environment became more complex with the introduction of virtualization , cloud technologies and big data
Admins who do Dev work should be called OpsDev ...
A DevOps, is a Developer , someone who started as a developer and now do extra admin and systems work
I kind of agree with you that having this distinction would be useful. The real key is it isn't about "dev work". It is a cross between Ops, Dev, and Build. The build is actually more important than the Dev. Even bigger is who you report to. In the old world as Ops you reported to an Ops manager. The to Devops is that you report to the Dev manager, hence breaking down the silos. You could just call it Devops, but I call it embedded Devops to be clearer.
On the flip side I seen just how poorly Dev managers and Dev team leads can understand Ops. It is even worse when they think they do, but don't. This can make Devops painful.
There is also a problem with having an individual Devops on each Dev team. You will end up with each Dev team doing things completely different ways. One will be Ubuntu, one Debian, one CentOS, one instances, and one containers. There should be some standardizing force across the teams.
> the development environment became more complex with the introduction of virtualization , cloud technologies and big data
Eh. Spinning up a new VM is wayyyyy easier than racking a new box. And deploying to an app-running service is generally way easier than managing a bare-metal/load-balancer/etc-interfacing deploy script.
A devops engineer building and maintaining that infrastructure is a split that makes sense to me, but generally doesn't obviate the need for systems/network admins and engineers, as you still have problems and issues with the underlying things often come up (sometimes it's outsourced to a cloud provider, though).
honestly i am not really sure what you mean by "a split that makes sense to me", split between who?
the article of this thread/post have a very nice diagram at the top .. if the DevOps is focused more on the tasks on the right (ops) in my opinion he is not really a Devop
Builder are maintainers of the infrastructure are not DevOps .. they are just Admins
Developers of the software used to make this infrastructure possibles, are just Developers
Title inflation is generally something one does to one's own title though isn't it? Isn't it also done to "inflate" the perception of one's status to others. I don't believe that's whats going on with "DevOps Engineer" at all however. If anything its just an unfortunate misnomer.
DevOps Engineer is not being promulgated by the engineers themselves but by both recruiters and managers.
A quick search using "DevOps Engineer" on a job site like indded.com returns 196 matches. There is even "Senior DevOps Engineer" jobs listed.
The other contributor to the circulation of this as a title is companies that have a DevOps department and then by extension engineer in that Department become known to management as DevOps Engineers.
Having worked on over 100 DevOps initiatives, I'm inclined to disagree.
8 years into DevOps, automation, cloud computing etc, it's time to accept that DevOps is a specialism and a function that companies - especially big ones - need to put into place in order to adopt the principles.
I disagree with your disagreement. There's nothing we do under the umbrella of "DevOps" that wasn't part of competent service ownership twenty years ago. I read your article and the point that stood out for me was the notion that the skills to drive tools like chef, puppet are somehow new and different. But I know better, because although I use those tools today, back in the day we used cfengine and a bunch of declaratively/injectively configured shell & perl scripts to achieve basically exactly the same effect.
Back then we called this "systems administration". But we were most definitely working with developers and running the SDLC across silos.
As a relatively old guy, I always thought that DevOps was just the new name for what people have always been doing. It is literally just now that I've realised that other people think it is different :-) Possibly younger people don't understand what system administrators used to do...
"Possibly younger people don't understand what system administrators used to do..."
System Administrators used to blame developers. Developers used to blame System Administrators. Devops is in part an attempt to mediate between "silos".
Bad System Administrators used to blame developers. Bad Developers used to blame System Administrators. Real Systems Administrators attempt to mediate between "silos".
You get people who aren't very good in every kind of role in every kind of organization. Sturgeon's Law applies to everything, even people. If you have more than 10% of your team doing really well you're ahead of the game.
DevOps role is different because it assumes full product ownership. Typical system administrators are focused on just the infrastructure portions. In a DevOps role you are more focused on the things running on that infra, as well as the infra. It's a more encompassing role than the traditional 'sysadmin' role.
You're talking about the difference between sysadmins of the 90s and devops of today. The comment you replied to was talking about the difference between product engineering and operations engineering -- he's defending the idea that devops is a specialization of engineering as a whole. It looks to me like you're right, and the comment you replied to is right, and you seem to be agreeing.
He's not saying that chef and puppet are different because they're new; he's saying that using chef and puppet and docker all day long is a different job than debugging CSS and JavaScript or implementing REST api in scala all day long.
Right, but your parent is saying that job is not some new "DevOps" thing, it is just well functioning operations. "DevOps" is meant to evoke bi-directional buy-in and cooperation between the people working on development and the people working on operations. Re-naming the operations role "DevOps" doesn't get you that - you could still have low buy-in and poor cooperation between development and "DevOps".
I agree, and I agreed with that previous comment and the one before it. But I don't see where the comment I replied to made any reference to the name not doing what it says. I only see him saying that what we call DevOps existed before we called it that.
In any case, of course the name doesn't cause buy-in to happen, just like hiring an "Engineer" doesn't mean the person will do a good job engineering.
You're not saying that we should avoid a term meant to evoke the idea that cooperation is expected, because it might not happen, are you? I can only see rightness in trying to be clear about what's expected and naming roles to suggest positive cooperation is part of the job. If "DevOps" is meant to suggest buy-in and cooperation, you've convinced me that it's a better description than whatever it was called before.
I am saying that using the "DevOps" term for a role undermines the effectiveness of the term as a definition of culture. "I'm just a developer, I don't need to worry about this DevOps stuff, that's what we have the DevOps engineers for!"
If you have the job title "DevOps X", then you've missed the point of DevOps.
The entire thing was brought in to remove the silos between SysOps and Dev. If you have people in the "DevOps" role, then you've just created a new silo, which means you just have traditional SysOps with a new and fashionable, but less meaningful name.
Unfortunately, "Cultural Engineer" as a title has some pretty scary implications. So the DevOps Engineer is becoming a cultural engineer of the company with a better name.
Which of course is title inflation and title overloading. Now if they'd just overload the salaries, we'd be set.
I've seen this a few times in the thread and I don't get it. Nobody is saying there is a "Cultural Engineer" role, they are saying that there are operations roles and development roles, and that DevOps is not a role, but a culture where the people in both roles understand, help, and cooperate with each other.
A devops engineer should know how to apply devops to your organization before the other roles have become proficient. It's someone to ask vs. reading the Internet for things that might not apply to you.
I don't really care about titles so if calling someone a DevOps Engineer helps them and connects the right engineers to the right roles, great. Fighting against this use of "DevOps" would be like fighting against how the word "literally" is misused to mean "figuratively".
However for me, "DevOps" is more than just CI/CD and tech automation, it's about not working in silos and having empathy for others in the team (and the customer!). This is a culture and mindset that everyone in the team needs to have.
Because DevOps also include things like Agile, Lean, MVP, avoiding local optimisation, minimizing the amount of work in progress, and many other things.
DevOps is not just about infrastructure, you need buy in and a change of mindset (compared to waterfall) from the Devs, QAs, PMs and even sales / upper management.
This is really a company-wide change aka culture. So unless you are the one putting all of this in place, "DevOps engineer" doesn't really make sense. Automation engineer is what most "DevOps engineer" really are: automating as much as possible everything that touches the infra.
Completely agree. None of the current top comments even mention what DevOps actually does. I can't understand where the motivation is coming from to obscure it into some fuzzy "human" wankery.
It is actually the opposite. The whole thing started out as a movement to improve cooperation between development and operations ("fuzzy human wankery") and has been obscured to mean "an operations person who automates things", which didn't need a new name, because they already did that.
and, if you are going to bring in a person or persons to be that 'Devops specialist' and help your devops transformation, don't bother if you aren't going to empower them to facilitate real changes, or if you are going to put them in their own little silo and leave them to get on with just automating some manual stuff - that way you will have some automation that may or may not do what you want, but you won't achieve real transformation and reap the benefits thereof.
It is a role - not something that can be done by OPS or DEV or SEC or what have you.
Allow me to explain my 2 cents worth:
- With the amounts of tooling (CI/CD pipeline, infrastructure monitoring, infra. provisioning, config. management, etc.), you do not want your DEVs or your full-time admins focusing on this bit at-all. It is good if they have some know-how (or even if they do the initial groundwork) but if your DEVs or sys admins are focusing on what 'entails' as DevOps -- then Houston, we got a problem. Because this means that DEV or sys admin or what have you is not focusing 100% on their role.
- Why is 'DevOps' getting so much flak?? How about titles/roles such as 'scrum master' or 'Agile Practitioner' or 'Product Owner'?
- I am not a noob by any means (~10 yr IT experience) and I am past the stage of getting excited by a job title. However, the 'DevOps' title given to a 'DevOps' guy is a correct move. AND, believe-it-or-not, it is a IT specialization -- just like 'cloud' is. Just because everything is a code doesn't mean you don't need a dedicated person who will not only preach DevOps -- but will also 'implement' DevOps -- and that fella should be called 'DevOps' -- just like 'Agile Coach' ;-)
Would argue that you have a tools/service team. The fact that you failed to mention changing what devs and sysadmins consider when "focusing 100%" on their role indicates that what you are talking about isn't the concept of DevOps in the article.
Saying you have a DevOps team is like saying you have Lean team, a Theory of Constraints team, or a Just-in-time manufacturing team. DevOps doesn't say that devs shouldn't write code any more then Theory of Constraints says a guy that paints car hoods should be building brake pads. It is about the fact that they should work together to ensure the organization as a whole meets the goals. Not as isolated units that only prioritize their own goals.
I've seen the same flak about Agile and scrum roles for the same reason. People think they can buy the DevOps/Agile/whatever. Only difference with DevOps is rest is that it is getting more noise.
I'd have a contrary view which is that yes you DO want your engineers to understand dev and ops (and sec and change etc etc) and actively be able to own a service through its lifecycle.
Those engineers might have different focuses; so maybe building a data store, or an api, or a Jenkins system, or an artefact store....
If you leave your cicd tooling and toolchains to a team then you create a dependency, and for me a key part of devops is reducing dependencies.
The suggestion in your post is that by blurring the traditional roles you make those people less productive. And therefore the solution is to create a new role.
That's fine: but that's the old thinking (IMO) hanging around. The original intent of devops would definitely run counter to that :)
Yes yes, we get it, it's a mindset not a role. Now could we hire a DevOps guy to setup that CI/CD pipeline, ChatOps stuff, Kubernetes and dev/staging/prod flow so the coders can ship. ;)
The original thought was beautiful but the way I see it: DevOps is a branch where you specialise. After school people might learn front/back/db stuff and then specialise to "DevOps stuff" -- or then you have more sysops kind of people who start to venture in the land of automating their work. Both are needed imho.
I hate specialization, and I want the person who develop something to think about how that stuff will be deployed and kept online. In particular, some tradeoffs have to be made between the development side and the production side, and I want the same person to see both side and decide with a technical criterion (or just choose one option, and later reverse course and change it), not on a "it's not my problem" criterion.
There is too much. It's specialization out of necessity. I know enough "DevOps" to be dangerous, but in addition to having a deep knowledge of our product's codebase, the multiple features of our product, configurations, etc. there are not enough hours in the day to master (not simply be familiar with) Jenkins pipeline, Docker, AWS, Ansible, Zookeeper, etc, etc.
99% of the people who could claim to have knowledge of all the above in their organization either have a very shallow (read: not useful) knowledge, OR their product itself is simply the gluing together of various DevOps technologies listed, so there is little more to know outside that.
it is definitely possible to have fairly deep level of knowledge at all layers it just takes effort.
That said, the larger the project, the more the need for specialization and the less possible it is for someone to keep up with all changes and be a domain expert.
I disagree. Maybe, there's a couple of guys/gals around that can go up and down the software and hardware stack and know where issues may lie when there's a problem, but I doubt there are many.
As a someone that's been in the tech industry a very long time, I came to the realization a long time ago that I couldn't be the expert at everything. Can I dabble in Oracle, Windows, Linux? Sure, would I want to bet my job on how much I know some of these products in the time of crisis, hell No!!!
My last job as an employee, I had to install AWS, develop some server software, develop some web facing software, develop some embedded software (with a bit of easy real time), get the freaking 2G to work, get the sensors to work properly, model the mechanical parts to install the embedded device on site, order all that crap (electronics and mechanics) from the providers, create the boxes, try to think about lightnings and how not to fry the customer's equipment. And get the sandcastle to stay put.
That's what happens when you work at startups, there is no plethora of employees, and a lot of work.
I do not know. We always had "DevOPS" aka sysadmin that knew C,perl and bunch of others. Then it became too much for a single person to know everything and we invented "more ops" and "more programmer". Now we have better tools albeit more complex and we have decided that we again can have this "DevOPS" person whatever that means.
But again we are facing this dilemma when there are literally thousands of different tools and ecosystems and you just simply cannot be an EXPERT or even ADVANCED in most of them.
We should clearly define what we expect from the role and the candidate. We can't waste other people's time by hiring for DevOPS when we clearly want a PowerShell programmer,windows administrator and database admin in a single package
DevOps is a cultural idea. It's supposed to be the collaboration between sysops and dev, in order to break down the traditional silos. Having a "devops" team or role creates a new silo, hence the reason it's BS to have it.
Doing development work, whether it is application development or infrastructure automation, is really hard if you are being interupted frequently.
Traditionally, you avoid interupting programmers by putting some kind of buisness analysis layer ontop of them. That might be a product owner or just telling everyone to communicate requirements through tickets.
But you can't build stuff without talking to eachother. So programmers working on similar stuff tend to chat eachother, sit near eachother. They chat in an informal way.
The trouble is that, for DevOps, often the developers are customers / users. If you chat to them the way you chat to the other DevOps, you'll never get any work done.
Infrastucture automation can happen 2 ways - either the developers take over the IT, or the IT take over that part of the development. IT support organisations have a very anti-customer culture to be honest. Combine that with the need to protect engineers from interuption and you get a very hard silo.
> But again we are facing this dilemma when there are literally thousands of different tools and ecosystems and you just simply cannot be an EXPERT or even ADVANCED in most of them.
Though there are only a handful of ideas and principles, and all of these
thousands of different tools are just reimplementations of some of them, and
usually not too fancy at that.
Of course you don't, but those nuances and specifics are usually superficial
and don't change how you approach your environment. You think there is that
much of a difference between running Puppet, CFEngine, or Chef?
They are not superficial and they definitely change how you approach your environment.
I have worked extensively with both CFEngine and Puppet and I can assure you that they have enough differences especially when you consider large heterogenous environments that in no way you can consider using one as a substitute for the other.
Yes, but this also doesn't mean that one person trained in one tool can't learn the other in fairly short order, provided they understand the core principles behind it and haven't been 'brainwashed' into thinking 'this is how things work'
And how these affect operator's workflow? That is, the ways an operator
interacts with the tools?
The facts that Java is garbage collected or that it runs on virtual machine
don't fundamentally change how programmers write code in it compared to e.g.
C++. It's the same type of tool ("paradigm"). Erlang, however, is
substantially different.
You are comparing apples to oranges (in my opinion of course).
I would say it's often hardly about just "operating" the machine. You need to know how the machine works how it can affect other machines and what can you do if parts of the machine break.
Not only that but also how many operators you can find, how many additional modules (libs) are there and how often you need to oil the wheels.
Sure the implementation of the known algorithms will be similar in Java and C++ but the overall usage and how it connects with everything (aka the workflow) will be very different.
> I would say it's often hardly about just "operating" the machine. You need to know how the machine works how it can affect other machines and what can you do if parts of the machine break.
So, in terms from programming analogy, you need how your runtime operates, how
it can affect neighbour processes/services, and what can you do when the
runtime breaks down. This still does not affect how you structure your code
(at least usually, when you don't work closely with the OS, which I bet is
majority of code in the wild).
> Not only that but also how many operators you can find, how many additional modules (libs) are there and how often you need to oil the wheels.
How does workforce availability affect the way you work with your system?
How does availability of libraries affect the way you work with your system,
apart from amount of code for you to write (libraries have to come from
somewhere)?
The architecture and strategies stay exactly the same. With this regard,
there's very little difference between CFEngine, Puppet, and Chef.
> Sure the implementation of the known algorithms will be similar in Java and C++ but the overall usage and how it connects with everything (aka the workflow) will be very different.
It will hardly affect how you think about the system and how you approach it.
It will only affect where you get your code from. Unless you think about
yourself as a guy who only puts together different libraries/modules and never
ever writes own code, then of course it's a new paradigm, but I don't think
you had this scenario in mind.
> So, in terms from programming analogy, you need how your runtime operates, how it can affect neighbour processes/services, and what can you do when the runtime breaks down. This still does not affect how you structure your code (at least usually, when you don't work closely with the OS, which I bet is majority of code in the wild).
Working with CFEngine's "DSL" v2 was very different from using Puppet mainly due to its limitations and lack of features. This really affects the way you structure your code.
There were / are many nuances in CFEngine that make it different from puppet. And this is also about availability and complexity that is what I meant.
> Working with CFEngine's "DSL" v2 was very different from using Puppet mainly due to its limitations and lack of features. This really affects the way you structure your code.
In this case it only affected where you put which part of the code, not how
you managed your servers with the tool.
Note that the analogy with programming languages is <how you manage servers>
=~ <how you structure code>, not <how you structure code> =~ <how you
structure code>, what you're trying to put into use here. For judging the
approach, it's irrelevant whether you write a shell script or use somebody's
Ruby plugin to detect mounted filesystems or something like that. It doesn't
change fundamentally how you structure maintenance of your environment.
> There were / are many nuances in CFEngine that make it different from puppet.
Of course, starting with their languages' syntax. Though both use the same
underlying ideas, even if they call them differently, and both operate in
similar way. If you know how one works, it's a knowledge that's easy to
transfer to the other.
Everywhere you see the word "devops" replace it with the word "empathy".
That was the true meaning of DevOps - fostering empathy and communication between different groups, and to stop the siloed "throw code over the wall to Ops" mentality that existed.
By having a "DevOp's Engineer" you have admitted you have failed at DevOps, because you have created a new silo.
A good team will have a combination of Operations Engineers, Software Engineers, Automation Engineers and Test Engineers all of whom work together, and collaborate on tooling, processes and deployments, who also share in the rewards for a projects success, and share in the responsibility for when something goes wrong.
DevOps is not a team created to "provide DevOps services to R&D" - which is the basic job description for far to many DevOps jobs postings.
It is also not just handy R&D pagers, and expecting them to manage a system.
Of course all of this is talking about the initial spirit of DevOps, not the current "Buy my Book", consultant driven mess that is now associated with the word.
That was the true meaning of Empathy - fostering empathy and communication between different groups, and to stop the siloed "throw code over the wall to empathy" mentality that existed.
By having a “Empathy’s Engineer" you have admitted you have failed at Empathy, because you have created a new silo.
A good team will have a combination of Operations Engineers, Software Engineers, Automation Engineers and Test Engineers all of whom work together, and collaborate on tooling, processes and deployments, who also share in the rewards for a projects success, and share in the responsibility for when something goes wrong.
Empathy is not a team created to "provide Empathy services to R&D" - which is the basic job description for far too many Empathy jobs postings.
It is also not just handy R&D pagers, and expecting them to manage a system.
Of course all of this is talking about the initial spirit of Empathy, not the current "Buy my Book", consultant driven mess that is now associated with the word.
Everywhere you see the word "devops" replace it with the word "empathy".
That's an interesting idea and might empower you to write truly heartfelt downtime explanations to your customers. I wonder if it could be broadened to replacing "Accounts Payable" and "Payroll" with "empathy".
It confuses me when I read the Agile Manifesto and then look at Scrum, Lean, Kanban, XP, etc.
On the one hand you have a manifesto advocating self-organising teams citing individuals over processes, while all those methodologies claim to be 'agile', but dictate your processes and how you should organise.
To me, they seem incompatible. It's like several of the original signatories have developed doublethink.
The one that really makes my mind boggle is that you can be a certified Scrum Master. To become the guy who tells everyone how to work? Is that not the very antithesis of the agile manifesto?
Yes, and no. Not creating a Devops position is just leaving the existing silos. The best way to tear them down is to put the Ops(DevOps) under the Dev manager. It reduces the code throwing, and gets rid of team politics.
As a DevOps consultant, I think it can actually be both. Culture is central to DevOps but someone has to lead. Sometimes this can be the CTO but often it helps to have a dedicated advocate, especially within larger orgs/teams.
Essentially it boils down to semantics. Often DevOps engineers are one part evangelist and one part automation engineer. The software industry has just chosen to call those people "DevOps".
1. It's a culture where dev and ops work together (as per TFA);
2. It is the application of software engineering practises (such as version control and test automation) to ops.
I prefer the second definition because it's the only one that's got anything to do with new developments in infrastructure-as-code and isn't just a new name for what competent teams were always doing anyway.
I've been on the sysadmin/ops end, trying to teach devs how to apply their software engineering to ops. It's not a short ramp, unfortunately; the various ops stacks now are as broad and deep as software development stacks. Worse yet, it is not Yet Another Imperative Programming Language for them to pick up. Even declarative configuration management systems embed a lot of hidden, assumed sysadmin/ops experience/skills that many devs don't arrive on the scene with, and in the vast majority of cases it was like training someone from scratch to become a sysadmin, and then start devops training. Add in cross-platform ops, and it gets more challenging. Add in enterprise demands and integration points, and it becomes a multi-year effort.
More unfortunately, it is also a long ramp up time to bring sysadmins who don't know programming up to speed on software development. I'd like to know what others are doing to quickly build a devops team if they are doing that, other than "find highly-skilled senior sysadmins who already know software development", because that's my go-to approach for the time being, as it is way faster to teach someone like that Ansible if they already use Chef, for example, than pick up sysadmins or devs and cross-train them into the other field.
I'm good at writing code, developing libraries, architecture, etc. I'm not very good at operations - mostly because I don't enjoy it and haven't spent much time applying it.
Not everyone is the pitcher, the catcher, the first baseman, etc. We specialize because that's how humans work best together. Everyone should not need to know how to do everything.
Developers must work with operations - of course. The way and the what developers build must be able to be deployed in an agile manner. But, that does not mean that developers must become operations engineers as well. Leave that to the specialists in that role.
you mean the ops guys. and yeah it's just an easy way to take nerds and make them do more stuff so you can save that extra 85-100k salary. More profit for the shareholders, what's not to love.
> and isn't just a new name for what competent teams were always doing anyway
That's interesting. I see that the opposite way: weren't competent ops teams already doing the things you listed in #2? Does keeping up with new developments in operations tooling need a new name? Shouldn't that just be part of doing ops well?
A big part of the problem is that DevOps are not really about new technologies but just a new word the same bottom up management theory that also gave rise to buzzwords like LEAN.
In essence it's a shift from thinking in terms of grand strategic plans drawn up by the high management to be mindlessly implemented by the "replaceable" employees, to thinking in terms of reacting to problems as seen from the bottom of the org chart.
The problems with buy in is that DevOps kills a lot of the prestige difference between different teams/silo's as it's basically about realizing that the no developer or systems architect have enough knowledge about the "real world" problem they are trying to solve to design/write anything in isolation.
Or to put it differently DevOps means that the implementation and development teams owe operations a steak dinner if/when the system tumbles over and operations gets the dreaded 3am pager call, due to an untested edge case, not that the two roles merge into one.
Here's my point of view as someone who's title currently is "Senior DevOps Engineer", and used to be a software engineer for many years:
DevOps has multiple meanings - in some (usually smaller) companies it can be thought of as a philosophy in that all developers are responsible for managing production infrastructure and their deployments. As a company scales, and the codebase grows, people have to start specializing. It's just not feasible for everyone to have perfectly overlapping skillsets and be completely replaceable by each other. It happens with frontend/backend, and it happens with "DevOps". I focus on our production infrastructure, making sure deployments are smooth, our database has backups, etc. That doesn't mean that other developers are completely detached from our production environment - if there's an issue in production multiple parties are involved in working through it and fixing.
I personally interchange the title DevOps Engineer with Site Reliability Engineer, Infrastructure Engineer, Ops/SysOps Engineer, etc. Different companies have different titles, but I'd wager that the majority of those positions tend to perform work of a similar nature. Depending on the size of the company, some may specialize even more (i.e. a large company may have SRE to focus on production/monitoring, and someone else to focus on deployments). It definitely varies company to company, but by and large I think it can all be grouped under "software operations", which itself is a subset of software engineering.
While I agree that DevOps needs to be a mindset/culture to be of any benefit beside ticking buzzwords checkboxes - but I really don't see any harm in giving someone a title containing the word DevOps.
Someone who can do the proselytism, take initiatives to drive the adaption and at the same time be the go to guy for the technical knowledge needed for any new processes/tools.
We all agree that the old waterfall inspired way of working when the developers work on their own, throws a bunch of new code down a hatch for the ops guys to deploy and they both blame the other team when something don't work is less than desirable.
Maybe there are some startups that have the devops mindset already from the beginning and a few old companies that can get there on there own, but I think many need some help to get there. Perhaps they could hire someone to get them started?
My official title is DevOps, but my coworkers know that I'm just the most cloud-oriented developer on the team. In that regard I also regularly push code into our products, with an insistence that I am not simply the guy that does AWS/Jenkins/Ansible work.
So, in one respect I entirely agree that DevOps is just another developer, at least the way I practice it. That being said, it's naive to think that DevOps isn't a specialization, or that everyone writing your code is going to be equally knowledgable on a product landscape as complex as AWS.
Ultimately where you fall on the spectrum of developer and ops is company specific, even project specific, and to try and make some generalizable dogma on the issue is a waste of time.
Related, I've found that the term devops is too poisoned to be used effectively when hiring. It brings out charlatans in far greater numbers than anything else at the moment. Garden variety technician level folks who self inflate their titles and roles to try to jump from 60k to 100k overnight.
It's been frustrating as they're able to get things syntactically correct for the most part that they get through early phone screening but absolutely fall apart once they reach someone with technical expertise, wasting everyone's times.
Devops is less about knowing the syntax of something like Terraform and more about understanding, communicating, and implementing 'devops' best practices across an organization. The early phone screen should be an easy place to weed out people.
I do agree that devops is the hot new title so people are jumping on it to try and make more money.
I mean syntax of the conversation. They're basically gaming the initial phone screenings and making us waste engineer time discovering that they don't really know anything.
It's not exactly easy to get good filtering for this with first level screening but it's what we've had to do
I've been browsing around job boards lately and see a lot of "DevOps Engineer" positions, but the descriptions of each vary widely. Some of them appear to be "real" DevOps roles, that is they sit somewhere between the traditional dev and ops roles and the basic mission is to get code out to users fast. But then many of them are simply rebranded systems engineers, while others are dev roles where managing infrastructure is part of the job, while other job description have read like very basic sysadmin job descriptions.
> Google has been doing this for ages without calling it Devops
Are you of the opinion that DevOps, like Agile, is a buzzword and money making opportunity? I had a brief stint in a DevOps-ish role and that was my conclusion.
100% from the two companies I've seen implementing it.
All it means in practice is "fuck, our IT guys don't understand this docker thing. We need a software guy to write up stuff to automate all this"
I looked at a couple Devops position and saw the same. Basically "we need a software engineer with networking skills".
IT jobs are quickly vanishing. All the stuff that doesn't require software development has been automated away besides things anyone can do like wire some machines to a switch.
Everything is either really easy or quite hard and it's hard to hire a guy that knows software at an IT salary. Now we have Devops, so that execs can justify hiring software engineers to maintain the network.
I agree its a culture, but there is a role to be played, and in all the cases where I have moved teams down the DevOps path it has always started with specialists who can help the team out as they transition.
Then there is the operations part, no sane person would say since we do "DevOps" and "Continuous Delivery" now, so lets give ever member of the team full admin on AWS. Even in a _very_ mature organization you have snowflakes. If your DevOps team is doing its job, more and more of the workflow is being shifted all the time to the product teams and less and less is being owned by them. Then once they have delivery fully migrated, they move on to be your SRE's which is something I think most teams who have devops people will never get to.
I think there is more to it than too cheap. I think it is not enough money and not enough need. If the budget is $150k and it is SV, you aren't going to hire a team.
Another problem is I think Google and like companies have been vacuuming up the best of the best sysops and DBAs. This has had the effect of making them hard to find, and even if you can, you can't afford them. Then you see the conquences of companies trying to figure out an alternative plan. The results are Devops and NoOps.
Here's a thought: skilled devops engineers are indistinguishable from skilled software engineers, and vice versa.
Who can be specialized anymore? Isn't devops just the compression of skill sets that we've been witnessing for the last two decades?
As a developer, you now must be able to deploy, to write and maintain unit tests, and possess the knowledge to debug code deep on the system and network layer, and write code that follows up-to-date security practices.
I used to repeat this statement, but a few years of the current pattern, I disagree.
(I'm going to generalize for UNIX systems, apologies in advance)
Your traditional systems engineers are well versed in the ways of UNIX, but that does not automatically qualify them to wear the DevOps hat.
And vice versa. Systems and DevOps are two different sides of the house.
There is an entire tooling set that someone with the DevOps attribute needs to be aware of (CI/CD, tools that enable code review, etc), how they tie into the tools that the developers use, and how to troubleshoot when these things break down.
You, Systems Engineer, will be called when someone can't run "npm install" and an ERRNO message scrolls down the screen. While it could be something as innocent as a missing header file from an external dependency, it's still on you to rapidly troubleshoot and resolve.
You also need someone with UNIX and networking infrastructure chops. Someone who is familiar with a how to generate and read a packet capture, or maybe even use strace.
You, DevOps, will be asked why does it take forever-and-a-half to run 'ls' in a directory containing 10,000 files. Scratch that, you will asked to make it faster.
It's unreasonable to ask either of these folks to stretch and cover the other role, it results in a lot of resentment in how they're expected to do "two jobs" as well as complaints from developers that the DevOps/Systems Engineer's priorities and timelines don't line up with their own.
Worse, you'll see abominations (a relative viewpoint) because everyone knows their tools the best. To a person with a hammer, everything looks like a nail.
I've seen automated DNS failover triggered via Jenkins, and I've seen automated code deployment triggered via cron on a 5 minute schedule.
DevOps is more than automation, it's about enabling developers to rapidly iterate in a sensible way that doesn't cause too much wake around a growing startup, an already functioning business, or a fast moving behemoth.
It's about enabling operations to rapidly provision new applications, monitor existing applications, properly perform capacity planning, and create repeatable infrastructure ...repeatedly.
And it's about both developers and operations coming together to onboard new employees into the environment in a short timeframe, so they can become productive.
This is all not to say that DevOps and traditional Systems Engineers + Network Engineers should live on different teams. Ideally, they're all on the same team, with different roles, and plenty of communication and cross-training.
It looks like you're just talking about a good, old-fashioned, well-built operations team.
In DevOps there is no "I can't run npm" unless the new guy is building his first personal env.
In DevOps ERRNO scrolling down the screen means your kit is broken and you need to rebuild it to a known, working state, yourself (and crap like that certainly wouldn't be making it into production).
In DevOps the entire organization is on board. Everyone has a role even if it's just feedback or a ring-ring fielding phone jockey and they'll know why and they'll know it's an important contribution because every lost piece of information affects delivery and quality.
If you're automating, integrating, designing, orchestrating, trouble-shooting or just about any other ing-ing you can think of than you're an engineer and you can front that with whatever you like be it systems, integration, quality-assurance, software, application, automation, monitoring, cloud, reliability, hardware, database, support, etc. etc. and even devops, if you like.
The difference is whether the organization you're doing all of your "engineering" for has accepted and/or is applying the DevOps "method".
I've perused some "DevOps Engineer" postings and a good amount of them look like garbage idolization of this or that toolset combined with looking for someone they can beat on until their infrastructure works like they think they'd like it to.
DevOps ain't infrastructure and it certainly isn't ace dev guy running the prod kit and no, it's not about "enabling" developers either. It's an operation comprising the entire cycle of delivery with expectations and requirements from all members.
Given the case of some new startup and they're on the hunt and poised to ramp then what they're really looking for is a savvy engineer (or 12) that can ing-ing all. day. long.
Sadly, because of business overlays and buzz-wordery and agile styles and .. certain software stacks, a more than fair share of time gets sucked right out of the actual developing and operating parts, especially if there's any palatable resistance to acceptance but, alas, it really is just another thing fronting ing.
I was thinkging about that Mike Dilworth quote: "DevOps is a culture, not a role! The whole company needs to be doing DevOps for it to work."
It's one of those kinds of statements that at first glance, looks really insightful and invites the head nodding up & down in agreement.
However, thinking about it some more, that type of quote can be applied to anything.
"Compliance is a culture not a role. Everyone from the CEO to the janitor should be aware of legal compliance. Having a compliance department is wrong."
"Security is a culture not a role. Every employee should create quality passwords and follow proper encryption procedures, ..."
"Marketing is a culture not a role...", "Design is a culture not a role...", "" etc, etc. Every employee should know/do everything. If you have a separate marketing team and design team, your company has failed.
Yes, I can see where one ideal of devops is to "break down silos" and therefore, having a set of programmers who specialize in that is "recreating the silo" which is a contradiction. On the other hand, I can see the opposite view: if a company wholeheartedly embraces "devops", they will acknowledge that by having a team that focuses on devops. If you're a Google paying a programmer like Jeff Dean $500k+ a year, you really don't want him working on Chef/Puppet scripts instead of Tensorflow research. At a more advanced level of skills above Ansible/Docker automation is the SRE (Site Reliabity Engineers) which has a lot of overlap[1] with Devops focused engineers.
A specialization in devops seems inevitable especially for larger companies that exceed 10 people.
In short, I'm not sure why "devops" in particular has to rigidly adhere to Heinlein's "specialization is for insects".[2]
As for other comments saying it's "title inflation", I'm confused as to the source of that sentiment. The "devops engineer" doesn't seem like a glamorous tile similar to "I'm CEO, bitch"[3]. There are no "50 Shades of Grey" movies where the female lead gets weak at the knees at the mere mention of "devops programmer". The "sanitation engineer" euphemism of "garbageman" I understand as inflation but "devops engineer" seems like a sideways or neutral label. I'm mystified by the outrage over this programming label.
I think the DevOps is a culture statement is just as misleading to folks as the DevOps is automation/tooling.
If you look at the early conversations, it was a discussion that outlined a management paradigm that borrowed heavily from Theory of Constraints in the manufacturing world. The discussion lead to a consensus among those involved that defined core values of culture, automation, measuring, and sharing (CAMS).
To many people, DevOps means that paradigm. The examples you list (security, marketing, etc) are not paradigms composed of several values. They are business functions as infrastructure automation might be. We don't call the accounting team the "cash-flow" team cause it is a paradigm. Not a function.
While I don't think it is worth falling on your sword over, the fact that folks care about the wording is good. It means they are thinking about more then just the function, but also the underlying paradigms.
i am not sure compliance, security and marketing etc. are cultural. I am pretty sure they are well defined skills/disciplines. The reasons why i believe devops is a cultural thing is because its not just about the technology, it is also about the process of executing work. you can use the tools and automate the shit out of everything, but this isnt devops. but if, as a full stack autonomous team, you are experimenting your way towards a target condition, (think lean), to elevate constraints, then you are working in a devops way. You cant work like this unless the organisational processes support it. We can then throw in all the great modern tools, software engineering methodologies, shift left, involve everyone in the organisation and then it does become a cultural movement. There is still a place for specialisation, but good specialists know how to work in loosely coupled way yet, function in an interdependent world. So,to summarise. its about the people, processes and technology and these things combined make it cultural.
I'll take your outrage comment has hyperbole, I've yet to see anyone that upset by it. Anyway, the frustration, I think, is it's just a new name applied to a role that always existed.
Linux basement monkey/Sysadmin supporting a dev Team and soon clients, here!
I dont care about nomenclature. I chose my weapons, and i can do fine with them; jenkins, artifactory, puppet, docker, vagrant and ammonite. All strung together using scala and bash scripts. Github MDs for docs and svn because it was there and the lads and ladies here like it.
Anyone can choose a stack. They just have to get good at it.
However, this article does serve as a good example of one of the big problems with most DevOps/SRE evangelism: It's usually coming from the Dev- perspective, rather than the -Ops perspective (or better, a balanced perspective). In particular, this focuses mostly on getting code to users, and speeding up that process. While that's certainly important, it leaves out any discussion of:
Successful DevOps cultures must address these things just as much as they address code feedback loops, time-to-production, rollback strategies, code pipeline automation, etc.