As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good.
What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system.
The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse.
It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-).
About ten years ago I remember talking to some old timers who were surprised that a lot of the younger sysadmins no longer even did any perl progamming, but rather did things like program Cisco rounters and set up Oracle servers.
At Google we don't have sysadmins at all, but rather "Site Reliability Engineers", and they are expected to be able to answer some fairly deep C/C++ data structure and algorithms questions. All of those skills are necessary to run a Google-sized infrastructure, and it's quite a far cry from the sysadmins that don't even write perl scripts any more -- and yet feel perfectly free to call themselves "sysadmins".
When I first started, I both wore a pager and wrote C programs --- including the 'ninit' program that was described in the Unix haters handbook. The fact that I was one of the people who might be woken up at 3am when the mail server melted down due to named core dumping informed how I wrote my C programs (i.e., very defensively), and is a prime example of the dev/ops concept.
So we agree on the definition of what a sysadmin, systems engineers, devops or whatever people want to call it should be. A sysadmin must program, that's a given. When did we forget that exactly ?
But why are so many devs shooting at the sysadmins these days ? I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. And they are numerous, willingly ignoring even the most basic performance issues and vomiting thousands of lines of codes, hundreds of tons of libraries, to simple problems.
It seems that devs are frustrated to not be able to push their random app to the web in 5 minutes, and throw their love to PaaS like an apple geek discovering the new iphone, blaming the sysadmins for not providing the same thing at their own company without even understanding (or trying to understand) how operating systems work, and how their own code should be built.
The real issue here is that when devs and sysadmins DON'T work together, and ignore each others problems, constantly blaming each other, then the result sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they drink their coffees together in the morning, you don't see that kind of problem. They built good, reliable, scalable infrastructure, whether it's in the cloud or in the basement of their house.
I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs.
They should be. At my current employer we're pretty good about this, but in previous misadventures I've run into a lot of people who don't even understand HTTP or how a web server works (if I have to explain thread-per-connection to a guy who's been doing web development for five years, there's a problem!).
I used to think like that. But then, doing web development myself on the side I've realized no, they don't. Languages and frameworks these days pretty well abstract the underlying technology. Developers end up writing some pretty amazing stuff. It's up to the sysadmin to weigh in on what changes are necessary to make it work well enough to succeed. A seasoned developer will know more, because they've been through the wringer a few times. The new guys, not so much.
Remember, your developer spends their time making it work on a dev box, or even their own laptop. Performance testing and such is an art that if they had time to learn it... well I guess they'd be a sysadmin.
I am that new-guy web developer (or I was a year ago, anyway, I graduated college in 2010). I am the developer working on a dev box, and I do do performance testing. I knew it then, because it's important. Web frameworks do not meaningfully abstract the underlying workings of a web server; hell, the inability to do so is why C# 5 has the await keyword (to allow for better asynchronous IO) and why Servlet 3 has its own async stuff (not to mention Play Framework, which does all manner of shenanigans under-the-hood!).
If you actually give a shit about web development beyond a paycheck, this is stuff you just gotta know. If you don't understand the system, your assumptions and assertions are so much more likely to be untenable that it becomes dangerous.
The below questions wouldn't fly at my current employer due to a lot of legacy concerns, but my own list of interview questions would include a number of stack questions in addition to programming questions:
-Explain asynchronous I/O to me.
-How do you have to structure your code to leverage asynchronous I/O? (Alternately phrased: what happens if you write code that depends on synchronous I/O operations within an asynchronous I/O web framework?)
-You've got 9 memcached servers with a cache hit rate of 80% resulting in a database load of 80%. What happens if you naively add another memcached server to your cluster? (a question our VP of engineering actually does ask, and I stole it 'cause it's great)
You don't have to be a hardcore nginx source code diver to understand your stack. You just have to understand the principles behind it well enough to make good decisions. A sysadmin's job isn't to audit your code for your terrible understanding of the model on which your stack is predicated.
I think you may have taken my comment in a harsher context than intended.
First, I don't think it's good for a developer to have all the crap in their head that I do in mine. They'd be spending way more time pre-optmizing code than they would just getting stuff done.
A good developer will write a well thought out TDO that infrastructure and business units can provide feedback to.
Your questions about asynchronous i/o and memcached are great, and if memcached was a requirement for a sysadmin at a position it would be a good question to make sure they understand the software.
As a sysadmin though, you're also balancing
- Can the new system exist on an existing server/chroot/virtual machine
- What monitoring is necessary for the application afterwards
- Does the application meet all the standard requirements for database access
... out PFC is over 100 steps, this is just some of the bigger ones
- Can the sysadmin support the system after go live? The developers job is to create it, the sysadmins job is to keep it running long term.
Also, you are correct. The sysadmin shouldn't be auditing your code. The sysadmin should have a stack in place that they know works, and be able to help you do one of 2 things:
- Give advice on how your system can best fit in that stack
- Help you change the stack to best fit your application
Which direction you go on those two choices depends on the nature of the application, how mission critical is, required timelines for launch, budget and more.
Again--I am that developer, and I find that keeping "all that crap" in my head is extremely easy and makes me write better code. It wasn't straightforward to learn, but it made me better at doing my job. A question about memcached isn't a "sysadmin" question, it's a developer question to demonstrate that they understand what memcached (or whatever) does, its limitations, and its knock-on effects. These are things that I am entirely and without reservation convinced that a developer must know, and know intuitively, to be worth trusting with code. We have to write the code that uses these things--we'd better know how it works! That a sysadmin should know it as well is of course true--but a sysadmin should be a developer, too, to be able to make correct decisions in everything you've said above.
I don't say this stuff just for show--I've been there and done that, and it's stuff I've done to become what I consider an acceptable developer. I can, and have, set up a three-tier fully-enterprise-up-the-ass application infrastructure on my own (I have a couple machines on a VLAN at home as a VM farm) in order to learn this stuff. A developer who can't work the stack is a developer I don't want touching my code, because they're a developer that doesn't understand what they're doing actually does. Abstractions are great, but every abstraction leaks--they should be treated as productivity enhancers, not as insulation from knowing what you're doing.
Nobody can know everything about everything (there's a guy here at my current job who has forgotten more about the JVM than I've ever learned), but you gotta at least be able to talk about it. If a developer can't explain to me high-level concepts like how your web server handles threading, I don't even know what to say. This stuff is just so integral it's not even funny. You just gotta know it. Specialization is for insects.
.
EDIT: I think you and I are coming at this with very different philosophies. At my current employer, we don't have a PFC. We don't have TDOs. We have two live-site guys and a few operations engineers who are able to jump into our codebase (one is probably the most knowledgeable developers at the company who wanted a change). We have developers who can be trusted to make good decisions, and we have senior devs and managers who are very well equipped to analyze the plans and ideas that come from junior devs to say "hey, that won't work." But to be able to trust developers, they've got to understand how stuff works. (We don't ensure full-stack knowledge as well as I'd personally like, but we do a pretty good job of it--at the least, there are no "front-end engineers" who can't sling Java and no Java guys who can't write HTML/CSS/JS if a project comes along that requires it.)
probably because at nearly every place I've worked at the sysadmin/ops group was the #1 obstacle to getting things done. Sure some of it was "saving you from yourself", but a lot of was pure obstinatance for no other reason than I think sysadmins just cant turn it off anymore... much as your mom says "dont frown your face might get stuck", it appears a lot of people (not just sysadmins, developers too - you know the one, in the corner who is old and surly, been there forever, and hasnt been promoted in about as long).
The deck is stacked against sysadmins - developers are the people who deliver business value to leaders, and sysadmins are the ones who cost money and say 'no'. When you are on a tight timetable, the person who says "sure" is the one who gets the high fives... assuming things work out, of course.
>sysadmin/ops group was the #1 obstacle to getting things done.
This happens when the company discovers that when the system goes down they stop making money. If you've managed to design and build a system to be very resilient and reliable on the chosen platform and has proved to need minimal maintenance, then chances are good you have a very small ops team who doesn't get in the way much.
I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs
There are plenty of DBAs complaining about the low knowledge level of a lot of devs, e.g. using an ORM instead of writing SQL and wondering why the app runs like a dog.
> It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-).
I would say, 25 years later, the need to fix C bugs is far less relevant. That said, I'd say around 50% of the system administrators I've worked since moving into the Unix world know C and can at least read the source code of the operating system they're supporting. Even if there's not much need to write anything or even submit bugfixes anymore. Also, the number of unix admins I know who don't know a scripting language well enough to write a reasonably-sized, useful tool is vanishingly small, and those few have been very solid with some other technology(network, storage, etc.) and valuable anyway.
"System Administrator" is a broad term, just like "Developer." I'm not sure google has either, right? If we're going by fancy titles, Google has "Software Engineers." I'm not going to criticize Google's culture but the fact is if I say "Software Developer" you know what I'm talking about and from context should be able to tell the difference between Linus Torvalds and a guy who creates wordpress sites for his friends.
Sorry, even there, you have sysadmins who just sit there and run through playbooks and never improve anything. This is how you have teams with 70 pages in a 9 hour daytime shift as a matter of course.
There are pure SAs around, and I don't just mean the caste system. The entire sysops organization was just one example. They may have been rolled up into another organization but the people still exist.
I worked as an intern for the SQL server project. If you can explain accurately how to optimally structure SQL queries, you can probably write database code yourself.
> If you can explain accurately how to optimally structure SQL queries, you can probably write database code yourself.
Yes, probably. That doesn't mean your role on the project, in the company, or in your job description isn't "Database Administrator." Or operations engineer or reliability engineer or whatever fancy name you want to call the fact that you are more responsible for running the system than building it.
A more realistic example, though, is that a DBA and developer collaborate; the DBA collects and analyzes data from the production system and shares this information along with recommendations to the developer, who is ultimately responsible for restructuring the query.
In a very small team there's probably no need for these roles to be divided.
Sure, but I think that in this case the difference between your view of an operations engineer and a developer are fairly minimal in terms of skill. They're both development roles.
When I was interning at Google I noticed that these roles were differentiated as SRE and SWE. I didn't notice a lot of competency difference, just different application areas of the same competence. I think many SREs and SWEs could probably switch roles successfully.
That's more or less the point. System administration vs. software development is as much about priorities, mindset, and day to day environment than technical competence. This is true at all levels, whether you are pushing boundaries Google or supporting internal corporate users.
I never had a google internship or anything similar, and my first technical job out of college involved supporting an internally developed, browser-based ActiveX/.asp/vbscript tool for viewing and annotating loan documents. For a long time we on the systems team were unable to upgrade IE on user workstations because of a critical, easily reproducible bug in the document viewer tool. Eventually, I managed to get access to a snapshot of their codebase and tracked down the bug in a few hours (despite never having written a line of .asp or vbscript in my life and not having access to any of the documentation for the 3rd party libraries and activex components). The devs, of course, didn't listen to me anyway, despite my basically handing a solution to them. The bug remained for another year until that section of code was incidentally re-written to support some other feature.
The next company I worked for, a web company, had a much better relationship between development and operations. It might have helped that we had a lot of Indian developers, who didn't seem to have the same prejudices that western devs do: that system administrators are people who just aren't skilled enough to do development.
Programming languages are simply syntactic representations of predefined concepts. For the most part, if you can understand the concept the programming language isn't hard to pick up. Hell, I taught myself Python when I was 9 years old -- it wasn't the syntax of the programming language that was hard or the reason why I wasn't instantly a good developer. I needed (and still need) the real world experience of understanding the complexities of real software in order to develop real software. Anyone who understands these complexities and is only hampered by knowing the syntax of a programming language is underselling themselves -- with only a little more hard work they should be able to easily begin a development career.
As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good.
What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system.
The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse.
Ease of use != flexibility.