With any non-trivial assortment of servers, you need a dedicated sysadmin. The security issues alone should dictate this, but your developers don't need to be worrying about sysadmin stuff. They need to be coding.
I totally agree with proper deployment procedures. But having them doesn't eliminate the sysadmin at all. Sysadmin should be deploying standard packages that are necessary for the system, along with updates for them. Developers should be deploying in-house code, and updates for that.
I worked at a company that hired a sysadmin too early. When he quit, I ended up doing his duties and mine, too. Of course, I automated 99% of it and everyone was happy. That was really early in the life of the company.
Later, when it came necessary to hire one, they didn't. They'd expect a developer to deal with the issues, and were always dissatisfied with the time it took, and the lackluster results. Eventually it got so bad they hired a dedicated sysadmin again. Now, unlike the past, they actually needed one and he was busy all the time. Developers stopped being unproductive for weeks at a time, working on projects they didn't understand and couldn't complete satisfactorily.
For a startup, yes, skip the sysadmin at first. But once you have a complex system, hire one. It'll save your company.
+1 - you can automate deployment all you want, but when something goes wrong with your production site (and it inevitably will), do you want to pull your top developers off of whatever new features they are working on and ask them to troubleshoot and fix production?
Sysadmins will always be a necessary evil when you are dealing with hundreds of servers, physical, virtual, or cloud. The tools and process will change. We have gone from physical Unix boxes (Sun, HP, etc) to virtual Linux boxes, to cloud servers, but sysadmins are still doing the heavy lifting of designing and building highly scalable systems.
I would say that sysadmins should look at moving into systems architecture roles, since one sysadmin will be able to manage hundreds of systems rather than a few dozen in the future. You really need architecture, networking, and performance tuning experience in order to succeed when you are managing more than a handful of systems.
People on this thread seem to be objecting to the idea that agile companies should avoid hiring people who manage their systems. But that's not what he's saying!
My read is that he's saying that the days of the "system administrator" with a job description that begins and ends with tar flags and Apache vhost configuration are numbered.
He's not saying sysadmins are being replaced by Puppet and Chef; he's saying they're being replaced by a new kind of sysadmin that can be expected to extend Puppet and Chef.
This seems closely related to a point Patrick McKenzie makes all the time, that being the difference between a "programmer" and "someone who provides business value by means of applying specialized skills with programming languages". Don't be a "programmer".
As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good.
What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system.
The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse.
It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-).
About ten years ago I remember talking to some old timers who were surprised that a lot of the younger sysadmins no longer even did any perl progamming, but rather did things like program Cisco rounters and set up Oracle servers.
At Google we don't have sysadmins at all, but rather "Site Reliability Engineers", and they are expected to be able to answer some fairly deep C/C++ data structure and algorithms questions. All of those skills are necessary to run a Google-sized infrastructure, and it's quite a far cry from the sysadmins that don't even write perl scripts any more -- and yet feel perfectly free to call themselves "sysadmins".
When I first started, I both wore a pager and wrote C programs --- including the 'ninit' program that was described in the Unix haters handbook. The fact that I was one of the people who might be woken up at 3am when the mail server melted down due to named core dumping informed how I wrote my C programs (i.e., very defensively), and is a prime example of the dev/ops concept.
So we agree on the definition of what a sysadmin, systems engineers, devops or whatever people want to call it should be. A sysadmin must program, that's a given. When did we forget that exactly ?
But why are so many devs shooting at the sysadmins these days ? I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. And they are numerous, willingly ignoring even the most basic performance issues and vomiting thousands of lines of codes, hundreds of tons of libraries, to simple problems.
It seems that devs are frustrated to not be able to push their random app to the web in 5 minutes, and throw their love to PaaS like an apple geek discovering the new iphone, blaming the sysadmins for not providing the same thing at their own company without even understanding (or trying to understand) how operating systems work, and how their own code should be built.
The real issue here is that when devs and sysadmins DON'T work together, and ignore each others problems, constantly blaming each other, then the result sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they drink their coffees together in the morning, you don't see that kind of problem. They built good, reliable, scalable infrastructure, whether it's in the cloud or in the basement of their house.
I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs.
They should be. At my current employer we're pretty good about this, but in previous misadventures I've run into a lot of people who don't even understand HTTP or how a web server works (if I have to explain thread-per-connection to a guy who's been doing web development for five years, there's a problem!).
I used to think like that. But then, doing web development myself on the side I've realized no, they don't. Languages and frameworks these days pretty well abstract the underlying technology. Developers end up writing some pretty amazing stuff. It's up to the sysadmin to weigh in on what changes are necessary to make it work well enough to succeed. A seasoned developer will know more, because they've been through the wringer a few times. The new guys, not so much.
Remember, your developer spends their time making it work on a dev box, or even their own laptop. Performance testing and such is an art that if they had time to learn it... well I guess they'd be a sysadmin.
I am that new-guy web developer (or I was a year ago, anyway, I graduated college in 2010). I am the developer working on a dev box, and I do do performance testing. I knew it then, because it's important. Web frameworks do not meaningfully abstract the underlying workings of a web server; hell, the inability to do so is why C# 5 has the await keyword (to allow for better asynchronous IO) and why Servlet 3 has its own async stuff (not to mention Play Framework, which does all manner of shenanigans under-the-hood!).
If you actually give a shit about web development beyond a paycheck, this is stuff you just gotta know. If you don't understand the system, your assumptions and assertions are so much more likely to be untenable that it becomes dangerous.
The below questions wouldn't fly at my current employer due to a lot of legacy concerns, but my own list of interview questions would include a number of stack questions in addition to programming questions:
-Explain asynchronous I/O to me.
-How do you have to structure your code to leverage asynchronous I/O? (Alternately phrased: what happens if you write code that depends on synchronous I/O operations within an asynchronous I/O web framework?)
-You've got 9 memcached servers with a cache hit rate of 80% resulting in a database load of 80%. What happens if you naively add another memcached server to your cluster? (a question our VP of engineering actually does ask, and I stole it 'cause it's great)
You don't have to be a hardcore nginx source code diver to understand your stack. You just have to understand the principles behind it well enough to make good decisions. A sysadmin's job isn't to audit your code for your terrible understanding of the model on which your stack is predicated.
I think you may have taken my comment in a harsher context than intended.
First, I don't think it's good for a developer to have all the crap in their head that I do in mine. They'd be spending way more time pre-optmizing code than they would just getting stuff done.
A good developer will write a well thought out TDO that infrastructure and business units can provide feedback to.
Your questions about asynchronous i/o and memcached are great, and if memcached was a requirement for a sysadmin at a position it would be a good question to make sure they understand the software.
As a sysadmin though, you're also balancing
- Can the new system exist on an existing server/chroot/virtual machine
- What monitoring is necessary for the application afterwards
- Does the application meet all the standard requirements for database access
... out PFC is over 100 steps, this is just some of the bigger ones
- Can the sysadmin support the system after go live? The developers job is to create it, the sysadmins job is to keep it running long term.
Also, you are correct. The sysadmin shouldn't be auditing your code. The sysadmin should have a stack in place that they know works, and be able to help you do one of 2 things:
- Give advice on how your system can best fit in that stack
- Help you change the stack to best fit your application
Which direction you go on those two choices depends on the nature of the application, how mission critical is, required timelines for launch, budget and more.
Again--I am that developer, and I find that keeping "all that crap" in my head is extremely easy and makes me write better code. It wasn't straightforward to learn, but it made me better at doing my job. A question about memcached isn't a "sysadmin" question, it's a developer question to demonstrate that they understand what memcached (or whatever) does, its limitations, and its knock-on effects. These are things that I am entirely and without reservation convinced that a developer must know, and know intuitively, to be worth trusting with code. We have to write the code that uses these things--we'd better know how it works! That a sysadmin should know it as well is of course true--but a sysadmin should be a developer, too, to be able to make correct decisions in everything you've said above.
I don't say this stuff just for show--I've been there and done that, and it's stuff I've done to become what I consider an acceptable developer. I can, and have, set up a three-tier fully-enterprise-up-the-ass application infrastructure on my own (I have a couple machines on a VLAN at home as a VM farm) in order to learn this stuff. A developer who can't work the stack is a developer I don't want touching my code, because they're a developer that doesn't understand what they're doing actually does. Abstractions are great, but every abstraction leaks--they should be treated as productivity enhancers, not as insulation from knowing what you're doing.
Nobody can know everything about everything (there's a guy here at my current job who has forgotten more about the JVM than I've ever learned), but you gotta at least be able to talk about it. If a developer can't explain to me high-level concepts like how your web server handles threading, I don't even know what to say. This stuff is just so integral it's not even funny. You just gotta know it. Specialization is for insects.
.
EDIT: I think you and I are coming at this with very different philosophies. At my current employer, we don't have a PFC. We don't have TDOs. We have two live-site guys and a few operations engineers who are able to jump into our codebase (one is probably the most knowledgeable developers at the company who wanted a change). We have developers who can be trusted to make good decisions, and we have senior devs and managers who are very well equipped to analyze the plans and ideas that come from junior devs to say "hey, that won't work." But to be able to trust developers, they've got to understand how stuff works. (We don't ensure full-stack knowledge as well as I'd personally like, but we do a pretty good job of it--at the least, there are no "front-end engineers" who can't sling Java and no Java guys who can't write HTML/CSS/JS if a project comes along that requires it.)
probably because at nearly every place I've worked at the sysadmin/ops group was the #1 obstacle to getting things done. Sure some of it was "saving you from yourself", but a lot of was pure obstinatance for no other reason than I think sysadmins just cant turn it off anymore... much as your mom says "dont frown your face might get stuck", it appears a lot of people (not just sysadmins, developers too - you know the one, in the corner who is old and surly, been there forever, and hasnt been promoted in about as long).
The deck is stacked against sysadmins - developers are the people who deliver business value to leaders, and sysadmins are the ones who cost money and say 'no'. When you are on a tight timetable, the person who says "sure" is the one who gets the high fives... assuming things work out, of course.
>sysadmin/ops group was the #1 obstacle to getting things done.
This happens when the company discovers that when the system goes down they stop making money. If you've managed to design and build a system to be very resilient and reliable on the chosen platform and has proved to need minimal maintenance, then chances are good you have a very small ops team who doesn't get in the way much.
I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs
There are plenty of DBAs complaining about the low knowledge level of a lot of devs, e.g. using an ORM instead of writing SQL and wondering why the app runs like a dog.
> It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-).
I would say, 25 years later, the need to fix C bugs is far less relevant. That said, I'd say around 50% of the system administrators I've worked since moving into the Unix world know C and can at least read the source code of the operating system they're supporting. Even if there's not much need to write anything or even submit bugfixes anymore. Also, the number of unix admins I know who don't know a scripting language well enough to write a reasonably-sized, useful tool is vanishingly small, and those few have been very solid with some other technology(network, storage, etc.) and valuable anyway.
"System Administrator" is a broad term, just like "Developer." I'm not sure google has either, right? If we're going by fancy titles, Google has "Software Engineers." I'm not going to criticize Google's culture but the fact is if I say "Software Developer" you know what I'm talking about and from context should be able to tell the difference between Linus Torvalds and a guy who creates wordpress sites for his friends.
Sorry, even there, you have sysadmins who just sit there and run through playbooks and never improve anything. This is how you have teams with 70 pages in a 9 hour daytime shift as a matter of course.
There are pure SAs around, and I don't just mean the caste system. The entire sysops organization was just one example. They may have been rolled up into another organization but the people still exist.
I worked as an intern for the SQL server project. If you can explain accurately how to optimally structure SQL queries, you can probably write database code yourself.
> If you can explain accurately how to optimally structure SQL queries, you can probably write database code yourself.
Yes, probably. That doesn't mean your role on the project, in the company, or in your job description isn't "Database Administrator." Or operations engineer or reliability engineer or whatever fancy name you want to call the fact that you are more responsible for running the system than building it.
A more realistic example, though, is that a DBA and developer collaborate; the DBA collects and analyzes data from the production system and shares this information along with recommendations to the developer, who is ultimately responsible for restructuring the query.
In a very small team there's probably no need for these roles to be divided.
Sure, but I think that in this case the difference between your view of an operations engineer and a developer are fairly minimal in terms of skill. They're both development roles.
When I was interning at Google I noticed that these roles were differentiated as SRE and SWE. I didn't notice a lot of competency difference, just different application areas of the same competence. I think many SREs and SWEs could probably switch roles successfully.
That's more or less the point. System administration vs. software development is as much about priorities, mindset, and day to day environment than technical competence. This is true at all levels, whether you are pushing boundaries Google or supporting internal corporate users.
I never had a google internship or anything similar, and my first technical job out of college involved supporting an internally developed, browser-based ActiveX/.asp/vbscript tool for viewing and annotating loan documents. For a long time we on the systems team were unable to upgrade IE on user workstations because of a critical, easily reproducible bug in the document viewer tool. Eventually, I managed to get access to a snapshot of their codebase and tracked down the bug in a few hours (despite never having written a line of .asp or vbscript in my life and not having access to any of the documentation for the 3rd party libraries and activex components). The devs, of course, didn't listen to me anyway, despite my basically handing a solution to them. The bug remained for another year until that section of code was incidentally re-written to support some other feature.
The next company I worked for, a web company, had a much better relationship between development and operations. It might have helped that we had a lot of Indian developers, who didn't seem to have the same prejudices that western devs do: that system administrators are people who just aren't skilled enough to do development.
Programming languages are simply syntactic representations of predefined concepts. For the most part, if you can understand the concept the programming language isn't hard to pick up. Hell, I taught myself Python when I was 9 years old -- it wasn't the syntax of the programming language that was hard or the reason why I wasn't instantly a good developer. I needed (and still need) the real world experience of understanding the complexities of real software in order to develop real software. Anyone who understands these complexities and is only hampered by knowing the syntax of a programming language is underselling themselves -- with only a little more hard work they should be able to easily begin a development career.
Author here - I step away for a few hours expecting HN to ignore yet another self submitted article and look what happens!
tptacek's comments sum up the point - hence the qualifier of "pure". I think the SysAdmin role is rapidly evolving to look more like a programmer's job description from several years ago.
Sure, someone still has to write the puppet configs and monitor and manage the deployment, but that person is a far cry from the skillset and job description of 5 years ago.
Who will make sure your provisioned configurations have proper redundancy?
Who will move your domain when an availability zone has an outage?
Who will work with upstream networks when you're getting DDoS'd?
Who will wake up in the middle of the night when a service you're integrating with is not working?
Maybe you've never run a large system at scale, or maybe ou just had bad experiences with "SysAdmins" that didn't, but the role of hand-holding a complex, integrated service is not going away. The fact that you may not need screwdrivers for certain systems is almost beside the point.
Nevermind that, when you're at sufficient scale, you can run your own data centers cheaper than Amazon. Break-even happens well before a thousand nodes.
(I'm not a sysadmin; i'm an engineer who is very glad we have good sysadmins around :-)
If he'll probably never hire another pure sysadmin then he'll probably never build anything of scale that people will pay for.
Sysadmins do a lot of stuff, and this is only somewhat reduced by cloud services. Or is he expecting to get the developers to maintain OS versions, security patches, log monitoring, proxy servers, database replication, backups, firewall rules, failover testing and so on across dozens or hundreds of systems? Right.
I'm not completely sure that I agree. While this might be great for small startups, once you get past the growth of needing more than a few dozen servers, your cloud and other computing costs are going to be exponentially higher than what they would if you had actually purchased the hardware from your favorite hardware vendor, hired a competent sysadmin, and managed the hardware (and storage/network) in-house.
Currently, at $DAYJOB, our database is growing by about 1GB/day, and our storage is growing about 4GB/day -- in a downward housing market. If/when the housing market kicks up, I expect to see those double, if not triple. Having someone in house who can keep the right people informed of the risk (of running out of space, of buying the wrong equipment) who is focused on keeping of the needs of the company balanced with the needs of the software is critical -- do you honestly think your cloud provider cares if your customers can't access their data during an outage? If you're a small fish...probably not...
I agree with you but I'll go even further and say this guy has no idea what he's talking about. As someone whose been in charge of a fairly small cloud outlay (8 virtual servers, 2 of which go off at night) I can tell you the cloud introduces its own set of challenges.
So there's less hardware replacement and more DNS configuration but none the less there's still a job for system administration.
In fact, I'd say the role of administration becomes even more significant because the cloud industry is so new. Hardware has, in many ways, become pretty rock solid in the last decade. We're to the point where a hard drive in a RAID setup can go out and the users won't even notice.
The same can't be true for things like SQL Server Federation and caching servers which used to be the exclusive domain of large web sites but are increasingly being used by smaller companies who don't have a whole staff to dedicate to their upkeep.
(One Caveat: If cloud models like Microsoft's Azure take over this could change. By that I mean clouds that are literally frameworks in themselves and not virtual machines)
You have to make sure you use the right tools. If you re-invent the wheel a dozen times, managing cloud deployments will be a nightmare. Tools such as Rightscale or Scalr.net are well with the money.
I manage 2 dozen Ec2 instances of various types as a side task to my regular development job. I script everything. When things are scripted they are consistently reproducible and fast. Humans make mistakes (including me).
I would prefer there to be a dedicated SysAdmin but there just wouldn't be a full time job for him to fill.
"Now, the watchword to the development team is: it's not done until I can one-click deploy it."
I have apparently been spoiled somewhere along the line, because I really don't like having clumsy manual processes that involve software/computers. I was recently tasked with figuring out how some researcher's code worked (with an eye towards automating the use of their tools), and it took 8 pages to get most of the steps documented well enough that I could follow them. Steps like, "edit the path in line 977 to point to your input files, recompile, run, click button A..." -- i.e., things that should never have been done manually to start with.
How on earth people whose primary job involves writing code to do things for them can fail to automate tedious (and frequently error-prone) parts of their job is beyond me.
That said, many companies are still going to need to hire somebody to write the one-click deployment. Somebody will have to know how to interact with the cloud services, and how to get around all the little minefields and gaps left in the cloud service offerings, and odds are you aren't going to want to pay that person as much as you pay your developers that do the "real work." That's probably where the sysadmins will go.
It's simple : The people you are complaining about set these things up once on their computer and then forget about it. They don't realise how many things they have set up once along the years.
It is not until you appear that it's time for the app to be deployed elsewhere.
In 1999, everyone was telling me they'd never need to hire another Java programmer now that Rational Rose would soon be able to spit out complete code based on the input of a team of architects. Just sayin'.
That might work for him running a small business. How do you think those images you 'just spin up' at a cloud provider are made? Who makes sure the boxes that run those images you've just brought up have power and run smoothly?
His point is that Amazon hires sysadmins, not him.
However, that seems based on a very narrow definition or understanding of what a "sysadmin" is or does for the business.
I identify "sysadmin" with "operations" but perhaps others see it differently.
People seem to think puppet and chef are some sort of revolution, as if we didn't automate all this stuff before with bash, rsync, kickstart and cfengine.
As is typical with this line of thinking, it only scales so far and only applies as long as you are "on the map" of what is standard. Stray too far and your devops personnel will spend more and more time "administrating" and less time coding. Being primarily a coder with considerable sysadmin skills, this has happened to me in every job I've worked in the last 5 years. I start out devops while everything is standard and slowly do more and more admin until I eventually quit (because I'm primarily a coder). The job I'm quitting this month is because I'm now 95% admin. Like all resources, you need what you need. Devops is sufficient is specific circumstances, but if you stray from that scenario eventually you need an admin. They're the glue when things don't quite fit seamlessly together.
Summary: You don't need to hire sysadmins because you can outsource your systems to a cloud provider.
This is true, but it's also a lot like saying you don't need to hire developers because you can outsource your development to Lithuania. Really what it means is that it's possible to outsource systems. If you're the little guy and your needs manage to mesh well with what you can get off the shelf from a cloud provider, this is true. Otherwise, this isn't true. If you're the little guy and your needs don't exactly fit what you can get off the shelf, it's going to be trickier. If you're the big guy, you can afford to cut out the middleman.
A development team had a problem. They were in need of a manageable production environment. "I know," the Chief Developer said, "We'll use Chef!". Now they had two problems.
However, I agree with the basic contention - that the role of a pure, old-school, risk-averse sysadmin is obsolete. In a world of Agile developers practising continuous delivery (or hoping to), the system engineer is now another cog in the development machine; not part of a separate machine.
The resulting closeness does wonders for understanding and quality of the result. Unfortunately, at the time of writing, system engineers who speak Developer are relatively few & far between.
Operations shouldn't develop and Development shouldn't do operations. That isn't to say that Ops can't program. I write a lot of Python to get things done but that's not Development.
I'm a sysadmin in a 100% AWS shop. Before I showed up, Dev was running things. It was horrible.
Versions differed greatly, they didn't use init scripts for anything, deployment was pretty manual, and their release cycle was really slow because they spent too much time in operations where they were clueless.
My job is to keep production rock solid and provide Dev with the resources they need to write code.
I think you need to distinguish here different kind of systems.
a) Deployment of 'standard' applications like Drupal, Joomla!, Wordpress, etc. It makes sense to automate it once and let people "self service" In this case either the sysadmin is responsible for packaging those apps or they can get a third party "catalog" that you can deploy to your virtualized/cloud infrastructure (that's exactly what we are aiming for at BitNami, you can check it out http://bitnami.org/stacks/)
b) Complex multi-tier, in-house applications. Those are much harder to automate because they tend to be "one off" systems and will require 'traditional' system administration for a while.
c) Web companies that run a logical application across multiple servers (like Zynga, Netflix, though obviously most people will do so at a smaller scale). Here you need to use something like Chef or Puppet, there's simply no other way of doing so.
In other words, before the advent of cloud it was 'optional' for system admins to automate their work. Now it is expected. In that regard, I think automation is the real benefit that cloud computing has brought us (rather than the capex/opex and autoscaling that people often mention)
If you look at it from the point of view of creating a new company that is based online, chances are that you can design the system from the ground up to be entirely hosted in the cloud. Even better than AWS is if you can host your site on a PaaS solution such as Google AppEngine. GAE negates the need for a DBA, netadmin and a sysadmin.
I'm building a web based company right now with a good friend of mine and we've gladly chosen GAE because of this single attribute. Both of us are 15+ year sysadmins as well as software engineers. That said, instead of worrying about how we are going to scale and run everything, we are able to focus entirely on the development of features and building the company. As a result, we hope to never have to hire any IT positions.
In the end, I don't think the roles will go away since someone still needs to run Google. ;-) I just think that the roles will eventually morph a bit for the companies that can take advantage of PaaS solutions.
It is an interesting line of thinking. Way too many startups I've been involved with/talked to have over looked the role of admins/an ops team. The mentality seems to be that developers should also focus as admins/ops. Surely nothing will go wrong on a weekend after a developer has already put in an 80 hour week, right?
From my experience, sys admins are important. If you are doing anything of scale, knowledge on how to setup, tweak, and debug systems is critical. It is helpful if the admins can stretch a bit into the app layer, just as it is helpful for developers to be able to stretch a bit into the systems layer, problems come up and knowledge of the parts helps.
As an engineer that can also setup systems for scale, I know my limits and really appreciate help from and learning things from good sysadmins.
Somebody has to write the deployment scripts, choose a cloud platform, and update the platform. If a cheaper cloud platform comes out, somebody has to figure out which platform is cheaper. You may have automated error logging, but someone has to go through the logging and see what went wrong. Somebody should also look over how reliable the platform is.
If your business only requires a standard Rails site, there may standard tools. However, when you develop new technology, someone has to deal with deployment and scalability issues. This may be more of a developer job, but there are significant sysadmin issues.
I think even "pure" sysadmins will have jobs until the cloud providers figure out how to solve their IO bottlenecks. I keep hearing horror stories about database performance and how people are forced to go back to good ol' dedicated hardware to solve it.
I think it'll be both faster and slower than he says.
Faster: Cloud adoption will be huge.
Slower: Cloud technology will increase the amount of apps and data used. Low end jobs will exist helping firms get on the cloud. High end jobs (fewer though) will exist at the cloud providers.
I work at a company currently in the process of deploying an Oracle 7 cluster. Regardless of technological advances in startups, enterprise will always need sysadmins, if for nothing other than filling in the correct check boxes on the audit forms.
It depends. If it's still around in 10 years, I'd be willing to bet not a single line of code currently used by Facebook will still be running. Some startups will become "enterprises", because of what they do, but I'm talking about financial services, banks, government departments, public services etc. Their technology doesn't move forwards, because it can't. I'm sure little to no startups are using Oracle IAS (Internet application server), yet it's one of the most used technologies in enterprise applications, and will be for some time. If you're a sysadmin with a deep understanding of IAS, or Oracle DB, or HP-UX, or any of the countless other "old" technologies used in enterprise, you'll have a job for life.
I'd really like to see more activity around creating, publishing and refining Puppet, Chef, Fabric, etc scripts that are usable in various projects. For example, publicly available scripts that approximate what is provided by Heroku and DotCloud.
I totally agree with proper deployment procedures. But having them doesn't eliminate the sysadmin at all. Sysadmin should be deploying standard packages that are necessary for the system, along with updates for them. Developers should be deploying in-house code, and updates for that.
I worked at a company that hired a sysadmin too early. When he quit, I ended up doing his duties and mine, too. Of course, I automated 99% of it and everyone was happy. That was really early in the life of the company.
Later, when it came necessary to hire one, they didn't. They'd expect a developer to deal with the issues, and were always dissatisfied with the time it took, and the lackluster results. Eventually it got so bad they hired a dedicated sysadmin again. Now, unlike the past, they actually needed one and he was busy all the time. Developers stopped being unproductive for weeks at a time, working on projects they didn't understand and couldn't complete satisfactorily.
For a startup, yes, skip the sysadmin at first. But once you have a complex system, hire one. It'll save your company.