As a dev, the whole "rapid release" thing made a lot of sense to me. As a user, though, I've turned 180 degrees on it. I hate "rapid release" with a passion now. Which means I dislike Agile by consequence.
Rapid release means I can no longer rely on the software that engages in it. It's constantly shifting, with frequent changes in UI and workflow, and new and exciting bugs constantly appearing.
I miss the days when I could plan ahead for new versions, so I could time them to have the least possible disruption to my life.
Yes, they do? I thought the entire idea behind "release early, release often" was to shorten the feedback cycle between code changes and user reports so that it's easier to fix the issues?
However, not every code release needs to include a UX change. I agree that changing the UX every week is a solid anti-pattern.
Feature/UX changes should be behind feature flags, and launched independently of a release. Releasing often let's us identify and fix bugs quickly. I hear you that as users we appreciate more stable UI, but to get to an intuitive UI, you have to experiment and iterate.
While there is value in discrete and planned releases, we value responding to change more.
I hear you that as users we appreciate more stable UI, but to get to an intuitive UI, you have to experiment and iterate.
But there is no rule that says you have to do those experiments in production and annoy your paying customers who don't want to see them. If you didn't insist on releasing every five minutes you'd have time to iterate privately. Successful software was made this way for decades before modern web applications came along and frankly a lot of it had better UI/UX than a lot of the junk that gets pushed out today.
While there is value in discrete and planned releases, we value responding to change more.
I suppose the question is what change you're responding to. In this situation it seems like a lot of that change and the need to respond to it are self-inflicted damage.
+1 for "let's run this big UI change using our TEST platform past some physical rooms full of typical users to see if they hate it" rather than just firing it into prod with some buzzwords to justify traumatising millions of previously satisfied customers!
Personal favourite was Spotify removing their playback widget to "improve the customer experience" and oh my God the backlash was ferocious. It was entirely moronic and screamed of Product Manager -> Customer with zero actual user feedback gathering:
26 pages of complaints produced this response, basically we removed a thing you love and use constantly, but you can vote for it as a feature request:
"We always take retiring features in Spotify very seriously, and do want your continued feedback. The best way for us to measure your comments are via votes in our Idea Exchange.
For this reason we're going to lock this thread. We encourage you to move to the Idea Exchange area where you can vote and comment on feature requests."
Approaches and terms like "iterating fast", "responding to change", I dunno, just feels like a poor substitute for better planning and evaluation because those can be hard.
The irony is that it often takes only a tiny number of test users to spot a glaring usability problem that could sink a major UI change. Someone (possibly the Nielsen Norman Group?) did some research on this a while ago and found that even single digits of test users would be enough to collect very useful feedback.
People are so obsessed with scale and techniques like A/B testing today that they seem to forget that kind of test can only tell you whether one of the options is significantly better. It can't tell you whether you're finding the best choice between excellent options or the least of evils between dreadful ones.
Totally. You could imagine even a tiny handful of trusted users who consistently give valuable, reliable, feedback.
Microsoft seems a company just oblivious to user feedback - Beta users howling about disastrous Windows changes mooted, and they just.... ignore them. I don't get it.
> frankly a lot of it had better UI/UX than a lot of the junk that gets pushed out today.
I feel like this is the key point that adherents of modern development practices need to address. We've been sold a lot of stuff on the basis that it would drastically improve the user experience - but a lot of people, myself included, feel that UX actually declined in the past two decades.
I think software quality has fallen in many areas - including UI/UX - as a result of widespread Internet access. It's no longer assumed that anything actually has to work when you ship it because you can always fix it with a software update later. There's no need to do real usability research before you ship because you can always spy on users later.
Unfortunately this seems to have created a culture where little that ships was really ready - at least not to the standards we used to consider acceptable. Everyone is busy pushing out new features and other visible changes every ten seconds. But somehow no-one ever finds the time to go back and fix the substandard quality that got an MVP in front of users on launch day.
Google even warned us about this years ago. Remember in its early days everything was Google XYZ Beta? It was an excuse to ship stuff with substandard quality then and it still is now. We're just less honest about it now.
You can’t have an intuitive UI if it’s constantly changing so designers look productive in sprints: you’re constantly defying the intuition for your current product, while giving users no stable target to learn the behavior of.
It's about learned patterns, but not learned patterns of a specific version of your app. For example at this point intuition is expecting the hamburger icon to open a menu. It's not about expecting the green circle to open the menu because it's done that in the previous version of your app.
You can definitely have both intuitive and changing UI if you think about it.
Easy when it's a color change, but this attitude is horrible when it comes to anything functional. You stop being able to learn how things work because someone wanted to make a small tweak, and now you don't even get notice it's happening.
Yeah scrapers and in any case if it changes the black and white text to both being the same cool gray. No joke.
This forum gets hacked into that, you can't see points a comment gets unless you measure it by the gray of the comment, in which case you can tell exactly when it has 0 points, or -1, or -2, or -3, or -4. Scrapable, and illegible. Opens itself to voting rings. Not even legible if highlighted, you need to copy-paste to another application.
No such thing as small when you don't know what features of your product users rely on. You might affect 0.5% of users at any release but given that is different users each time you will piss of many of users. The thing is, no one seems to care since everyone is doing it.
> Rapid release means I can no longer rely on the software that engages in it. It's constantly shifting, with frequent changes in UI and workflow, and new and exciting bugs constantly appearing.
There’s something terribly wrong in the picture you’ve described. I do not know what Agile stands for, but our company was doing “a new version each Friday” for like 15 years. One of the axioms of such a fast turnout was “a new version should work in the old environment exactly the same as the old version” - except bug fixes of cause, some of which were made optional nevertheless.
I think this is a great point. When agile and rapid release is done correctly, it should work out as "kaizen for software" not "we overhaul our UI and workflow every 3-6 months".
Small incremental change is easier to test in most cases, the issue is, changing software that was designed as a whole over to incremental feature improvements is hard, because often the existing bulk of code cannot be meaningfully unit tested.
I hate both sides. Rapid development loops should be just that, development. Bug fixes are fine because they tend to guarantee functional consistency and no UI/Ux changes (unless it's fixing one that was broken).
Today it's just the wild west of people basically having production not far sepeeate from development in terms of changes. UI/UX changes are the worst in this context. I opened Venmo the other day and I had to waste 5 minutes because someone decided they wanted to change the UI and I had auto updates applied (shame on me I suppose).
Do not change the interfaces with your users folks unless it's truly broken.
It's a hyper competitive space. That's the equivalent of waiting 20 minutes in line for an ice cream you don't really want all that much. Skip.
You may have more patience and composure but most of everyone is sifting through countless activities in a day. 99% ill close the tab than wait for a 5 minute update for an activity im not getting paid to do.
There is a lot of software I've used where I would have been just fine if all development stopped. I don't need or want new features, keeping up with security fixes, bugs, and OS updates, is fine, but feature wise many things I would have been happy to call the software done.
But that doesn't keep programmers employed, VC's making money, or management justifying themselves.
Yep. Word for Windows 6 was pretty much the end of the road for word-processor improvement. It has been nothing but regressions since, to the shitshow that is Word today.
meanwhile after 20 years, foobar2000 only gets better with each release. I even still use (the Windows version) on my Mac thanks to Wine. The memory footprint of Music.app when I load a large playlist is an order of magnitude greater than that of foobar2000 despite all the overhead that comes with Wine.
Or that every ducking time an application developer pushes an update, they feel the need to pop up something in front of me the next time I open the application. The worst time to do that is when I first open the application because that's when I've made the intentional choice to use it to get something done and the last thing I want is for that flow to be interrupted.
The irritation is tripled if it's an un-skippable "let us walk you through the nineteen changes in a slowly constructed slideshow!"
This is second only to applications that don't properly retain state--especially after an update--so every time I open them I get a brief glimpse of what once was (and what I probably wanted to go back to) before I get a 50/50 split of being dumped to the application home screen or "oops, something has gone wrong! lol, software, yannow! Please to be logging in once more, thx."
"We've improved the way you interact with our app!"
>click next.
Cursor slowly moves to the upper-right corner of the screen. "We've added a 'search by topic' functionality to our help menu! No more need to scroll alphabetically!"
>click next
Cursor slowly moves to the lower left corner of the screen. A red circles appears, pulsing. "The toolbar now displays the two-letter country code that your operating system is set to. Useful for travelers!"
>click next
Cursor slowly moves to the upper middle of the screen. A popup appears. "If you have forgotten to fill out your phone number and email address in the Account Options, a helpful reminder will appear each time you start the program. This will help you reap a number of useful benefits that sharing your information allows!"
"Hello, my inquiry is whether or not AppleCare+ covers spontaneously throwing my laptop out the window thus it falls down three stories into a storm drain. No, this is purely hypothetical. Yes, I'll wait while you research."
If it requires a tutorial for all users, that’s called a manual and it doesn’t have to appear on launch or force you to read it. Unskippable step by step tooltip guides are the absolute worst.
Many features are better off hard to find and left that way than shoved-in-your-face discoverable, or introduced by a tutorial. Those who want to get the most out of their tools will find them and will appreciate you treating them like fellow intelligent beings. The rest will never know what they missed and won’t worry about it either.
> If it requires a tutorial for all users, that’s called a manual and it doesn’t have to appear on launch or force you to read it. Unskippable step by step tooltip guides are the absolute worst.
Lots of game developers have conducted extensive research and found that player retention is considerably higher when you "annoy" them with these things rather than just throw them to the wolves and tell them to RTFM.
There's basically two reasons to make games: to make money, or to make art. You can try to do both at once, of course; and focusing on one doesn't necessarily detract from the other.
Consider FromSoftware, for instance; often pilloried by the gaming press for their lack of difficulty scaling or tutorials, but adored by their fans and able to turn a tidy profit with each release. Similar, but less well-known, would be Spiderweb Software; which has steamrolled along for decades without breaking their artistic style.
Not everyone wants to be the next Candy Crush or Clash of Clans; many just want to create an experience that they would enjoy.
I don't think the presence of tutorials automatically count against artistic merit, unless they are done badly.
From Software games are more hands-off because their goal is to make the player discover the mechanics and systems on their own. Not every game wants to do that, and as such I don't think games like Halo: Combat Evolved or The Legend of Zelda: Ocarina of Time are automatically worse for this. In the latter case, I think the unskippable tutorial section pulls double duty in setting up Link's backstory; You're with him as he first learns how to fight, use his shield, etc. before going off on a grand adventure.
From Soft games are also not immune to the unskippable tutorial mission. Both Elden Ring and Sekiro start you off in a relatively safe linear level with pop ups teaching you their basic mechanics.
I want to circle around back to the comment that started this though, because that user said they immediately uninstall and refund any game with an unskippable tutorial. Maybe they meant this as hyperbole and I'm just sitting here wooshed, but it seems like an extreme stance to take.
Indeed, tutorials do come in a great many varieties, and when it's done well the user may not even be aware that what they're experiencing is a tutorial. That has a strong tradition in game development; where a new ability or a new tool is provided to the user and the level is designed thereafter to allow the user to explore that new thing. That's closer to your Zelda or Halo.
Even FromSoftware games take this approach. Elden Ring has a whole opening stage that's clearly designed to expose users to core gameplay mechanics.
But then there's those unskippable dialogues, or the tedious "now press this input to perform this action" prompt found within a purpose-built tutorial area. Those, specifically those, are what I find cause for immediate product return.
There was some Need For Speed games on the PSP that were particularly awful for this. You'd be _racing_ and it would pause the gameplay to show you a dialog about how to drift or draft or somesuch; completely breaking the flow of the experience. That's what I would demand a refund for.
That said, art is subjective, and the experience is ultimately defined by the beholder. Maybe some people actually enjoy being brow beaten with unskippable prompts and hand-held through tediously constrained environments, with their experience jarringly interrupted at unexpected times by dialogs that break the suspension of disbelief.
If I applied this rigorous requirement to all games I purchase then I would have ended up missing out on some of the best games I've ever played. Like Ocarina of Time or Halo.
You want a chuckle, turn off automatic app updates on an Android phone and see how many prompts you get. And how terrible the patch notes are ("Fixed some things")
In fact, my Android 11 device has no browser enabled, the play store is disabled, and almost every app it came with is disabled. It functions solely for email, instant messaging, a TODO tracker, and taking photos.
Edit: not sure why this was downvoted. Did someone find the idea of a narrowly-focused phone to be offensive or unworthy of discussion?
There's almost nothing to auto-update on my device, aside from less than a couple of IM apps and an email app. Sharing that info was the point of my reply.
I wonder if developers who do this simply don't want to publicly identify and take responsibility for bugs that they introduced earlier. It's easier to just say "bug fixes" then it is to say something like "I overlooked several surprisingly common edge cases in the parser module and have updated it to correctly handle several character sequences that were causing exceptions to be thrown." With automatic updates and vague update notes, anyone who did not directly experience the bug never has to know it existed in the first place and the developer never has to admit it existed in the first place.
I have 25 applications on my android that never get updated. I don't use gmail, and I don't want it installed on my phone at all. (I have the mandatory account to activate the phone, but all my mail is via fastmail). I update the apps I use (or sometimes apps I think I should use but don't)
> (I have the mandatory account to activate the phone, but all my mail is via fastmail).
Interesting, I don't have a google account associated with my phone. It means the google app store doesn't work, but everything else works fine AFAIK. I use FDroid or just download apks directly.
Nowadays even on iPhone half the apps phone home to their server, determine they’re not running the latest version, and then refuse to let you use them until you update.
Counterpoint: as a user, while you realize you hate the frequent changes in UI and workflow -- you're not realizing that you're using software with a much better UI now overall than it would have been if not for the frequent experimentation over the past e.g. 2 years.
Rapid release and experimentation means ultimately finding better solutions for the users faster. If you go slow, things don't change as much but they don't get better much either.
We tend to assume that all of the good changes would have happened anyways and all the bad changes are because of rapid release.... but that's not how it works.
Better UI is not universal. And the absolute, hands down unquestionably best, UI that can possibly exist for any given customer is the one they're already an expert in.
Can a different UI end up being better for them? Sure. Will it be better the day it's released? No.
The user has to pay a cost every time the UI changes, regardless of whether it's actually better for them or not (and it will never be better until much later, once they've become an expert in the new version).
Worse - I see more and more companies optimize UIs for the "initial user experience" and onboarding new users (more paying customers). The sad truth is that many times that means dumbing down a much better UI for the "already expert" customer - just like putting training wheels on the bike is great for the person that's never biked, and utterly infuriating if you already know how to ride.
This is why existing customer bases are so rarely fond of any change - regardless of whether they end up liking it in 3 months time. It has a real cost, right this second, and they're being asked to pay it.
It’s also an existential risk for many business customers.
If you already spent months training a new hire on how to get something working with all the different random tools and software they use, changing something meaningful means breaking that persons ability to do what they need to do for their job, at least for some time.
Even changing something inconsequential will confuse some users for a bit, require making/updating new training guides, etc.
If it’s a business on the edge or under during a stressful period? That can literally tip it over and make it non functional. That is especially true for small businesses.
Business productivity software is usually much more slow-changing than consumer software, precisely for this reason. They know customers' training guides need to be rewritten so changes become a big deal.
This is one of the reasons business productivity software becomes unwieldy after a while -- it's fine to add buttons and menus, you just can't change existing ones or what they do. So then version 5 will finally introduce the massive, desperately needed UI refactor after ten years, while plenty of clients just stay on version 4. (An example of this is the Blackbird CMS -- old Blackbird vs new Blackbird.)
So then version 5 will finally introduce the massive, desperately needed UI refactor after ten years, while plenty of clients just stay on version 4.
If your new product with ten years of improvements was so unattractive compared with the existing version your customers are currently using that they were choosing not to update, what does that say about the value to the customer of all those improvements?
In the world before SAAS we saw both extremes often. Some products had incredible dev teams and people would upgrade relatively often even when each new version cost extra money. Other products barely produced an interesting change in several new versions over several years and users mostly didn't pay for those updates.
SAAS is a convenient business model for the developers in the latter category because it locks in the ongoing revenues whether any future changes add value or not. They just need enough lock-in effects that once you've signed up you can't easily move to one of their competitors.
You are posting on a site that is performant, light on resources and has great UX - and has been mostly unchanged for a decade. Its counterpart, Reddit has only regressed compared to its biggest competitor, it's old self.
I haven't noticed any significant UX changes in any software I use in the past two years, and would struggle to name improvements, even over a longer timeframe.
I'm an eternal lurker and UX happens to be my passion. I'll break my vows of silence to state this: saying that HN has bad UX is an understatement. I spent years missing out on the site because, in its natural state, it's quite simply unreadable to me. It reminds of 4chan (not that I would ever want to use 4chan, but that's precisely why I mention it). I can't browse it unless it's on an app on my phone, or with the set of custom CSS snippets I wrote for Stylus on my PC (and that's just to make it readable).
It's also built with table elements. The programmer in me is screaming because that's not the proper use of tables (I know that was the way to build layouts in the past, but that was then and this is now). The UXer in me is screaming because it's a nightmare for accessibility.
I want to add that I'm not coming from the "Overengineer-and-Bloat-Every-Website" UX School of Thought, so I agree with your point about Reddit. Being performant and light on resources is part of what makes a good UX, but that's the thing: there's more to UX that loading fast and running without hiccups. I think HN's experience could be greatly improved without compromising in any of those aspects.
The default font size is unreadably small, it's impossible to reliably tap on upvote without accidentally hitting downvote on mobile screens, comments with code blocks with too many characters in a line break the width of the interface, huge comment threads indented 15 times aren't collapsed by default, downvoted comments lighten to become unreadable, I could go on.
But there's no decent alternative for HN's content and community, so we put up with the terrible decade-old interface because we don't have a choice. Although people are constantly inventing their own clients for it precisely because it's so terrible.
The only of those complaints I even half recognise is the voting buttons on mobile, but I miss them rarely enough that I’d probably still rather not give up any more screen real estate (horizontally or vertically) or add an extra click.
Otherwise HN has basically the UX I wish every site had.
The vote buttons are almost unusably miniscule and close together on desktop as well. There's not much excuse for it when so much of the visual real estate is empty space.
Frankly, neither programmers nor graphic designers are qualified to make good UI decisions. Human factors is a real field that people get PhDs in. You really can't just wing it, especially if you're a programmer whose mind has been thoroughly addled by decades of using programmer-designed UIs from the Nth circle of hell.
If HN had such great UX, there wouldn't be multiple browser extensions (and clones) that focus solely on fixing the UX. I'm not saying the UX is terrible, but I wouldn't call it great. I think the best term for it would be "consistent." It's basically Craigslist. It's fine if you've been using it for years and have adjusted your behavior to work with its quirks, but it does not provide a smooth experience that would work for a community you're starting today.
There is no UX that is the best for everyone, there are always trade offs that hurt some or help others.
Just because there are other products with different experiences that skin it doesn’t mean the core product is bad!
It actually means the opposite - the core product has such value and there are so many users, it actually is able to support sizable enough communities of people ‘outside the core’ to justify niche repackagings for them!
Exactly this. Too many definitions of "great UX" seem to focus on pulling in as many morons as possible because we refuse to force people to learn how to use something
Great UX should also mean fast, efficient, and dense with information. Respectful of time, attention, and intelligence. Not making an assumption of stupidity on behalf of your users.
If the "skins" were just preferential type changes, that would be one thing. But many of the fixes within HN Enhancement Suite are basic UI fixes.
Making links in profiles and text posts clickable. That's like the most basic thing you can do for your users.
Adding the ability to collapse a massive comment thread without having to scroll to the top. Also, indent guides (especially needed for the massive threads around here). Shit, HN didn't even have collapsible comments for the longest time.
Being able to comment inline and not having to navigate to a separate page where you lose context of the surrounding discussion.
Highlighting new comments since you last viewed a thread.
These are all basic things that HES addresses, there are many more I can point out. I'm not trying to insult the admins, I think they do an amazing job of running the site, but let's not pretend HN is some pinnacle of UX design. We are all just very used to it.
Still, going by the fate of new Reddit, there should be a 'hyppocratic oath' among UI designers, so that they make sure they don't make the user's experience worse in their quest for UX improvements.
Adding a ton of features almost nobody uses and rearchitecting the site using 'modern' frameworks, only to make it 10x slower goes right againts that oath.
Reddit's redesign sucks not because of new features or modern frameworks, but because they don't care about the web experience and spend all of their effort funneling users into their mobile app.
I pointed out some changes that are addressed by HEN in a reply to another post, but here are some others that would help.
The funny thing is that HN doesn't even adhere to the basic UX practices that many around here constantly complain about in new design like knowing what is clickable.
Take a look at each story list item on the front page, what do you think is clickable? Everything in the subheading is formatted exactly the same, but not everything is clickable. Same goes for the headings of each comment in a thread. And I think HN's violation of this is especially egregious since the formatting of these things aren't just the same, but they are greyed out. That's a pretty universal sign that something is not clickable. This is 100% confusing to new users.
This one is a bit more difficult to implement without a lot of JS, but one thing that I have run into many times is reading through a comment thread, trying to reply, realizing I'm not logged in, logging in, losing context of where I was, scrolling through the thread trying to find where that comment was, giving up and not commenting.
In addition to that, not remembering which comment threads have been collapsed. Accidentally navigate away? Good look finding your place again, especially if there were many new comments added.
The mobile experience can be summed up as "very precise clicking"
I think the top menu items are also named in a weird manner. Why does comments take you to a list of random comments in chronological order? How is that useful? You would expect that link to take you to your profiles comments, but no, that's the "threads" link. There is a lot of that on HN where things are (imo, purposely) obscure. I get why they would do it, but let's not pretend it's a good experience.
HN caters to the most technically literate community on the internet. They will figure out how to use this site no matter if you had to navigate the site via the terminal (some would probably prefer that "NO JS!"). We've all gotten used to how the site works, we ignore all of its issues because we've adjusted our habits to work around those issues.
And I'm not saying HN has terrible UX (there are actually many things we can learn from it) but it's definitely not the site I reference when I need inspiration for a good experience.
I would disagree that UI constantly improves, though. I can think of lots of examples where the opposite is true. But that's not due to rapid release, that's due to other issues.
> If you go slow, things don't change as much but they don't get better much either.
It's hard to think of when I've seen improvements that were so good as to be worth the cost, though. Things certainly change, but change is not always better, and even when it is, it's not always worth the cost to the user.
Pick up a vintage Mac and a copy of Word from the 90s. The UX is superior in every way to modern word processors. It’s more responsive, more intuitive, faster to start up, more reliable, easier to learn, and so on.
Decades of “experimentation” have not improved the act of typing out a document in any meaningful way. Now, they have improved the experience for the corporations who can profit selling out their users data to the highest bidder and locking them out of their own devices while charging a subscription fee.
The rose tinted glasses people have for old computers. Here’s a transcript of me using a Mac SE in 1995 to try and finish writing a college essay in Microsoft Word:
That is a commentary on the performance of that software, not on the quality of its interface. Run the same app on a machine from 2000, and it'll be blazing fast - and have sane UX.
> more responsive, more intuitive, faster to start up, more reliable, easier to learn
More responsive? No. No it wasn’t. Disks were slow. Memory capacity was low. You used to have to use third party RAM disk software to complete some tasks in a reasonable time. Or install a ram doubler to let you work on reasonable sized documents at the expense of slowing your machine down even more.
More intuitive? Hard disagree. The example I gave above was of getting a word count. You had to find the menu option to do that - it wasn’t just visible on screen. Same with spellcheck. Global search and replace was slow and clunky and felt risky. You only had one level of undo.
Faster to startup? Hells no. Again, slow disks.
More reliable? There was no auto save. Everyone who used a computer in the 90s has a story of losing hours, or even weeks of work to a crash. Everyone.
Easier to learn? Learn how? Word came with a few hundred pages of dead tree manual, but if you were on an institutional machine or running a ‘borrowed’ copy you didn’t have that. There was no internet to look up how to do stuff. Maybe you could pick stuff up from the tips section of a magazine? Sure. It was super easy to learn.
Compared to all of which, google docs or Word365 is just… light years ahead.
Fair enough. As far as I'm concerned, "peak UX" on the desktop was somewhere around late 00s, anyway. Although even early 00s were superior to the current sorry state of affairs.
It is very unusual for software that's been around for a while to actually have better UI than it did two years ago. Every time I pull out an old laptop or phone running 5+ year old software I notice that things look different and buttons are in different places, but functionally there's very little difference.
Most changes to already functional software are just changes. They aren't improvements or regressions; they're just making things different.
This is an excellent example of the headwinds/tailswinds asymmetry; we often don’t notice when a rough edge gets sanded off the UX but we scream when one gets added.
Short of having an army of testers whose job it is to continually retest the same UX sequences with every release, how is that possible? Rapid release to me implies releasing changes faster than they could possibly be manually tested. And automated tests are going to do a rotten job of detecting most UX regressions.
So you're saying that the way to maintain a stable UX...is to release changes directly to customers (or at least a subset of) to test, and if they react badly, roll it back?
It is one part of more wholistic UX/PM practice. Yes, you should have mechanisms to validate the impact of your changes on your customers. You should be having regular conversations with various subsets of your customers for qualitative feedback, both on specific changes and roadmap. You should be doing things like heuristics reviews and user-testing. You should have UX designers with experience designing interfaces with plans for gather feedback _before_ deployment and then validate that change _after_ the deployment with further analysis. During development you also need all the regular QA type testing as well.
But yes, if you are regularly changing your UI in order to improve your system and roll out changes, you need to validate the impact. When I was last a PM, I spent a bunch of time monitoring the rollout of changes via Pendo to understand BOTH, if was getting the adoption of features I expected from certain cohorts and seeing if there where any shifts in user behavior (either organically, due to intentional or unintentional changes) And we did this for "beta" groups as well to really validate that feature were working with early adopters before broader rollout. I guess you might not consider the later "production" in sense, but these were real customers with real workloads.
In my experience, that only happens when customer attrition is double digits. Otherwise, everyone has their head up their ass enough they’ll ignore or spin the feedback at 99% of organizations.
>Counterpoint: as a user, while you realize you hate the frequent changes in UI and workflow -- you're not realizing that you're using software with a much better UI now overall than it would have been if not for the frequent experimentation over the past e.g. 2 years.
CounterCounterPoint: If the software was perfectly functional two years ago, all that can happen with a change, any change, is to make it worse :-)
Why do you assume the changes automatically mean "much better"?
In my experience it feels about 50/50 for better/worse, whether that is features taken away due to some business concern, blatant contempt for existing muscle-memory, or misguided designers reworking things that did not need a rework.
I presume GP would call a weekly release "rapid", too.
Given the tone of the message, I expect "non rapid" to mean with a much longer time frame, and, most importantly, the release must be controllable by the user.
It is possible to separate security updates and other critical bug fixes from functional/UI changes. At least it is if you're any good at software development at all. The world would be a better place for users if more software developers still did that.
Of course you do need a strategy for maintaining multiple versions of something in that case instead of just forcing the latest version of everything you do on users whether they want it or not. With the lock-in effects of the always-connected web applications of today we all know which option makes more money.
I take some comfort in knowing that eventually most of the developers that force unwanted changes on their users will probably see their market share erode and lose their dominant positions. It's just unfortunate that we have so many quasi-monopolies in both essential and niche software products that it's going to take a long time for that to happen.
That is completely orthogonal and the whole selling point of things like Windows LTSC or Red Hat: Stable foundations with regular security updates but no surprises.
Yes, was going to post the same reply. Iterative UI, A/B testing, webapp-driven (vs. installed versioned apps) are completely orthogonal to the OP discussion of Continuous Deployment / rapid deployment.
CD is a deploy strategy saying the aspiration is to do one deploy per commit, rather than batching commits and deploying once per N weeks or whatever. It has approximately zero to do with UX and product design/iteration approaches.
One deploy per commit for a product with large team sounds insane - we'd easily have 50 commits a day for some hosted components. Even deploying this often to the non-prod environments is problematic (it takes a fair bit longer than 1/50th a day for a start!), but doing that to your customer-facing production sites sounds bonkers - what class of problems is there that that's the best way to prevent them?
Yes it's aspirational, and of course you have to do batching when the rate of commits gets too high. Many companies don't actually end up at the end of the spectrum with full continuous delivery; most of the value is captured by building what's needed _to be able to_ if you chose to, and doing say daily or hourly automated deploys.
If you're not familiar with the general arguments for why CD might be desirable, a good place to start would be the books Continuous Delivery [https://smile.amazon.com/Continuous-Delivery-Deployment-Auto...] or Accelerate [https://smile.amazon.com/Accelerate-Software-Performing-Tech...]. The TLDR would be that most deploy problems in practice turn out to be because the process is manual, infrequently-used, and/or unreliable, and by practicing "if it hurts, do it more often" you are forced to iron out those kinks. For example you can't deploy automatically without a good acceptance/smoke test suite, and without solid DB migration testing/linting/practices, to name just two of many. You could just build all these things without trying to do CD, but in practice it seems that it's quite hard for organizations to prioritize this work in isolation, and the goal of CD seems to provide a fairly clear map to a much higher-quality deploy process. The practice of deploying more frequently to flush out issues, is also quite unintuitive as you note.
Yes I do get the general argument for CD (though I have issues with how universally it can be applied based on the sort of BVT I've had to do for SaaS platforms I worked on in the past, where it just wasn't possible to safely release new versions of key components without needing several days' worth of testing, most of which was waiting for verification that interactions with banks etc. had worked as expected).
Agreed, like almost everything it’s not universally applicable.
If you have external contracts that are onerous to validate then it can be quite expensive to build high-fidelity internal representations of those contracts. Could still be ROI positive to do so!
In most cases the contracts are either internal or simple (non-temporal) external contracts like a REST API, so most orgs benefit from moving in this direction.
We came from very different lands I guess. Batching 50 commits with no easy way of knowing which commit caused the issue seems scarier to me, in case of an issue. And if you code correctly and have warmup requests, your deployment could be without any downtime. Obviously there will be some extra resource usage because of unready resources.
But if you're doing 50+ deploys a day, how can there possibly enough time to know which commit caused whatever issue that gets noticed anyway?
If those 50 deploys are done during regular business hours that's 9 minutes per deploy on average!
Reply to wizofaus: 9 minutes is more than much more than enough to revert, at least in our setup. Specially if some automated alert triggers, then we could revert well within a minute.
That’s what staging environments are for : deploying every commit to check everything is fine, doing the QA and get the alerts.
Doing this in production is being disrespectful of your users. Users are not waiting for your PRs to be approved and merged, they can be as delighted by your new button’s shade of lightgrey if it is released some days (even weeks !) after.
That's my feeling, that you get all the benefits of rapid/frequent deployment by doing so to non-production environments, and deploy to production when you have a good reason to do so.
But either way I don't really understand being driven by a desire to ensure every commit results in a deploy. It boggles my mind that a complex product, especially involving mobile platforms, could be fully built, ci-tested, deployed then e2e tested within 10 minutes, even with a massive array of distributed computing power. Mostly due to 3rd party tools that give you no choice but to wait 20 minutes or more for certain steps to complete. I'm intrigued to know how it's done.
More advanced/modern tool chains like k8s can help; rollout can be done in minutes. Also makes it easier to do blue/green deploys if you’re serving high throughput services where you can get an error signal from upgrading a small fraction of your service workloads.
> It has approximately zero to do with UX and product design/iteration approaches.
The connection is that if the commit changes how the product works in a way noticeable to the user, then the user must suffer the changes frequently. Since changes incur a cost to the user, it's requiring the user to constantly be paying those costs.
If the deployment happened less frequently, then the user would incur the cost less frequently, and perhaps in a way they could schedule when to take the change on so it is the least disruptive to them.
I've worked with software that has to work. Our customers put it in a lab where they beat on it in simulation for 3 more, months - any bugs and they stuck with the old version. (which is one reason land lines were so reliable in the 1990s) We did a code freeze 6 months before release in hopes of catching everything.
Most software nobody cares if it works or even is possible for untrained humans to use, and it shows.
The glut of isolated B2C SaaS products/developers has absolutely destroyed any communal valuation of “known issues.” True in-house heads still remember.
Rapid release is fine but companies will try and cheat:
1. Release often
2. When a release goes badly (and they will) do some version of '5 whys' and fix the root cause.
3. Continue to release often
That step 2 is often missing and what usually happens is that no engineering time is committed to the solution. Instead they implement more process which costs nothing, doesn't solve the issue and slows development.
The YouTube app for tvOS is a perfect example of this. Every single release that fixes one annoying bug introduces another equally annoying bug.
Just this summer, we went through these bugs one after the other:
1: Putting the ATV to sleep after finishing a video results in in a black screen with the video's audio playing from the beginning. This can be rectified by backing out of the video.
2: Putting the ATV to sleep while watching a video will cause the video output to freeze for 10 seconds or so when the ATV is turned back on. Eventually tvOS recognizes the app as unresponsive and kills it.
3: "Right" on the circle pad can no longer open the subscriptions tab, even though it still works on all other tabs. You have to use the center button.
4: When starting video playback, the first several seconds of sound are cut off, requiring the user to rewind if they contained anything important.
I don't think any of these issues existed simultaneously. Issue 4 started happening with the exact same release that fixed issue 3.
I like new releases when they mean new features and bug fixes. Not so much when it’s just a designer messing around with the UI and graphic design or some “product manager” removing or changing features on a whim, due to A/B testing or whatever.
New releases should almost always only add features or fix bugs.
Yes the end user experience is not good with too frequent updates. Vs code asking me to restart every single week for updates is just a bad user experience. I would much prefer monthly or quarterly bigger updates.
> I miss the days when I could plan ahead for new versions, so I could time them to have the least possible disruption to my life.
That's one of the many advantages of desktop apps: unless it's a bad player that forces the update down your throat, it is you who control if and when to upgrade, not the vendor, who may or may not have their interests aligned with yours.
(It's a bit similar on the mobile if you disable automatics updates, provided that you can ignore that obnoxious red dot forever.)
I think this sounds like an issue with the change management part of the cycle. It is also a bit of a chicken and egg problem, where agile development has created an environment where changing requirements from customer/user are almost expected, and now we justify those by using agile more and more. Apps also look to implement ever growing numbers of features that no one really cares about, but sound like a good idea in the weekly leadership call.
I think a multi-pronged approach works best. A pipeline for users who want the latest changes, and a more stable pipeline for users who dislike change and mostly just want security updates and bug fixes.
The trick would be giving users more distinct choices. For example, with something like Chrome the choice is between extremely frequent and just plain frequent. They don't really offer any long-term version.
Look for software that has "long term stable" releases and switch to that. Lots of software does. People are more aware of it at the enterprise level (because we have to manage it) but you might be surprised. Chrome has (FINALLY) released an LTS version and Firefox has a great one. Even Windows 10 has it as an option (although a bit difficult to access)
All distribution channels have a way to offer beta releases. But not all teams take advantage of it. It might a bit more work for SASS companies, but there is really no reason for downloadable apps to push each minor bugfix to the customer as they happen. Let beta users have the weekly/fortnightly updates and bundle then into major/minor releases once every 6 months. Jetbrains does it well IMO.
This used to be a endemic issue for the mobile apps. And now even the desktop apps have started doing it. I am looking at you web browsers.
- Agile
- DevOps
- Continuous integration
- Continuous deployment
- Rapid release
- Growth hacking, if you have 10+ employees, A/B testing, heatmaps.
I’ve been working alone for 12 years and all those concepts were already working in the last large corp I was in.
What trends am I missing by being disconnected from the employment market? Am I missing anything or do we still develop like it’s 2010? Is there no new process or evolution on the style of management of software? Is Agile the final form of product management?
The notable trend after DevOps was Kubernetes since 2020, and the notable trend for the frontend was the generalization of NPM-based programming, from React to Typescript, which enabled larger teams and more stable UIs, at the cost of 1GB of JS in the browser.
But they don’t seem as earth-shattering as missing the Agile boat.
If I were to generalize, I’d say most of the companies I’m picking on fall into one of two categories:
They believe in DevOps ideas, but they’re unwilling to invest in changing their ways.
They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all.
Isn't it amazing how one could take "DevOps" and substitute "Agile" and it would fit exactly?
> Rapid release means I can no longer rely on the software that engages in it
I’ve been thinking that UI changes should follow something like semver like APIs. UI is a contract between you and your users and completely shifting layouts should be seen as a breaking change.
These breaking changes should be avoided more critical the product is. E.g. health management app for elders.
It's better than nothing, but it isn't really sufficient. Typically, the "stable" channel still releases too frequently.
What would address my concern is if it were possible to skip releases, however rare or common they are, as I wish. But that raises an entirely different Bad Thing in the modern software industry -- that security updates are increasingly inseparable from other updates.
It can in some contexts, but I've never seen a "release channels" approach for e.g. a configuration UI. I'm not sure if that's because it's fundamentally incompatible with business needs or it's just not something UX devs are aware of.
It adds a huge support burden to devs and UX, as they need to support an increasing number of different UX versions or basically never change anything (which is bad for their career).
Supporting multiple versions adds a huge burden, for sure, but I'm not sure it's bigger for a configuration UI or webpage than it is for any other application that has some kind of backend or data shared across the versions.
I hate it as a release engineer. The corners cut to do "rapid releases" usually means sacrificing regression testing which results in angry customers blowing up phones and inboxes.
In a previous job, I joined a team that was supposed to be introducing DevOps to the organisation.
It started out well -- we spent a few months hacking with Terraform, Docker, Vagrant, Kubernetes, and related technologies to implement an infrastructure-as-code approach -- automating the process of provisioning and decommissioning servers, and building a setup where development teams could deploy updates with a simple git push.
Unfortunately it all went downhill fairly rapidly. We ended up spending the majority of our time manually applying security patches to a bunch of snowflake servers that we'd lifted-and-shifted from another hosting provider to AWS, and fielding support requests from the development teams. Within a year, we were being told in no uncertain terms by our project manager that we were an operations team, not a development team.
It felt like a complete bait-and-switch. Within two years, I had left the organisation in question and moved on to a new job elsewhere doing actual development again. Last I heard, the entire team had been disbanded.
It sounds like the author of this article must have had a very similar experience. I wonder just how common it is. It seems that in many places, "DevOps" is all Ops and no Dev.
> It seems that in many places, "DevOps" is all Ops and no Dev.
This was definitely my experience at my last couple of jobs. At my last one they "scaled out their DevOps team" by hiring tons of juniors with next to no software development background. And then they "empowered" teams by assigning the juniors to each dev group. As a result, we ended up having to train them how to do their core jobs, which... went about as well as you'd think.
Eventually, there was an attempt to shift everyone to kubernetes. They had a special "DevOps" team build a layer on top of it to handle the non-kubernetes aspects of deployment as well, and somehow manage them together using Helm. If you're wondering "what the hell does that mean", well, it turned out nobody really knew. These "DevOps" engineers didn't really seem to understand kubernetes core concepts, and just ended up hacking away with some scripts on top of terraform delivered via Helm until something got configured. It was incredibly slow to deliver, hard to use, and I just stayed away from it until some exec threw down the mandates. (And then everyone started quitting because it was an absolute disaster.)
Ultimately, these are really stories about bad management, not really anything to do with DevOps. But that's how these things roll - some new hot concept comes to town, and bad managers try to adopt the term, without really understanding it.
What you just described is the exact same reason I started working on https://stacktape.com 3 years ago.
When doing the market research, I talked to ~150-200 companies (mostly SMBs). Everyone was trying to "do DevOps". But the complexity of running a Kubernetes cluster (or a custom AWS setup using ECS) is just overwhelming for most of the teams.
In most cases, the DevOps/platform team requires atleast 2-3 experienced people that have successfully done this before.
Considering how few experienced DevOps people with such kind of experience are currently available on the market, it's no surprise that only the "coolest" companies around get to hire these people. These successful companies then write blogposts about how successful they were.
And the circle starts all over again. Less successful companies follow them and (in most cases) fail.
Most of these companies don't admit it, or don't admit it soon enough. They also don't write blogposts about their failures.
From my experience and research, roughly 70-80% of companies fail to deliver the expected results (or deliver them with order of magnitue more effort than initially expected). Yet 90-95% of the content we get to read about these topics is overwhelmingly positive.
PS.: If you don't have an A-tier DevOps teams, check out https://stacktape.com. I promise it will make your life easier.
It's not always bad managers or management. The fancy new hotness gets pushed onto them by smooth-talking evangelists. For every criticism or question they have a snazzy little quip and retort that makes you look like an idiot for not knowing "the obviousness" of your errors and how this new fad/tech/framework/methodology solves it. And if that retort doesn't work, they just tell you "it's standard in the industry, dunno what you want me to tell you".
And from the outside, this all just looks like resume padding and job-security. Devops is the new priesthood, subscribe or be reduced to irrelevance by config warriors.
The fancy new hotness gets pushed onto them by smooth-talking evangelists.
Gets pushed onto them top-down. This means there is no real competition, no real empiricism, no real comparative merit involved in the switch.
How should never be imposed top-down. It's only goals and how success is measured which should be top-down. The classic example of this was when Jeff Bezos mandated that all Amazon software systems should be accessible by APIs over the internal network, or else one would be fired.
Whenever I've seen How imposed top down, I've only seen the lower level managers talk about how they could put off and passive aggressively stymie the initiative.
> We ended up spending the majority of our time … fielding support requests from the development teams
This has been my experience at 3 different small-medium companies now. A too small DevOps team suddenly is in the critical path for even the most trivial software task, then engineering productivity grinds to a halt. I think a much better pattern would be to enable dev teams to self serve. Set up the required infrastructure and guard rails, then let teams handle their own deployments and infrastructure. Give people what they need to do it themselves instead of having to open a support ticket for everything.
> I think a much better pattern would be to enable dev teams to self serve. Set up the required infrastructure and guard rails, then let teams handle their own deployments and infrastructure.
I think that's how DevOps is actually supposed to be done in the first place. You view Ops -- and the code used to manage and support it -- as a product, and get a specialised team of experienced Devs (and architects) to build it.
Once you've got the basic infrastructure and architecture in place, you then train up the individual development teams to customise it, extend it and troubleshoot it as they need to. In much the same way as they do with any other software product.
My experience is what inevitably happens is the ops team goes an writes a layer on top of Kubernetes, and now instead of dealing with Kubernetes you're dealing with a half baked poorly written abstraction with zero documentation and no StackOverflow on top Kubernetes. So you need to become an expert in both.
Most organizations don't have the resources, mindset, or skills to support a software library product and should only do it as a last resort.
Most developers won’t know how to do it without footgunning themselves constantly is the problem.
If the dev ops team is staffed enough to develop integrations that won’t allow that AND won’t get in the way, and then train folks and ‘keep the line’ enough to stop the scope creep - they’re probably not at a shitty small/medium sized shop.
1. An operations team with a different name.
2. A platform team with a different name.
3. A development team with a different name.
4. A "CI/CD team".
5. A role (ex. "dev who automates ops", "ops who codes", "support specialist who codes", all three in one).
6. A chart that the delivery manager maintains.
Here is what DevOps should actually be:
1. Delivering rapidly and consistently with extremely high levels of confidence.
2. The right people address problems correctly, immediately, the first time, and fix it so it doesn't happen again.
3. That's it.
Increased velocity is what business get promised, yes. It’s what you want.
The reality is that you can’t just magically make that happen.
Draw a line. Now draw the rest of the owl! Easy~
The problem has never been understanding what the desired state is… it’s always been that getting from the current state to the desired state is very very hard, and continually road blocked by:
- people who don’t want to learn new skills
- people who don’t want to seed control they currently have (over process and product)
- a lack of clarity on who is responsible for what systems
Devops is a load of hype.
There’s never been any reliable process to move to rapid delivery from it.
Yes, some teams have managed to get something that works, and there are a lot of tools and a lot of training which has resulted in over all better SRE processes.
…but by and large, that’s because of using better tools (eg. infrastructure as code) not because of devops.
When was the last time you had a “devops” guy you got to do something for you?
Right. That’s ops.
When was the last time something broke and the people responsible for making sure it never happened again were the “platform team” or an SRE?
Ops again.
You had all that before you started your devops journey.
It’s all just ops, with slightly better tooling, less outages, higher reliability and absolutely zero increase in product velocity.
Devops was the promise that by bridging operations and development you could get high reliability and faster iteration by having teams that could “cut through” the red tape and get things done.
That appears, largely, not to work.
Yes, developers that understand systems tend to build more reliable software.
No, it is not faster to do it that way, and the transition will be painful, and, because businesses mostly care about iteration speed more than reliability, even a technical success, it often a fails to deliver on its business value.
It has worked well for organizations that embraced it. It hasn't worked well for organizations that paid it lip service. That's the way of the world. There is a path laid out by DevOps methods and dozens of ways to get there, but the path doesn't walk itself.
Note that this admits strategies where nothing of consequence is ever delivered (but each deploy has some quantifiable and measured churn), and the people that break stuff get credit for fixing the stuff they broke.
I've watched this particular breed of organizational cancer destroy many companies and products.
The end game is that people creating useless, but highly visible churn get promoted, as do the ones that repeatedly break stuff. Even if that doesn't happen, the engineers that want to build stuff inevitability flee.
That's where value chain management comes in. If you can't show business value being delivered, there's no point to any of it.
It's also worth acknowledging when you don't need DevOps. Banks, for example, shouldn't need it. Their entire purpose is to be slow and reliable. Most of their money is literally just old people keeping lots of money in one place and not touching it. They shouldn't need to churn on features and ship constantly.
I've not seen one place that has escaped this problem though.
Something is either old and nobody feels like fixing it or something doesn't fit into the current constraints of the platform featureset. So they build something on their own, probably undocumented and without consulting the supposed DevOps team, but still using parts of the platform that have not been declared an API. But when you want to exercise the freedom you supposedly built with your platform, all these edge cases fall back on you and inhibit change. "We can't change the ingress controller, we rely on this implicit behavior of it" "You can't change how database credentials are provisioned, we pulled these from the Cloud SQL console and comitted them to git". And facilitating any change as soon as you can cover the use-case is a fight with stakeholders and POs that I usually have no nerve for. "Why do we need to do anything??? It works?". And then you get blamed when it breaks. I love this job.
I was one of the biggest proponents of DevOps (even started and the DevOps orange county meetup) but ... after working at AWS for 5+ years, my entire perspective shifted.
While DevOps does work in some organizations, sometimes what you really need to do is operationalize your software development team. Don't get me wrong, system developers/engineers/operations are needed but for sometimes for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).
> for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).
That's actual-devops. Grandparent poster has worked at firms which have abandoned all its lessons and just use it to mean something like "regular old ops teams, but they use software". So he's using the term like all the big companies do, basically.
This is also where you will find things like a "CI/CD" team that doesn't do continuous delivery: they just build Jenkins pipelines, and run them as part of the manual cluster bringup every 2-4 weeks. Or "RESTful" to mean "APIs which use JSON and HTTP verbs" regardless of any hypermedia. And let's not even get started on "agile"...
Maybe we need to come up with a new term to be more specific. “DevOps for developers” vs. “DevOps for operators” or something like that. (DevDevOps va OpsDevOps?)
I always took DevOps to be some combination of 1) applying dev best practices to ops, which could still in theory be done by a separate team, but more 2) unifying dev and ops in one team/role, perhaps enabling developers to own it by providing tools and abstractions that developers can work with, and workflows that unify the processes and improve collaboration by lowering cycle time.
It seems that the “bad” DevOps is OpsDevOps where the old ops team just gets some rebranding, and maybe some new tools like Terraform. I’ve always found the “DevDevOps” approach to be quite empowering. Giving developers a tool like k8s so they can abstract away the hard half of ops, and understand the rest.
In my experience it went poorly because us devs are now handling our own deployments and building alerting systems and such, but the "devops" people won't give us the level of access necessary to do that with tools designed for the job, so we end up building custom ops-related tooling into the app and writing backdoors that let us use it without having wake the ops folk.
When you're accustomed to having to build everything you need, and you're tasked with ops work, you end up reinventing a lot of wheels in a rather weird way and compromising your app in the process. Or at least that's what we did.
Yeah, to me, the core of the ‘DevOps’ philosophy is that the dev teams owns the product end to end, meaning they write it and are responsible for it in production.
There seems to be an increasing disconnect between the marketing value of a term ("DevOps") and what we're actually supposed to be doing, which might best be described as blameless accountability.
Even in information security it basically revolves around the same things before you can even do any sensible work and improvements to processes.
- Know what you have
- Know where it is
- Know what it is doing
- Know what it is supposed to be doing
If you don't have any of those pieces you're just guessing and hoping for the best. For software that might be "know what services you are building, know where you're running them, know how they are doing, know how they are supposed to be doing". But if we call that "DevOps" is suddenly marketable.
They are only hard if they don't get the same priority and attention you'd see in regulated situations. Replace software or information security with "bars of gold" and suddenly it's not so hard.
That's because marketing why you should know the what/where/how/why of your gold to business decision makers is apparently much more palatable than abstract concepts found in IT, even if the value is the same.
If you get the resources (including time, money and manpower) do just do it, it can be done. But you usually don't get all three (and maybe not even one), and therefore you're stuck with a task that cannot be executed well.
Did general aviation become cheaper or more available to folks when it got more heavily regulated?
Fatality rates dropped, but it basically froze the technology and capability into the late 70’s/early 80’s and locked in just a few companies. The overall number of people working in that industry is tiny now. It’s almost a complete dead end.
It’s the classic trade off - cheaper, better, faster. Pick any 2 if you’re lucky. Most only get to pick 1.
Yep, and that is generally also why IT is the way it currently is. Most projects aren't all that complex (in business logic terms), it's mostly CRUD interfaces and a fancy viewer for the end users anyway. But industries that are still on mainframes are almost completely the opposite where every single element is tagged, catalogued, tested, described and traceable at any point in time. With a price, rigidity and lead time to match, of course.
On one hand we don't all want to buy mainframes, on the other hand, doing it as cheap and unmeasured as possible makes everything flakey, demanding on individuals to keep it running, insecure, and opaque to everyone involved. Too bad there isn't really any middle ground, it's almost like it's either "pretend to be a mainframe" or "do it as cheap as possible".
Regarding the aviation analogy: it definitely doesn't become cheaper due to regulation (it did become cheaper due to subsidies), but I'd rather have a system that's regulated (or forced) into safety than having it be optional and left to a sociopath (i.e. a for-profit publicly traded company - all the benefits of pretending to be a person but none of the responsibilities or limits to govern it - except the law and regulation).
No, "Ops" is legacy siloing of departments and by extension, tasks. Imagine driving a car, but it takes one person to do the steering and another person to do throttling. Seems pretty dumb, doesn't it?
On larger vehicles like ships, you have exactly that "legacy siloing" of responsibilities. You don't need the person responsible for navigation and steering run down into the engine room or to the rudder to check whether everything works as expected there.
True, but the analogy for a ship would be: the person cleaning the deck isn't allowed to get his own bucket, that's the bucket operator's job. Pretty silly again.
The original definition of DevOps was what you describe in your second paragraph - development teams being responsible for operations.
Now DevOps means "We've created a DevOps team! They do DevOps!" Frequently its the managers of old school operation teams changing their names to DevOps teams to get a promotion.
"Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it." - Werner Vogels, Amazon CTO (2006)
DevOps meant that Developers are also responsible for Operations. But that meaning is lost to time, just like hacking meaning programming.
It may seem pedantic, but it's an important difference: that's _not_ what the original definition was. Instead, it was a cross-functional team of developers and operators being responsible for a whole system.
Yes, I should not have said a "development team", as it makes a distinction between "people who create new software" and "people who run that software in production environment". Your definition is better.
> development team's operational posture (e.g. run books, alarming, monitoring.
Isn't DevOps in large part forcing the developers to have responsibility over the code they push and giving short and frequent pushes to easily rectify problems?
Making developers do support work reduces support in the long run because they'll get frustrated and automate solutions.
Yeah, I feel like the term has been taken over to mean using kubernetes and having continuous deployment pipelines for your microservices.
My understanding of root of the term is about ownership and responsibility, where the team that develops the app also owns how the app deploys and runs in production, including monitoring and responding to issues.
If the team decides that the right way to deploy their app is once every two weeks using a shell script and a zip file, cool. As long as it's documented and monitored so you know if it's working.
Yeah, what's worked well for me is having ops or security people draw up guidelines (for security, it may be something as trivial as require ssh key auth, for ops it may be: write a script that matches an existing deployment procedure and we can get your app up and running very quickly).
And/or become burnt out and develop a severe ‘learned helpness’ complex. It depends on the environment they’re in, and if they have the tools and support to be able to fix issues.
> for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).
But that's literally what DevOps is? DevOps isn't a special DevOps team that writes terraform.
I was a network engineer at my previous job, I was fed up with doing things manually so I learned to code better. Now I'm a full stack developer writing code for the network team of a different company.
One thing that has changed drastically over the last 15 years is the sharp divide between IT and development. It used to be that developers were clueless how to deploy their own apps, and IT was terrified of writing code preferring to do things manually, or purchase something prepackaged. Now I see developers that understand firewalls and load balancers, and IT folk that can script stuff out for themselves. I think we have the push behind DevOps to thank for that progress, and while there is still a long ways to go, I think it's a good thing overall.
> ...writing code for the network team of a different company...
Something like is the kind of job that i think i want to do next. It need not be for a network team...but i'm headed more for back-end work and these days i rather engage with internal teams.
Separately, to your comment on sharp divide...The sharp divide that i have seen is a lack of cohesion for the following 4 IT groups: first-level support which gets sent to off-shore; the folks who (honestly have the most face time with internal clients who) roam around the office fixing desktops and building network/wiring, and setting up/fixing A/V equipment in conference rooms; sys admins; and developers. Furthermore, in the last 2 companies that i worked for, there has been a push to "centralize" certain functions like IT and HR to "shared services"...but at the end, none of these groups seem centralized! They all now work more separated than ever before...and most of the interaction between each of these groups - and even between the 4 subgroups i noted above in IT alone - mostly communicate via tickets, or some other thick-layered mechanism. Sure, they've been centralized in that these people were pulled from some respective business unit, and brought under a more single umbrella...but in the end, these workers hate it, the effecicny drops, more bureacracy is created (which creates no value)...and some executive gets to lead the shared services...and of course some management consultant company gets their payday. Oh, and yeah, IT at these orgs definitely does not use any of the good parts of devops, agilescrum whatever. Some orgs try and adopt some cookiecutter approach only to have it create yet more bureacracy...but "hey, we're agile!" /sigh
I worked at a large internet company in the mid-late 00s, and the typical team composition was a handful of developers, a product manager, a designer (if you worked on customer facing UI), a project manager (for large enough projects), QAs, and a SRE.
Every team had dedicated engineers whose sole purpose was to place purchase orders for servers, set up and manage databases, deploy changes, monitor uptime and more. And this was standard practice across the industry, as soon as your company outgrew a shared host.
Even the thought of something like this would be absurd today. So I'd flip the "no one admits they don't to it" around – your company does DevOps already, whether you realize/admit it or not.
> the outcome of applying the most trusted principles from the domain
of physical manufacturing and leadership to the IT value stream. DevOps relies
on bodies of knowledge from Lean, Theory of Constraints, the Toyota
Production System, resilience engineering, learning organizations, safety culture,
human factors, and many others.
Anybody that's worked in an organization that "uses DevOps" or has "DevOps Engineers" knows that this simply isn't true in the vast majority of circumstances. Too often DevOps is just a new name for an ops team. "True" DevOps is all about aligning incentives, sharing responsibilities, and fostering collaboration between people who build and maintain systems whereas "DevOps" (with air quotes) is just a new spin on being a sysadmin with a greater emphasis on AWS and Terraform.
Because they do both jobs, folks understand both sides, and work together to make things work better end to end for them and for customers.
Alternatives include siloed dev teams that throw shitty software that crashes in production all the time to an overworked ops team that takes all the blame.
And overworked ops teams that have gone pager-blind and only care if a VP yells at them, so ignore all but the most major issues - after they already happened.
No, it's not. The definition they provided for DevOps was markedly different:
> DevOps is [...] the outcome of applying the most trusted principles from the domain of physical manufacturing and leadership to the IT value stream. DevOps relies on bodies of knowledge from Lean, Theory of Constraints, the Toyota Production System, resilience engineering, learning organizations, safety culture, human factors, and many others.
If somebody reads that sentence - without already having knowledge about what "DevOps" means - are they going to think "oh yes, I see, so DevOps means having the same team do both Dev and Ops work"? Of course not! They're not going to have any idea what DevOps means, other than the fact that it's something consultants sell with word soup.
The fact that engineers are not taught those topics are why DevOps has failed. We're expecting whole organizations to magically improve when they don't understand how. I can count on one hand the number of people I have met who understand OP's comment.
I agree the terms have become pointless. I have been thinking about opening an actual school for DevOps, or at least IT Ops. There is virtually no education for it, but it's a complex subject matter that requires years of experience. The SRE books are a peek under the covers but leave so much unsaid that it's really only useful at "doing operations", not digital transformation.
To properly build and run a high-performing organization, you need to understand at least 1) the entire history (up to today) of software development practice and production operations, 2) all of the competing requirements of product design, development, and operation, 3) architectural best practices, 4) the industry standards that apply to the work, and 5) tech product-aligned business management methods.
Basically you need a generalist who has 15+ years experience and has tried and failed at everything under the sun. I think of it like carpentry before there was a building code, and before the introduction of apprentices and licenses. Trying to build high-performance orgs without a master builder to show you is like trying to build skyscrapers after having built your own home (and having never seen a skyscraper). Or worse, taking an old barn (non-tech enterprise) and trying to build a skyscraper out of that. Without a trade school it will continue to be just a craft.
Most companies that have sufficient talent and organization/operational setup don't see a need for DevOps.
There seem to be a lot of other companies that are sucked in by its promises and think that it will solve their issues. Surprise... this didn't fix your shitty policies, disorganized/unempowered org structure, or cheap talent (cause you're unwilling to pay or train).
I assume there are a few places that can benefit from this. Maybe some medium to large tech companies with great procedures, documentation, and an empowered org structure. Especially if they're paying enough for people who can manage it. The other is probably for very small places wirh fairly simple tech where you can't afford entire teams of people so you empower someone to really own the system top to bottom (without ever calling it DevOps).
IME, letting developers have free reign over service architecture/deployment/CI/observability is a bad idea. Especially when every team comes up with their own option.
You need some sort of Devops (best practices if not a team) to provide guidance and help implement standards for these.
I agree, if we're talking about places big enough to have multiple teams. The thing I was pointing out is that most places that have multiple teams are terrible at setting up best practices. They'd be better off with a centralized "environments" team provisioning the server, network, etc so the dev teams can work on the actually implementation. At least it's easier to maintain standards, or at least consistency, across apps if the low/mid level config is handled by one team.
When I was talking about small places, I mean like tiny mom and pop non-tech companies with an IT department of 1-5 people.
Every company that has a software product also has DevOps, whether or not they call it that. If someone manages your cloud infrastructure, they're doing devops. If someone is coming up with CI/CD setups, they're doing devops. If someone came up with SOP for distributing new releases, they're doing devops.
Whether or not anyone's title has "DevOps" in it, devops tasks are still devops tasks. By the same logic, even though I watered the office plants this morning, I am still a firmware engineer- not PlantOps.
The real trap of DevOps is the assumption that it's anything new or non-essential. DevOps has existed since the notion of "software as a product" came into existence, we just didn't have a name for it yet.
None of this is to say you need a dedicated DevOps person (or PlantOps, for that matter). DevOps is an integral part of any software product, even if no specific member of the development team is dedicated solely to DevOps tasks.
I don't think those are DevOps by themselves. You have to be doing the Dev part of the system in addition to the Ops part. The traditional format is that some team or tram member would do ops work while others did the dev work.
Recently I've been reading "When Agile Gets Physical" [1] and "Extreme Programming Explained":
1. "Extreme Programming Explained" explains the importance of separating values from principles and practices.
2. "When Agile Gets Physical" unpacks why using agile software best practices are horrible for hardware. This is "duh" if you're a hardware person, but the book goes back to agile/lean first principles, and then builds up an approach for agile hardware.
It got me thinking that we often conflate values and practices. Blinding adopting Scrum or DevOps without understanding some of the underlying value and principles leads to problems. If you understand the foundational principles, you're in a much better position to adopt/reject certain practices for your given situation.
In reality, most organizations just adopt a few agile practices, while rejecting the basic values.
Not directly related to the question, but I personally hate the term "Sprint". Physiologically, a sprint involve by definition is a short-term high effort activity that can't be sustained without a break.
Does anyone use a different term that doesn't imply working oneself to death?
Just changing the word isn't going to change the practice. You're supposed to take breaks between sprints. There needs to be planning and alignment to an easily-obtained goal, which requires time. But most people don't know how to do Scrum and hand-waive the goal, the planning, the retro. The end result is unsustainable work with no specific goal in mind that exceeds the sprint cycle. Or 20 epics all in progress at once. Or business goals that shift and put unobtainable expectations on teams.
It's not explicitly described. "The break" (imo) is giving enough time to planning at the beginning, and actually dealing with the review & retro items. The length of the sprint can be shortened so that there is more learning and planning. Sprints are cancelled if the goal becomes obsolete.
When I was a wee lad, “programmers” wrote software and sorta tested it and then kicked it over the fence to “systems administrators”, who chain-smoked and were very, very un-woke.
This was a bad system.
To the extent that “DevOps” means anything, it’s that the spheres of capability and responsibility need to have tons of overlap: SWE’s need to be in the oncall loop and know how the filesystem works, and SRE’s need to be serious coders.
That’s it: everyone codes, everyone operates. We all automate where feasible.
Sidebar: What’s with the crazy surge in BuzzFeed-style titles on the front page? It’s not a federal matter but it’s not a slight change YoY either.
The test cases for Unix must have been kept on some tapes that were lost, because a test case is hardly to be seen in the historic sources you can find online.
My team at Nasdaq (I work on other products, not the stock exchange) does DevOps surprisingly well for not being a Real Tech Company TM. I don't know that the business sees benefits, as it meant that all of the devs on my team have had to learn Terraform, Helm, AWS, and Kubernetes in addition to the actual SpringBoot Java application code, so our feature velocity dropped quite a bit, but it was fun to learn, and it's probably better to have people who understand the full deployment and not just pass blame between teams during incidents.
Oh man, that's the trap. I do like to learn new things, I swear. But at work I want to focus on what I do best: feature development. Sure, I do know terraform and k8s and what not (because I play with them via my side projects), but at work don't give me the Ops/Platform hat please! It's cheap labor (your company doesn't want to hire a dedicated platform engineer to handle terraform/k8s/etc. and instead assigns you (the developer) that responsibility claiming "you are gonna have fun learning" and paying you 25% more in the best scenario)
I don't know if you're right or not, but I do think your hesitancy mirrors why a lot of organizations are unable to adopt DevOps fully: many people don't want to wear a bunch of hats. They just want to do their one thing, and I can't really blame them.
But I don't think the original DevOps (TM) specified that every person was required to work on every part of the stack? Just that you need to have full-stack skills and responsibilities within the same team?
That said, I probably wouldn't want to hire someone with just one hat. It's ok to have a preferential hat, it's not ok to refuse to wear any other.
> But I don't think the original DevOps (TM) specified that every person was required to work on every part of the stack? Just that you need to have full-stack skills and responsibilities within the same team?
Then you have Ops person integrated in every dev team/squad, then those Ops people have their own meetings to talk about Ops things. Then there aren't enough Ops people to staff every dev team (maybe they are on vacation, perhaps just not enough in general), so they start sharing them. Next they need to work on some special Ops tasks, so they want to have their own board. Oops, now you have Ops team.
The point of DevOps is that the team is responsible for code being shipped, and after they shipped it, so everyone needs to know how to work in the stack that runs your app (Kubernetes, Nomad, AWS Beanstalk, bare-metal with Ansible or whatever it is you use). That means devs don't go to some other team to fix their CI or CD pipeline.
I worked in a company that was doing DevOps, we had no Ops team, we had a few people who only worked on infra, but they were part of our team. We didn't call it anything, it was just how to work, about 3/4 of the team was capable of setting up infrastructure and working on Jenkins.
I also worked in the company that "did DevOps" had a "DevOps team" and devs knew absolutely nothing about what happens to their code after they did `git push` (that if they knew how to do `git push` without asking "DevOps team").
Another company "for sure did DevOps right", they didn't have "DevOps team" instead they had "Infra team" and infra team was responsible for bare metal provisioning. Devs could write CI pipelines and stuff. However, devs had zero responsibility for code after they shipped it - it was infra team that would be woken up by nightly Pager Duty alert.
This sounds like a manager's ideal. It's a luxury to demand people wear multiple hats up to the point the market allows you to turn the luxury into a commonality.
If it was up to bosses, they'd rather have jokers ready to switch from customer support, to development, to sales, and back.
It's ok until the market decide it is no longer ok. So far, it looks like all these "hats" are complicated enough for the average to make the market think twice about demanding that combination of breadth and depth of skills.
I'm with you with a minor adjustment. I'd call it not "feature development" but software development and programming languages. Two majors points:
* as a developer my free time should be spent on keeping up with my ecosystem (e.g. new JDK versions, Scala/Kotlin taking over Java, Akka-like modern libraries, the latest Flink features) and keeping an eye on possible alternatives (e.g. are golang generics good enough to make it competitive with Java?).
* once they standardize on a single DSL sign me up. I don't want to constantly waste my time on switching to CloudFormation, then to Terraform, then to Helm charts. Sometimes even moving to a different team in the same company requires it. Life is too short to be a full time YAML developer. Also, AWS CDK is the only option that at least resembles programming and it's apparently the least popular one.
Same here. I recently had to take on some more roles. It’s fun learning but I don’t perform at the level I could perform when I was mainly writing C# code. Its not even close. It’s good to have some insights into all aspects of the work but somebody who can focus on one area but can perform there with multiple productive of someone who constantly jumps around.
I'm confused about DevOps - I thought it was about dev teams owning the infrastructure, testing and deployment pipelines, so that there is no clear delineation between they person who writes tests, the person who sets up CI/CD pipelines and the person who codes the app.
It has nothing to do with frequency of deployments, and it does not mean that no ops team is necessary for running the show, it's about team autonomy and better understanding of different teams' concerns because everyone is doing a bit of everything.
> If I were to generalize, I’d say most of the companies I’m picking on fall into one of two categories:
> They believe in DevOps ideas, but they’re unwilling to invest in changing their ways.
They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all.
Option number 3 is the biggest one for me: thinking that having a separate "DevOps" team from the Dev team that does ops via modern tools is DevOps. That may be a good approach for your company, but it ain't DevOps.
I think there is a third category - companies that have heard of DevOps, think it means that the development team and operations team talk to one another, which they already do (!) and shut down conversations after that.
I've come across this archetype in several contracting gigs and I find it to be especially pernicious as it gives the incumbents (who usually have more weight in the org) ammo to effectively do nothing.
The author is right that DevOps seems to often mean different things to different people. I would have loved it if they had taken more of an attempt at one definition than just that it's when developers have an understanding of how and where their code is deployed. That does sound sensible, but what other best practices does DevOps usually entail?
DevOps (and Agile) both usually run aground on the same question of trust. If you aren't willing to trust your delivery teams then you aren't going to see the leverage that both practices bring to an organization.
And if you aren't willing to trust your delivery teams then your organization has a much bigger problem than either DevOps or Agile can solve.
The problem with these fad methodologies is that it either works, or you are not doing it properly. The methodology can't be wrong.
It is like saying that C programs never crash, because if they crash, it means you are not using C properly. It is true, but no one will pretend that, everyone knows that under real life circumstances, C programs can crash, and there are plenty of other programming languages that address this issue, at the expense of other things, like performance or simplicity.
But for some reason those who push methodologies don't seem to consider that if that methodology is difficult to implement, or if it is not robust against things like conflicts of interest and office politics, or whatever, then maybe it is a problem in their methodology. It doesn't mean the methodology is bad, just that it doesn't work in every case, and that some tweaking may be necessary.
I don't think anyone ever thought of it like this. Well, some dumb managers probably did. If you want to eliminate QA people, then you need an enormous automated E2E test suite. To have that, you need dedicated people who think of and write those test...one would call it a QA team.
When I point to the moon with my finger and say "moon", the fool looks at my finger and calls it the moon.
Is it more important to mindlessly tag yourself "Agile", with all the bells and whistles adorning the capital A, or would you rather just be agile according to the sought out goals and philosophy, and adapted to your specific situation?
Is it more important to have a clearer definition of "DevOps", so that you can more accurately implement all the prescribed workflows, or would you rather understand the pursued ideals, so that you can pick and choose what you need, according to your specific context, unencumbered by a slew of best, but ultimately useless, practices?
The entire article avoided talking about definitions of DevOps, while rambling a lot about how we have different ideas of what it means and how we need better definitions.
I appreciate the alert that there's a problem here, but there's no substance to the article.
DevOps has lost all meaning as the first paragraph in the article says. Anything it says after that is just semantics. Everyone is doing it on some level, but what that means to each company differs widely.
when John Willis explained it in 2016 devops had enormous potential for the initiated. id guess maybe 10% of anyone who says 'devops' does anything remotely close though.
devops in 2022 means if youre a dev, youre expected to know the bowels of the OS inside and out as well and be willing to spend hours debugging it alongside your normal programming work. if youre in ops? it means debugging someones undocumented middleware from nine years ago while the front of site burns so "rockstar" coders dont jump ship having to do it instead. if youre in dev or ops, it now means standups where you sit down for an hour and talk about where to put your story in jira or how to enter it at all, and where you discuss broad strategy too. instead of leaving after your status update, youre expected to stay. its just another meeting now.
and once boardmembers and leadership realized they could spare themselves the crucifixion of any real accountability after a breech, they turned devops into devSECops, meaning your infosec team sits in awkward silence as you explain why double defined nginx headers arent a feature and the same management team that made them part of the standups also unilaterally declared ssl3 can never be deprecated because customers.
I find the article somewhat ridiculous in that it takes the tone that everyone thinks the idea in question are good.
> They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all
For some cases, rapid deployment might make sense (purely front end stuff). For a lot of cases, I'm firmly of the opinion that it does _not_ make sense. I've never worked on a project that deployed that often, and I don't feel the need to. I have depended on services that deployed that often and "oh, sorry we broke that, we'll fix it later; you'll just have to live with permanently broken data" gets my knickers in a twist every time.
> I don’t buy into all the ideas behind agile, particularly some of the Scrum-specific ceremonies.
That's because a lot of the Scrum people present it as "the" Agile. And a lot of the non-Scrum people are fairly convince that Scrum is awful. To which the Scrum people present "no true Scotsman" arguments.
I'm all for the underlying "listen to your customer, listen often, and change plans as often as necessary to make them happy" ideas of Agile. But every "methodology and process" for it I've seen completely loses that idea. It's infuriating.
Yes, but testing takes time. Automated testing is a wonderful thing, but it only catches those things that the testers have already though of AND have had the time to write automated tests for. Some things are notoriously difficult to automate testings of. And there are always things that don't get considered during the writing of tests; especially those things that only appear when multiple systems are interacting together.
Taking the time to deploy code to a staging server, where QA can check it, and other systems can interact with it, can prove to be very beneficial. And runs contrary to having every push (or every push that passes the automated testing suite) automatically go to production.
I'm a big fan of something like a 2 week release cycle, plus the ability to release quickly when needed (generally same day, but only when necessary). I'm a big fan of automated tests (both unit and integration, both dev and QA written), plus a QA person/team to bang on things in a way developers don't think of, plus a staging environment where all the (staging versions of the) system interact with each other.
I'm not a fan of pushing things to production willy nilly, with a high risk of breaking things "because you can push again with a fix in an hour".
> I think companies feel that if they aren’t going to “do DevOps”, that they need to rebrand the term “DevOps” to mean “what we happen to already do”.
> To me, the most egregious example of this is when a company simply changes the job titles of all its ops people. John use to be an “IT Operations” guy, abut now he’s a “DevOps Engineer”.
Agile is a philosophy, it is not sprints, story points or whatever crazy methodology an expensive Agile Coaching company has come up with. The same applies to DevOps. DevOps could just be a bash script, written by the developer, to feit their codebase. It doesn't have to be fancy or require an extensive amount of investment.
You sound like Mr. Miyagi from the Karate Kid movies (which is a good thing!)...specifically it remains me of that scene when Daniel-San asks him what "belt" he is...and Mr. Miyagi responds "Calvin Klein"! (At some point, Mr. Miyagi explains that his exppertise of Karate is only for the need to survive encounters, and not worry about which "belt" he had achieved.)
SRE seems the better approach. Make an error budget, measure reliability, stop releasing in favor of reliability engineering if the budget is missed, use the free time reduce toil. DevOps seems to degrade to disorganization because the incentives are not aligned between product development and software engineering.
“Make an error budget, measure reliability, stop releasing in favor of reliability engineering if the budget is missed”
These are all things that fall in the realm of ‘ops’.
‘Dev’ops is an approach to ‘doing ops’. It is one that adopts certain practices that are common in ‘dev’elopment - such as defining things programmatically, using automated repeatable processes to generate work product, and using automated testing and verification.
So what you describe as ‘SRE’ doesn’t seem to me to be ‘not devops’. Seems like a set of ideas that would benefit from a devops kind of approach.
If you think SRE and DevOps are opposed to each other, then you might be doing one of them wrong. I headed up SRE at Netflix, and our sister teams wrote the software and managed the tools that enabled the developers to continuously deploy and operate their own services.
We were in charge of reliability, but we did it through chaos testing and accountability, and enhancing the deployment tools to warn devs that they might be doing something high risk, but still letting them do it.
I think you've hit at the heart of the problem, but swapping the DevOps model for SRE is robbing Peter to pay Paul.
Last place I worked with a 100+ person tech team had more of an SRE model. The SRE team were charged with reliability. They had no incentive to delivery features or to improve developer experience. As a result, developer tools sucked, there was s rigid release schedule, every bug was met with a rollback. This was probably an extreme case because the team was also not very good, but either way it was a clear case of misaligned incentives.
The entire point of DevOps was collective ownership with a shared goal of customer value that is function of feature richness, quality and reliability. It's the Holy Grail of efficiency, it's just really hard to build that kind of culture.
This is also very true in the presales engagements by top consulting firms under "Digital Transformation". These presales teams would come and show, on paper, how great they are with the agile practices, devops, microservices, you name it.
However, it's a totally opposite story during the actual implementations. I've witnessed that it took a top consulting firm 4 months to provision a sandbox environment while being 1 month away from the first release. When asked about their proud DevOps practices, you'd hear excuses that are in the blatant lie territory. They knew they could get by by managing the stakeholders instead of really doing what they had promised.
You go up at 5 AM in the morning. You ride your bike to a customer. You pass some security checks like, known guy, yes, no. You are shown to the servers in the cellar. Cause they are safe there. You get access to the admin account after 2 hours. You do some backup to a local disk. Somebody screams something but you can’t hear what because of the noise. You say yes and hopes it was about coffee even though the room is like a sauna. Now you pick up that CD and insert it. You are not allowed to use a USB-drive because it is mutable. Some clicks and the installation begin. And now you wait.
While I generally agree about the annoyance with UI changes, and how that grinds on people, I hope we don't just handwave over the "CD for bug fixes is fine" bits.
It varies product-to-product obviously, but I'd guess there is also a significant chunk of folks that are pleased that their functional annoyances/blockers (bugs) can be fixed quickly, without waiting for a monthly or quarterly release.
These folks might not mind the fluctuating UIs, if it means they get the functional value they need asap.
I hate devops. Mainly because I don’t want to do it. I liked it better when there was a dedicated infrastructure team instead of pushing it down to developers and giving it a new name.
Yeah. As a developer, I need a competent QA and operations team to support me. I'm not competent at those things, but 90% of developers I've worked with are somehow worse at them.
I also need 1-2 months between major context switches, and the ability to declare a subcomponent "ready for release".
Moving forward, I will actively avoid companies that claim to use agile, scrum or devops.
Edit: Also, I want to work on a team that has a good release manager.
If you have a separate DevOps role in your org, arguably you're not doing DevOps. So probably most orgs aren't doing DevOps, at least those that are talking about it.
I wonder how to use a devops flow for machine learning workloads. Something we are working on at my current job is how to balance the speed of releasing together with keeping data, pipeline and model versions aligned (where it for instance takes multiple days to release a new model).
Releasing new features which need a model retraining is currently a very jarring experience.
How about this? Let's describe levels of adoption. Think of the "normal form" concept for databases. Each level has a strict definition, and you can objectively observe which level you are using. And not every level is appropriate at all times.
So maybe a team could say, "we're at DevOps level 2," or, "we use level 3 Scrum."
I think one of the problems with the terms "Agile" and "DevOps" is that they are pretty general and generic terms that they are trying to cast very specific meaning too...
“So you’re saying you’re not an agile team?” Sure we are appropriately agile for our needs, no need to be rigid about it...
This is a content-free extremely meager blog post that doesn’t really belong on the front page of HN but the discussion in the comments is quite illuminating.
The discussion shows how a lot of people have wildly different ideas about what devops is and that in and of itself is interesting.
For me devops was never a great idea. Ideally you want one team to decide on the architecture. You do need front line ops, ideally they'd take some ownership and write some scripts, but the buck stops with one team for inhouse systems that is usually developers.
No one admits they don't do it because everyone does it. I have worked at 6 different companies since I've heard that term and all of them were able to deploy to an environment by clicking a button on a CI/CD tool. Maybe I'm the outlier?
Anyone can deploy a file with one button. DevOps is what happens when the app starts randomly crashing.
If the site keeps humming along like normal, or the app is automatically rolled back, or (rarely) the devs get a page & quickly diagnose in production and apply a fix in 10 minutes, and deploys don't randomly fail due to unpinned versions etc, that's DevOps.
If people start running around like chickens with heads cut off and pointing fingers, that's not DevOps.
The mistake was turning DevOps into a job position which resulted in silos being created more when the whole purpose was to remove/prevent silos and to allow developers to operate at their own pace.
All these processes are a template, a starting point. Begin there and optimise and refine to suit your team and your environment. Dogmatically following a process that doesn’t work for you is silly.
Just being devil’s advocate, whatever the profession is, if someone keeps changing things, how would it reflect on his or her maturity in getting things done?
I feel like devops became a marketing juggernaut then somehow evolved into creating a third team, apart from dev and ops, that does handoffs with both of them.
I only do as much 'ops' work as required by the idiots in charge. Its much more productive for me to be working on business solutions than screw around with getting traces and logs into datadog and figuring out all the permission issues in AWS. Someone can specialize in that knowledge instead of everybody needing to know everything.
This seems like an example of something similar to the no true scotsman fallacy. (it's inverse) These ideas are supposed to be aspirational, akin to asking "is it a true democracy" or "is it really capitalism". I don't entirely object to the notion of trying to be more precise, but gatekeeping "devops" or "agile" also seems to sort of miss the point.
While I get the underlying argument you are trying to make, there are definitely still sysadmins in the world who see their job as administering systems, and refuse to look at anything resembling automation.
I see the point, but it’s funny that the author lays out the reason why this goalpost-moving is happening.
As pointed out, you sound like an idiot if you’re against Agile. So it’s a lot easier to get what you want if you just call it Agile.
I get that it’s not intellectually rigorous or whatever, but good grief, when has that ever mattered in business. Is there any class of concepts that survives unblemished after contact with reality?
Your pithy dismissal leaves out hundreds of pages of religious rites and project management dances which invariably turn into productivity-sucking meeting overload, unachievable goals, and constant product churn, effectively alienating everyone from the developer to the user, leaving only the managers satisfied. Like telling a college student that communism doesn't work, any objection is met with "well they're just not doing REAL Agile," skipping over the fact that nobody on Earth seems to be "doing REAL Agile."
Agile could just as well serve in the headline in place of DevOps.
This is not how it works in real life. Contracts are made to the lowest bidder, there are promises to keep or heads will roll.
By iteration 10, your customer is fuming that the half baked idea has more exceptions than handled scenarios and the finger pointing begins. Also, forget about architecture, performance, security and other pesky little problems. I have never seen it play out differently.
The biggest problem is that the person footing the bill does not care about your story points or retrospective- and I don't blame them for that
I'm not sure I agree with the author wrt Agile. I've never been on a team that did anything that resembled agile but I have been on teams that actively pushed back when anything that looked like it sort of resembled agile got floated. I'm not saying its 100% bad but I'm not sure it has a universally good reputation
Or, alternatively, it might not be a great idea in all cases.
Maybe your development team shouldn't develop terraform the four times a year they need it. It might even be faster to have two or three individuals with a singular vision for the system in a smaller team. (There may not be enough work for some teams to have someone with specific DevOps skills. However, that ends up in a partial-shared-service model where team members get called off for organizational concerns.)
Maybe you need a subject matter expert that will develop standards for teams and select a set of tools, setting those up so that teams can build off those for their needs. (This is why Ops are often shared services teams.)
Maybe it's not a great idea if the first team to encounter a problem picks the tool that works best for them, and the organization ends up with multiple tools because there was nobody with both the time and expertise dedicated to the analysis. (This is why Ops are often shared services teams.)
We need to break down barriers. We need to adopt valuable practices. However, the parts of DevOps that don't make it to the real world may just be because they don't solve the problems of the organization.
Rapid release means I can no longer rely on the software that engages in it. It's constantly shifting, with frequent changes in UI and workflow, and new and exciting bugs constantly appearing.
I miss the days when I could plan ahead for new versions, so I could time them to have the least possible disruption to my life.