> Besides. Let’s not take the fun out of everything.
there's nuance in everything. But this line is what gets me about this article. Let's not prioritize the "fun" of devs for unnecessary toil of ops/SRE.
If you look at the current major offerings like SentinelOne, they start off with a generic best practice baseline, then slowly “learn” the normal traffic on the network to be able to better define the abnormal incidents to the IDS.
You make a good point. What occurred to me is that some more
standardised method for coding threat events is needed first. But
that seems tractable given existing CVE taxonomies etc.
... and leave ops/ devops / are team to pick up the pieces (or put the pieces together) because now your too busy to iterate because your shipping another new thing?
1. If you're agile, you're not too busy to iterate because you're shipping another new thing. You're able to come back and iterate if that's the most valuable thing you can work on.
2. When waterfall delivers the wrong thing because the requirements from two years ago weren't updated to reflect two years of new reality, someone has to pick up the pieces from that, too. It may not be devops - it might be sales instead. But it's still a problem.
The standard fix for this is to put the devs on call.
Recurring operational issues get fixed mighty quickly when J. Dev has to wake up a few times at 02:00 on Saturday.
Heck, design and architectural issues suddenly get a lot of scrutiny at the whiteboard phase and people decide they don't really need Kafka or Kubernetes or Mesos or GenAI anymore.
That thinking is one reason why I left engineering and I don’t see myself coming back to it anytime soon. I don’t like being on-call, I’m not qualified to do Ops-like decisions and I haven’t seen this as the practice in most other industries - Boeing doesn’t have staff flying planes for the airlines, for one.
Compliance is a set of stories/tickets/requirements like any other, with a priority to be assigned and eventually worked and reworked at some point in the process. There’s nothing wrong with not addressing it as the first thing - it just blocks release. With that, it will eventually get worked on, hopefully at the time where the pieces that need it are understood and the work to reach it is understood.
At a $LASTJOB all the devs had to be on call for some services and literally nothing improved. Still haphazardly throwing code into production, poor monitoring tools, and no time or desire to make stuff better. On-call was just seen as shitwork and devs would find other teams that didn't have on-call to transfer to.
That's kinda how it works at my current job, where a support rota is manned by members of the various development teams (front end dev, backend dev, app dev, platform ops, etc). You can indeed get woken up at 2am on a Saturday, and expected to fix whatever mission critical issue has popped up at that point.
It doesn't seem to have changed much as far as the architecture process goes though. With a few exceptions, the things that pop up for support to handle are usually either 'something broke on the content side, and the editors can't fix it', or a third party broke (glares at Google and Tag Manager, Musk and Twitter/X, etc)
This is why I am a fan of build & run teams instead of having a separate ops org which is responsible for keeping things running. The incentives aren’t aligned and almost always result in silos and excessive outages. The shared responsibility means no one is actually accountable and solvable issues persist for years. Almost every client I work with who has major ops issues have separate teams handling build and run aspects of their projects.
In 15 years of experience here and there, only one has had on-call devs. I’ve come to the conclusion that neither dev nor their management care if someone else’s phone rings at 02:00. I’ve also seen open hostility to 1) the very idea of dev going on-call, and 2) technical recommendations for solutions to recurring issues. So I think it’s not quite “the standard fix”, in fact I think it’s pretty unrealistic.
this reminds me of the 'leading change' performance metric some mil leaders were (are?) evaluated on. the incentive is to be seen making changes, but there isn't anything about following up to ensure the change accomplished anything.
People need empathy for the customer. One possible way to do that is mix up roles but like many other commenters have pointed out, that may not always be the best use of their time.
At a minimum, the devs should have regular updates, even anecdotal user stories about frustrating experiences. Perhaps if there are feedback loops customer context is getting removed from it either via unintended optimization or normal 'corporate telephone' .
> People need empathy for the customer. One possible way to do that is mix up roles but like many other commenters have pointed out, that may not always be the best use of their time.
Rather: if you want your developers to become deeply misanthropic and hateful of the users/customers, let your developers do user support.
If you use CI you can add such things there so any commits that didn't run them fail CI. I do the same with the `pre-commit` utility, it's very very handy for running checks repeatably.
Even more proof that Douglas Adams saw this all coming
"There is a theory which states that if ever anybody discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened." [1]
I can't believe I'm the first one to get this quote in :)
there's nuance in everything. But this line is what gets me about this article. Let's not prioritize the "fun" of devs for unnecessary toil of ops/SRE.