Yes distributed systems are complex, but IMHO at least a large part of that complexity is caused by people who think they understand the problem, and go off and create some "great" new technology. Which then gets a lot of press, and other people building systems on it, only to discover that the edge case being ignored as inconsequential starts to create lots of failure modes. Sometimes these critical problems are "human factor" ones like with kubernetes configuration, or they are just a lack of proper engineering hygiene. Then you get another layer of crap on top. Repeat that a few times and what you actually have is a giant mess that frequently only works through the sheer effort of a large operations team.
To put this another way, if you read the Jepsen reports overwhelmingly what comes across is a lot of hubris. Its overwhelmingly old stodgy technology these new systems are meant to replace (posgres for example) that are satisfying the promises being proclaimed on high by $NEW_TECHNOLOGY.
I don't look at the Jepsen reports as being a representation of hubris. Most of what Jepsen finds may have been preventable problems, but they are also fixable problems, which is why I don't see hubris.
There's just a reality that we don't really build these systems on top of a distributed operating system. They're starting with an platform abstraction that is already terrible imperfect for the job. There is a ton of pressure to get something out quickly and make it accessible to a broad set of developers rather than to do it correctly (MongoDB is perhaps the most classic example of this). I was building big data systems in the early days, and the whole mentality was that you were building tools that were riddled with flaws, limitations, and outright bugs, but in the right context would make capabilities accessible to a broad set of developers that were otherwise completely off the table.
The truth is, if developers were already experts at distributed systems, they wouldn't need or want most of these tools; a LOT of the value is in accessibility and having something that is "good enough" rather than correct. The products reflect that more than they reflect hubris.
What I do see is a combination of marketing and customer ignorance that does believe in silver bullets and belief that these tools don't have to be understood to be effective. It's an unfortunate byproduct of focusing on that accessibility over correctness.
To put this another way, if you read the Jepsen reports overwhelmingly what comes across is a lot of hubris. Its overwhelmingly old stodgy technology these new systems are meant to replace (posgres for example) that are satisfying the promises being proclaimed on high by $NEW_TECHNOLOGY.