There are people attacking the author for a statement made about CoreOS, and for some hate towards Kubernetes.
The key point of the article is not really being addressed here: vested interests from large companies are able to introduce huge complexity into simple, well-designed projects. While the complexity may be good for some end that said vested interest has in mind, they are also in a position to absorb the cost of that increased complexity.
In the meantime, the simpler version of the software is long gone, with the added complexity placing a large burden on the solo developer or small team.
It's almost like there's no good representation in the open-source world for the solo developer or small team. Funding and adoption (in the extreme, some might say hijacking) of open-source projects from large corporations dictates the direction of all major software components today. Along with the roses come some very real thorns.
Agreed. I saw it as a general lament against over-engineering. I don't think the point got lost in the super specific example...
You could just as easily level similar rants against the likes of React and it's wider ecosystem, Tensorflow, Typescript (many will disagree), Docker... I'm sure others have their own bugbears.
Much of this is subjective, of course. But to me, it feels like software development is trending towards unecessarily complicated development processes and architectures.
And the only beneficiaries appear to be the large technology companies.
I suppose in exchange, you're getting a guarantee of maintenance. But is that really worth the additional complexity associated with the common use of these tools?
TypeScript is a funny one. I love the language, but at the same time, I totally agree with the premise that it is unnecessary complexity! And yet I swear by it. I can't explain why there's not more cognitive dissonance there.
JavaScript taught me to love async, then functional programming, and TypeScript taught to me to love static types. I'm now desperately wishing for a world of OCaml/Haskell, but where are you going to find teammates using those? And so I'm back at TypeScript.
I think all of these "higher order" languages that do transpiling (including Scala, Closure, Kotlin, Groovy, etc) fit this same scenario. Increased toolchain complexity for a decrease in development/test/maintenance complexity.
With any toolchain, though, it can be hard to know where to draw the line for "worth it for this project" until you're already an expert in using the toolchain, at which point, why wouldn't you use it?
You choose the tool chain to fit the anticipated team. If that team is just you, then who cares? Pick the easiest thing for you. But if you’re an expert in the tool chain and you anticipate new people hopping on, then the on-boarding process should definitely be one of your considerations.
The argument for simpler tool chains for simpler projects is that the time it takes to on-board should not outweigh the time saved by the toolchain’s amenities.
So the argument for TypeScript despite its complexities is that you believe its approach to type decorations on top of JS strikes the right balance for ease of on-boarding (IMO fairly straightforward) and supporting code maintainability (IMO a big improvement in 99% of the cases) despite the extra complexity of lengthening the toolchain (IMO it has some counterintuitive parts but you can mostly just forget about it once you find a working setup.)
I won’t use TypeScript if I’m whipping something up quickly, but if I anticipate open sourcing, adding unit tests and CI, etc., then TypeScript feels like a useful addition.
I used to make the same argument, about not using TS for prototyping, but the tooling is so good that at this point, I generally just change the file extension to ".ts" and write no manual types for the mistakes and type-hints + intellisense the compiler and language server provide.
Yeah exactly, and we can also pick the features we need in a language. E.g. C++ is super complex after C++/11, but we can still just pick a small subset to make it easier.
Then some poor soul gets to wrap the old patterns into the new patterns!
But seriously, if modern C++ has one thing going for it, it has a great backwards compatibility and forward refactoring story.
I'm not a huge fan of the "evolved" syntax, and I rarely use C++ anymore, but I like modern C++ substantially more than C++03, which was where I first cut my teeth as a programmer.
> Then some poor soul gets to wrap the old patterns into the new patterns!
This isn't always an option. Google's C++ guidelines generally forbid using exceptions, but concede that you might have to if you're dealing with existing code, particularly on Windows.
What is unnecessarily complex about TypeScript? It's JavaScript, with static typing plus type inference, and pretty nice generics. The ecosystem of modern JS surrounding it is horribly complex but TypeScript itself seems like a fairly straightforward programming language.
It hasn't been a big detriment for me as someone learning Typescript on their own, but it is another moving target for looking up "how do I do x..." and finding most of the forum posts are a little outdated and the latest version of Typescript has a different/better way of doing things than just a year or two ago. I find myself scrolling through github issues comparing my tsconfig to figure out why my stack behaves differently than someone else.
That was my experience with TS maybe two years ago - at this point project scaffolding tools are good enough to generate sane output that I spend a little bit of time upfront but then keep plowing away. Maybe I got better at it as well - but I haven't kept up with TS news in a long time and I don't feel like I'm missing out on stuff or encountering things I don't understand.
I've written >50k LoC of TS in last few months for sure (doing a huge frontend migration for a client) and I can't remember the last time I googled anything TS related. Actually I remember - a month ago I wanted to know how to define a type with a field excluded, took 30 seconds of google.
Meanwhile the project started out as mixed TS and ES6 because most of the team was unfamiliar with it and there are a few dynamic typing evangelists - we ended up going back and just using TS all over the place, the complexity introduced is minimal and the productivity boost from good tooling on product of this scale is insane.
Typically for me the time cost is in going down rabbitholes to attempt to improve implicit static types for getting closer to "whole program" functional type inference (TypeScript repeatedly seduces me into this), and the decision inflection point is generally not for application code but for the space between application and script code, things that you might also write Perl or python scripts to accomplish...the types are especially useful in this context because they tell you a lot more about the script than your typical script, but they also introduce a bunch of overhead for a few lines of code.
Yeah I probably shouldn't complain, I've written less than a couple thousand LoC so I'm still googling a lot, but TS has definitely paid for itself in code clarity already.
At least I can pick it apart and deduce it. I can even use the compiler to give me information about it via editor integration (all same editors have it now). So I’ve I’m unsure it might take a few minutes of investigation to completely understand the signature.
I’ve seen some JavaScript written so tersely it was nearly impossible to figure out without spending Possibly hours on unwrapping the code. That’s the value provided here.
When it comes to spending hours unwrapping code, gotta love js that has undocumented heavy use of string based property access for things like imports. Like the worst of both functional and OOP combined, and generally no IDE support to be had.
Implementation approximate, but otherwise true story, both LHS and RHS:
It's funny that you consider that complex; I consider that a pretty normal, expressive type signature that gives the compiler critical information about how I expect to use that function, which in turn allows it to completely squash several classes of bugs I might write.
But I love strong type systems; people who prefer weak type systems would likely consider things like this to get in their way.
Could you elaborate on this point for someone who isn't familiar with typescript (or javascript, for that matter). I'm no stranger to strongly typed languages but that function signature seems pretty complex to me. By elaborate I mean explain what's going on in that signature, and what the critical information it supplies is?
It's from the TypeScript 4.0 beta blog post[0], which describes it as:
> partialCall takes a function along with the initial few arguments that that function expects. It then returns a new function that takes any other arguments the function needs, and calls them together.
The type signature looks like:
type Arr = readonly unknown[];
function partialCall<T extends Arr, U extends Arr, R>(f: (...args: [...T, ...U]) => R, ...headArgs: T) {
First of all, know that in TypeScript, colon separates a variable or parameter name from its type. So in C or Java you'd say "Date lastModified", in TypeScript it's "lastModified: Date".
Now, looking at it piece by piece:
function partialCall
declares a function named partialCall.
<T extends Arr, U extends Arr, R>
says that this is a generic function with type parameters T, U, and R (the first two of which must extend Arr, that is, a readonly array of objects whose types are unknown).
This function's first parameter is:
f: (...args: [...T, ...U]) => R
The name of the parameter is `f`, and the type of this parameter is `(...args: [...T, ...U]) => R`. This means that `f` must be a function whose return type is R, and its parameters must match `...args: [...T, ...U]`.
The `...` in `...args` makes it a rest parameter[1] (variadic parameter in other languages). The type of `...args` is `[...T, ...U]`, which you can think of as the concatenation of the T and U arrays. (It's a bit strange to see `...` in a type specifier, I can't recall having needed this before.)
...headArgs: T
says that the `partialCall` function itself is also variadic, and its arguments are gathered up into headArgs. The type of headArgs is T.
I don't know if I'd call this normal or clear compared to the signatures I encounter daily, but it's pretty elegant for a higher-order function that takes a function with any signature, and some arguments that match the given function's parameters, and returns a function with the matched parameters removed. And the implementation of partialCall is just this one line!
I would say, there is some complexity in reading (and writing) the thing. But there is probably not as much complexity in using it. And you are explicitly codifying the complexity in one place, that would presumably still exist in a dynamically typed language, just implicitly, and likely spread out across the code. This looks pretty nice to me at a glance, just like anything once you write a few of them and come across a few in the wild and take the time to pick them apart they no longer seem so scary.
Offtopic but what's nice about Typescript generics? Typescript doesn't even let you specify variance. It's one of the unsound parts of the language's type system in fact.
You cannot explicitly specify covariant vs contravariant, but TypeScript absolutely does allow you to express these relationships. Unless I misunderstand you.
That said, the type system has come a very long way even in just the last year. The biggest improvements imho being around recursive types (which was one of the biggest holes for a long time imo), literal types / literal type inference, and tuple types.
It's not complete by any means, but it's improving quite rapidly.
It appears that function parameter bivariance is still a thing? [1] Although there seems to now be a flag to make this one use of variance correct.
I would assume even Array<T> is still bivariant as well...
Both of those are horribly unsound, just for convenience. Sure convenience and compatibility are Typescript's ultimate goals, but to actually praise it for its generics? That's very strange.
> TypeScript absolutely does allow you to express these relationships
How would you express a class with covariant or contravariant type params in Typescript?
describe the steps to release the simplest ever code in javascript to production: write a js file, host it, done.
The same thing in TS adds at least one step (not to mention the rest of the tooling you will want)
So while a prefer it over JS, there's no arguing that it is more complex as now you require a build step for a language that only exist because people wanted a language without a build step.
A lot of fortune 500 companies with some developers who missed the trendy stuff still do it that way. I made a medium size website (30 pages) in React with pure javascript and dependencies being script tags in index.html to vendored files.
So not even JSX. I did it that way because it was the easiest way to develop and deploy in that environment
And also don't use modern frameworks like React or Vue, or don't mind sticking all your templates in strings, or in your index.html, and shipping 100kb of template compiler to to user, or write render functions directly ala mithril.
my team (in a large enterprise) uses js for scripts using a shebang interpreter declaration, eg
```
#!/use/bin/env node
console.log("hello cli")
```
While it does depend on node, and there are arguably better crossplatform languages for this purpose, it is a zero-tool chain use case that is very convenient for us.
"Nobody" here means few, or more loosely, much fewer teams than before, not "literally 0 people/teams".
And the group mentioned is (I deduce) not generally individual devs, enterprise devs building some internal thing, and so on but teams in companies doing public-facing SaaS, teams in startups, companies like Amazon/Google/Facebook/Apple all the way to AirBnB etc, and so on.
Yes, exactly. The larger the team, the more likely someone is a front end expert and wants to use latest cool framework, which will by its nature require a build step. Even for something simpler, you'll probably want it for cache busting, minimization, etc.
At the rate the Typescript is releasing, that'd be a support nightmare. Perhaps a better solution is for TC39 to propose optional types. It could be modeled on Typescript for sure, but it would still be backward compatible.
Javascript of today borrows liberally from coffeescript of yesterday, so it would make sense for javascript of tomorrow to borrow liberally from typescript of today.
But then if that's an option, I think Typescript will be the last language I migrate to, because Typescript development culture, tending as it does towards overcomplicated solutions to simple problems, is unpalatable to me.
I'm drawn to the idea of using Rust over WASM as a frontend language, and I think I'd rather choose that approach to develop any browser UI where type safety is critical, provided there is no discernable difference in performance (when compared to TS over WASM).
Yes, it's probably a better idea to improve WASM than add a proprietary format (TS is by Microsoft) to the open browsers. Google tried to do the same thing with Dart and it was decried about a decade ago, so now they use it for Flutter.
I think it will be great once support is broad enough.
It might, ironically, increase the current fashion for framework churn, but at least there will be no single language for developers to derride.
In fact, I wonder how ECMAScript will fare in a post WASM world... I suspect it would still thrive tbh.
Or perhaps people will take to other flexible, expressive languages for UI development. Like Python's niche in computer graphics, or Lua in games and AI research.
I can still see myself using JS in that future.
But not for everything.
Can't put my finger it for you, but it defn doesn't feel as good. Perhaps because it needs to interop with javascript, and js objects are all over the place with types.
All the JVM languages you list aren't transpiled. They target JVM bytecode just like Java. They're first class, even if Java obviously gets the overwhelming amount of VM level support. Engineers working on the JVM are definitely aware of and want to support non-Java langs.
Scala can target JavaScript. Kotlin is usually used with the JVM, but can target native machine code (and JavaScript too I think?). Transpile was the wrong word for me to use.
> I'm now desperately wishing for a world of OCaml/Haskell, but where are you going to find teammates using those?
Having walked that way I can recommend you to look at F#, and if you aren't a solo developer (or you want to get real regarding jobs and colleagues) make sure you do not have allergy to C# interop.
You hit the nail on the head with tool chain complexity. Transpiling adds yet another layer in a stack that's already deep. Web isn't my field but it looks like from the outside that obsevability[1] is also lacking. Some friends that are in web programming mention a lot of younger engineers don't realize how bad it is.
It makes me think of military contracting as a parallel example. (at least in the USA) The rules required to develop, build, and deliver military equipment to the US is exceedingly complex. And it isn't necessarily a benefit for the Pentagon, as much as it is for the established defense contractors. The barriers to entry in that industry are huge, purely based on the contracting requirements.
So complexity (in tools and process) do not necessarily serve the individual developer the way they serve larger organizations.
> And it isn't necessarily a benefit for the Pentagon, as much as it is for the established defense contractors.
As with many government related procurement systems, there is so much paranoia about abuse, and desire not to repeat various disasters from the past, that the system has by perceived necessity become complicated.
Of course it makes it frightfully expensive to the degree that few companies can actually throw the resources at it to be able to navigate it. The lack of competition can result in projects costing a large amount of money, and the buyers in the agencies having no real options to look elsewhere.
At an e-government company I worked for, we a service for the state that helped companies navigate the state's procurement system. We created a centralised place for all data to be gathered by the applicant, generating all the forms they needed, and provided a way for the applicant to track the state of their application. It introduced quite a change for the state, dramatically widening the potential pool of companies that could bid for contracts. Also pissed off quite a few of the already entrenched ones :)
It's like a macro version of the story told in Capt. David Marquet's book Turn the Ship Around! Huge piles of bureaucracy, complexity, and waste build up in an organization risk avoidance is allowed to become the primary goal.
Hawaii Information Consortium (I guess it's now, NIC Hawaii, https://nichawaii.egov.com/), which is a subsidiary of NIC.
HIC was a great company to work for, as was NIC from the limited interactions I had with the larger corporate powers that be. Most websites/services were provided for free to the state / state agencies, instead relying on small fees per transaction on some of the stuff we did, to fund the work that didn't have transactions (I forget the amount, but we're talking something like 50c per transaction).
Being free was a key incentive to getting various agencies to come on-line.
My first boss (rip) had a internship with a defense contractor his junior year in college. They gave him one project. Design a cover for the air intake for an APC or some such.
Easy!!! No not actually easy because of all the constraints.
It had to be stowable. So a hard cover was out. It couldn't produce toxic fumes if it caught fire. So most plastics were out. Cloth was a problem because it couldn't get sucked into the intake if someone tried starting the engine without removing the cover.
He designed heavy canvas cover with metal stiffeners. And snaps. And then had to switch to a draw string because the snaps failed at -40F. And tended to get clogged with snow.
Whole thing took him three months.
Then there was the friend at college who worked on a VCR to record gun camera footage. Also a bunch of requirements. Higher video quality. And can't lose lock when the pilot does a hard pullout after firing his munitions. And the total production run? Well exactly how many fighters does the airforce have? Couple of thousand?
I think military tech has a problem that it's trying to keep up with commercial driven tech which operates on a scale that's a 1000 timers larger. Military produces a few thousand artifacts to commercials few million artifacts.
Those constraints seem a lot more reasonable than what I've seen in other aspects of the business world. At least they are grounded in the realities of the actual purpose. Well, mostly anyway. I'll leave justifications for the $1000 left-handed hammers as an exercise for later.
I've seen (and removed) plenty of requirements that were put in the specification just because. Because someone needed X amount of Y technology in their projects for the year. Because it seemed cool. Because other people were doing it. Because they wanted it on their resume. Because. But not because it was appropriate to the project or its purpose.
I'd say the majority of the work I do now is pruning these nonsense "just because" clauses out.
Its entertaining at least. I regularly shake my head in disbelief as I go through a specification and wonder, "who hired these people?" or "Why is the person who hired them still working?"
Fortunately for me I also get to hand these questions back, much like the good uncle who only needs to uncle and not actually parent: "Here, have them back!" I say at the end of the visit.
The the hammer was waaaaay more than $1000. But that's because the military said "We'll pay you a total sum of X, but to make it easier to fund the project we'll let you break it into n parts and pay X/n per part".
The contractor decided that to make their cashflow smoother, they'd include "manual impulse force generator" as one of the deliverable parts.
The hammer was "only" $435 and that price is really a reporting artifact.
The hammer was part of a larger contract that included spare parts and R&D effort linked to those parts. When the spending was reported, the same absolute amount of R&D ($420) was allocated to every item, inflating the apparent price of a $15 hammer to $435. By the same token, the engineering work on more complicated systems (e.g., an engine) was an absolute steal at $420 and since the total amount of R&D spending was fixed, nobody really got ripped off.
Because IBM was there 20 years ago and not only left behind a pure-IBM stack, but got their RUP requirements written into the policy manual as a barrier to entry for any non-IBM vendor to take over.
I'll leave justifications for the $1000 left-handed hammers as an exercise for later.
Those tend to be an accounting artifact, I heard. If you order a thousand different items for a million dollars total then each one of them will show up as costing exactly $1000 no matter what they are.
> I think military tech has a problem that it's trying to keep out with commercial drive tech which operates on a scale that a 1000 timers larger. Military produces a few thousand artifacts to commercials few million artifacts.
Not to excuse military contracting pork and cost padding, but this is a good point that a lot of people seem to miss. There's also the fact a military contract will be for a production run and follow on support.
So they buy a thousand full units and parts/tools to fix and maintain them up front. You can't just go to Autozone and pick up a new tank tread or parts for a jet turbine.
Another huge factor is the service lifetime of military equipment.
There are aircraft that were designed and created in the 1970s that are still in use today. Sure, many of the components and internal systems have been modified and upgraded, but much of the original design is still there and operating.
>> a military contract will be for a production run and follow on support.
When "follow-on support" has to last for 50 years or more, it makes a big difference.
React I have been thinking about, having worked with it a lot and also recently done a React / TS / GraphQL (GraphQL may have been the poorest choice I made; time will tell) project.
I think React itself is awesome and I've always enjoyed it. While I'm not an expert, it's conceptual foundations and core abstractions felt right, and I do think it makes lots of frontend tasks simpler, especially for non-small projects.
At the same time the frontend ecosystem is a mess, as we all know, and as my CSS skills advance I see more just how much could be done with HTML / CSS / limited use of JS and your classic REST app, that's not a single-page app, and how much complexity might be able to be avoided and thus time saved.
At the same time, I don't think it's just "trendiness" that has caused the growth of React and SPA's. I think they give us various wins that we probably take for granted because we're now used to them.
So, the short answer to my long ramble is, I don't know.
But I do think React itself, specifically, hit a sweet spot in terms of being a powerful yet understandable tool, and still remaining a tool, as opposed to a framework.
Me: But we've been doing CRUD for 30 years without an API this is a small project
Dev A: But I don't know how, its not best practice, my team lead agrees, here is a medium article, get with the times, etc.
Me: Ok so you don't know how to do your job.
Dev B: Here, put the React SPA in a Docker container and run it on some cloud its easy...
Me: But all I need is still CRUD, I dont need to scale it, I dont care if its isolated, it should have been 100 lines of code
Dev B: its not best practice, my team lead agrees, here is a medium article, get with the times, Docker run everywhere, etc.
Me: But what I need to connect it to internal resources and use DNS?
Dev B: load balancer! If only you have an AKS cluster!
Me: We work in a 25 person company that needs some very basic CRUD. It will never scale past one deployment, ever, if it does, dont worry we will pay someone to do it right because we will be swimming in money.
The problem is that people are starting to not know how to do simple things. So you get grumpy admins losing their marbles over the complexity of this BS. Most of the time things could be fixed faster, easier, deployed and managed easier using simple technologies that have been around for 20+ years.
But lets wack it with the Python, Docker, React, NoSQL, GraphDB, K8S hammer because...? Not only that, the juniors don't know multiple technologies anymore they just know X. So sysadmins complain that devops is pushing more roles on them, while I have developers that have written full Node.js sites and dont know that IP or a Port, or TCP, UDP, is...
Dude! You need to run more servers. 250 should do it. I realize its just a simple page with a pair of boxes for name registration, but that's irrelevant! All the cool kids agree. You need infrastructure. It needs to be done as complex as possible! Complexity-as-a-Service won't just happen, you have to want to make it happen. You don't want to be uncool, do you?
Looks at a specification on the desk Yep. We'll keep the first and last page. Everything else is gibberish. Next!
Edit: Looks like a few people disagree. Perhaps they might like to explain why I saved a bunch of companies around 100k in AWS fees just by pruning their server requirements? No? And yes, one of the projects was a simple set of web forms that required 7 servers to run. We pruned it down to 3 and that was just for redundancy and load. That's just one example.
i didn't downvote you, but since you're asking – the first part of your comment is needlessly snarky and doesn't really add much beyond "yeah, i'm frustrated by people adding unnecessary complexity just because it's trendy". on the other hand, the part added in the edit sounds like it could be an interesting story! expanding on that would make for a more interesting comment:
> I saved a bunch of companies around 100k in AWS fees just by pruning their server requirements. One of the projects was a simple set of web forms that required 7 servers to run. We pruned it down to 3 and that was just for redundancy and load.
EDIT
and if that thing about cutting down the specification really happened, just tell the story – no need to wrap it in a performance piece:
> I've actually had people come to me with specifications so over-engineered that the whole 20-page doc could be simplified down to a single page without loss of functionality!
EDIT 2
and i'm not saying "never use hyperbole"! just don't make it the whole point of your comment :)
I made that comment once about "Soylent". (Remember Soylent? The nutritional drink?) That company made a big deal about their "tech stack". Not for manufacturing or quality control, but for ordinary web processing. Their order volume was so low that a CGI program on a low-end shared hosting system could do the job. But they were going to "scale", right?
Amusingly, they eventually did "scale". They started selling through WalMart in bulk, and accepted their fate as yet another nutritional drink, on the shelves alongside Ensure Plus and Muscle Milk. Their "tech stack" was irrelevant to that.
That's an amusing story. I suppose the Soylent board met a good salesperson, and the pitch worked. So when the team landed the contract, they had to justify their costs some way... I suspect almost every software project in existence is in some way a victim (or beneficiary...) of fanatical marketing.
I remember working on an simple CMS site, amongst other things, for a pretty large company. When we begun the project, we were tasked with re-purposing an overworked Plesk instance to host the site. We eventually managed to do it, but then found the small amount of disk space that was left to us was getting chewed up by logs.
So I reported this to my project manager, suggesting that we procure more HD space. I think I said something about us 'running out of memory on our hard drive'. The PM promised to feed this back to the client... A week later, our PM said that he and the client had resolved the issue. The client will pay for a new server with a stonking 96GB of RAM!!!
That ought fix our 'memory' issue, right!!?
I mean, it also came with a 1TB HD, a second box for redundancy, dev time for migration, and additional dev time for a switch to Journald, or Logrotate, so I wasn't rushing to point out the misunderstanding... Working with the second box over RedHat Pacemaker was all new to us though, and a complete PITA.
But it was also another 'feature' to sell back to the client. They loved the sound of that. A site that couldn't go down... We made it work, but there was absolutely no real technical need for a Pacemaker cluster and 96GB*2 of RAM. It was just a simple CMS backed site.
Occasionally, that's where such complexity comes from. Not the developers themselves, but some loose cannon of a saleperson. That said, these 'sales people' may even be developers themselves. I often think that's a large part of how and why questionably complicated software exists...
A company I used to work for needed a website. They already had a backend and a REST API (served on the same domain as the website should be) and the old website was served directly by the backend that also served the API.
I am not aware of why it was chosen to retire that and separate the website into its own service - maybe there was a good reason, so I won't comment on that.
However the approach they (or rather some frontend developer) chose was a React front-end (fair enough) with a Node.js backend that translated between the REST API and GraphQL (WTF).
The existing API was served on the same domain so the new React website could've directly interacted with it without any problems. But no, this guy wanted to put "GraphQL" on his resume and so introduced an unnecessary extra component and potential point of failure that the company now has to support.
I made this same tech-stack decision while working for a company in the manufacturing industry. React, GraphQL, Node.js. There was a reason behind it.
The head of the company was pushing to modernize our process flow via software meant to drive manufacturing. We had contractors on site from three different companies who were each using us as a test-bed for their software/manufacturing integration. Every week or two they'd plot out some new data they'd need to move from sales to engineering, manufacturing, QC, etc.
In our case, having GraphQL as a translation layer for the sales website saved everyone involved time. However, I can also see many scenarios where that wouldn't have been the case.
It definitely comes down to using the right tool for the job. Knowing how to identify which tools fit and which don't is one of the skills it's very important to help new devs develop.
Yeah, GraphQL seems like a cool piece of tech that's overkill for 90% of the places it's used.
I'm a bit confused as to how it's become so popular for normal development when it seems to have a lot more boilerplate and setup than a simple REST API.
I feel like GraphQL is the new NoSQL. Not just that it's mistakenly adopted, but also that it is quite valuable for the right use cases, but the public doesn't seem to understand what those are.
Do you really, _really_ need to support queries? 99/100, I'd guess no, and thus constraining people to a more defined interface and access pattern is simpler.
Oh... you need to fill your RDMS with a bunch of JSON responses from your 3rd party API. So you can then decode them in memory because I actually need to select * where identifier = 'banana' to do analytics.
Meanwhile the API returns highly structured data perfect for a RDMS, but we dont know how to query SQL without an API in the frontend. So welcome to my hell.
It's a combination of resume-driven development and premature optimization.
The former doesn't need explaining but the latter can be explained as developers being concerned about requiring the benefits of GraphQL in some uncertain future and decide to include it from the start, even though in most cases they end up never actually reaping the benefits while still being plagued by the extra overhead of using that technology.
> The existing API was served on the same domain so the new React website could've directly interacted with it without any problems. But no, this guy wanted to put "GraphQL" on his resume and so introduced an unnecessary extra component and potential point of failure that the company now has to support.
There's a simple solution to this kind of problem: let the resume driven developer use his skills - fire him.
So how would you operate each of these bespoke services? One service uses TCP, one uses UDP, one uses SCTP. What happens if your buffer sizes are incorrect? What about if your keep alives are too aggressive? What happens if a problem in your TCP connection pool makes it seem like there's a networking issue and so you play around with your network settings only to have all your other protocols dive in performance?
I sympathize with the OP for single developers or small teams (by small, I mean a team of 5 not in a larger corporation), but whenever you have more than a single team, you want to keep system management overhead low. Unifying around a single paradigm like gRPC and Docker containers means that not every team will have their own bespoke chroot configuration and you won't have to retune an HTTP client every time you interface with a new service.
I think there's in general too little representation from small-scale or indie developers. I spend a lot of time working with the Gemini protocol outside of work and I appreciate the simpler approach that the protocol takes. I would love to see these sorts of stakeholders have a say in greater net architecture as well, but let's not pretend like this is all complexity for complexity's sake; this stuff is needed at scale.
>So how would you operate each of these bespoke services? One service uses TCP, one uses UDP, one uses SCTP. What happens if your buffer sizes are incorrect?
Such bullshit. Since when we use UDP for a CRUD service ? Since when SCTP is even considered outside of the telecom world?
HTTP and Rest-like API have been able to handle properly simple CRUD API since 20 years without problems, way before gRPC was even a thing.
gRPC has its usage for large, complex API that required a proper RPC framework. But in 99% of the case, yes it's an overkill.
> Such bullshit. Since when we use UDP for a CRUD service ? Since when SCTP is even considered outside of the telecom world?
You're making a strawmam out of this. I'm responding to the following:
"while I have developers that have written full Node.js sites and dont know that IP or a Port, or TCP, UDP, is..."
But thank you for the insult.
> HTTP and Rest-like API have been able to handle properly simple CRUD API since 20 years without problems, way before gRPC was even a thing.
Hm, what are you talking about? Are you talking about the client? Are you talking about a load balancer? I don't think anyone is proposing that a simple static blog use gRPC to serve content to its users.
> But in 99% of the case, yes it's an overkill.
Compared to what? I'd argue that CRUD over HTTP is wrong. In fact, folks have been writing CRUD over TCP for ages. Why do we need to massage CRUD syntax over HTTP when we can just stream requests and responses over TCP? In fact, TCP predates HTTP by 20 years, so if you're using the historical argument, raw TCP is even older. How much bullshit is there around AJAX and Websockets and HTTP multipart and HTTP keep-alive when we're just trying to recreate TCP semantics?
So why are you drawing the line at HTTP? Seems arbitrary to me.
That's a very good argument against the modern framework-conatiners-k8s stack. We used to be able to keep an organization wide track of things such as tcp services, buffer sizes, timeouts etc. but now it's all a black box deployed by a guy who cut and paste some yaml from a Medium article, at best.
It's probably acceptable to not learn the details of what you're doing, as hardware is cheap and all that, but it also constitutes a glass ceiling for how much the organization learns. That's a bigger problem in the long run.
Hopefully any organization that decides to use k8s goes into it understanding what they're getting into, but yes if you just get onto the hype train without thinking, you'll probably have a bad time when scaling.
Dev B: Here, put the React SPA in a Docker container and run it on some cloud its easy...
I have 100 different devs telling me "do this thing, it's easy" and it's easy for them, but now I have 100 different things to think about and make work together and devs never think about how their tiny thing fits into the big picture.
99% of web apps could be pure HTML + CSS on the front end, and Flask or something on the back end, talking pure SQL to Postgres. Really. That's all you need. And probably 99.99% of those apps that run internally to one company.
Docker facilitates the "throw a Dockerfile onto Ops and that's their problem now" mentality. That's perhaps not intentional, but choosing to over-optimize the developer experience makes life very hard for those who have to support and run the application over time.
Docker facilitates the "throw a Dockerfile onto Ops and that's their problem now" mentality. That's perhaps not intentional, but choosing to over-optimize the developer experience makes life very hard for those who have to support and run the application over time.
It also enables pushback along the lines of "your container doesn't run, and it's 100% your fault, as it is supposed to contain all its dependencies". So developers have unwittingly played into the hands of ops there ;-)
Agreed, mostly. To be fair I think that, based on my limited experience with React and Docker and Python, those 3 technologies have valid use-cases in smaller orgs. The other technologies I'm not familiar enough with to comment on.
Also if I was gonna complain about someone I wouldn't complain about devs, I'd complain about team leads, engineering managers - the folks who should be guiding devs - and, the people at the business level who choose the team leads / engineering managers.
The devs are at the bottom of the food chain; it's not their fault.
GraphQL is unfortunately still part of a low grade hype cycle where more and more companies are using it because "it's the next REST". But if you don't have a real need for the advantages it offers (for example a mobile app whose client side requests you can't easily change in sync with your API, or a data model that really is graph-like), it can be more cost than benefit.
React on the other hand: I've lost count of the number of small prototypes I've started in vanilla JS, thinking "this time I can just roll it myself", then ended up pulling in React. Even the simplest things are easier and simpler with it, and hugely complex applications are also easier and simpler with it. It has a fantastic API surface.
> towards unecessarily complicated development processes and architectures
please note that what's "unnecessary" for you is not necessarily unnecessary for others.
this is an important point.
i guess what these projects need is a way to communicate explicitly the costs of maintenance and developer mind-share when adding more complex features to their product.
Because especially as small dev team it is difficult to figure out these costs during a short evaluation of the software, is not easy to pick the best tool for the job unless some members have got experience in some previous job with these tools.
I can’t even begin to clearly express how much I agree with you. I work for a very large company, not a software company but one that produces a lot of internal applications. The infrastructure that we leverage to do even trivial things is mind blowing. Want to send an email to customers? Roll a series of APIs and fuck it let’s do some machine learning bc yolo, integrate all the things and don’t forget to run everything through APIC. It’s insane.
That said, I also see the opposite when this gets some traction with "normal users". The Abseil issue was fixed, and the TensorFlow 2.0 release did make TensorFlow more usable by smaller teams in many situations.
It seems to me the problem is all of our software is built by and for megacorps. In the context of what they need it for its the best tool for the job but it was never made for smaller orgs where 90% of the bells and whistles are not needed and end up as needless complexity.
I'm not sure how react fits in here. I have found it to be very simple to understand enough to be productive and upgrading to new versions has required next to no changes to our large codebase. Perhaps you mean SPA react apps which I agree are a bit more work than needed for a solo dev.
Docker seems to be optimal for medium size teams. Configuring a server the old way is easier than docker for 1 server but less so for when you have 10 or CI/review apps.
React fits in because it's in the process of jumping the shark because FB needs it to. React is already hugely bloated for no discernible reason compared to Preact, but they're just going to keep bloating it with Suspense. Cf. https://crank.js.org/blog/introducing-crank
It's a shot across the bow at the Kool-Aid which everyone here has imbibed to some extent. Of course they're going to shoot the messenger. When I clicked this thread I said out loud to my team, "oh, this should be good," and I hadn't even fathomed precisely how predictable the discussion would be. It's so far removed from any consideration that it lands something like explaining to an American that they're a subject of the Queen. It makes no sense to objectively review the thinking when you're inside, and buy, the thinking.
Bear in mind the livelihood of nearly everyone participating in this conversation is impacted by the situation being lamented. It's telling that CoreOS employees felt compelled to set the record straight and casually "float ideas".
It's further curious to note the difference in HN's reaction to an extremely salty and offensive post from, say, Linus Torvalds, versus this one. People are really bad at objectively setting aside their ideology and tone is a kind excuse to get out of having to genuinely think about whether the angry individual might have a point, and when it hits close to home, well. All bets are off.
You're 100% correct on your overarching point but the less kind interpretation is "the incentives of modern software engineering value complexity to artificially strengthen a software engineering job economy". The good news is that the more Big Tech screws the pooch with user data management and software engineering, the stronger an argument I have among very receptive ears in government circles to start thinking of Big Web/FAANG as hostile to responsible computing innovation, because they never really bought what Mikey sold them, and the troupe that followed Mikey to Washington didn't hang around to explain the downsides or long-term management needs of the current landscape.
I think it's pretty drastic to say "everyone here has imbibed" that Kool-Aid; I work in embedded, and am not super familiar with the frameworks he's complaining about, but I found the blog post entertaining nonetheless.
Sometimes I feel like the people who are deep down in this "full stack developer" land should come up for air and look around at the bigger picture. There's a much bigger world of software development out there and actually I think this post was kind of an attempt at that.
The web-facing industry seems to have dug itself down into a very deep rabbit hole of frameworks-on-frameworks-on-buzzwords, in a way it wasn't 10 years ago when I last played in that space.
In order to deploy a web app in 2020, one must write his frontend using react and typescript that would transpile the code into a nicely bundled and minified javascript and css files along with thousands of your 3rd party dependency libs. Yes, you heard that right! I just counted the number of libs on the node_modules in one of my small side project and it's over 1000s. You might also wonder why the typescript code would produce css when transpiled. That's because in 2020 we no longer write stylesheet directly into a css file. That's so 90s! We write the css as javascript objects now, then let the transpiler generate css from that. Isn't that neat?!
For the backend, you can choose one of the many cool frameworks and languages. Heck, why not use 5? In 2020 the backend should be written as a collection of microservices that talk to each other over grpc. Of course you still need http or how else would your react frontend talk to your microservices? So your microservices now talks multiple protocols, which is very cool!
Now pack each of those service into a collection of docker images, then write 4000 lines of yaml files to orchestrate them on kubernetes, and you're ready to go! Now you have a glorious, scalable app with 99.999% sla that can withstand your show hn frontpage traffic thanks to kubernetes, until you eventually screw up that yaml config and the whole house of cards collapse, but that's on you because you're bad at kubernetes orchestration!
I think we're indeed in a deep trench but damn it's quite an impressive trench we got here!
I'm genuinely curious - do you really have to do all of that? Why not just ignore the frameworks and technologies that overcomplicate things and use what works best for you?
I am wondering why someone just hasn't forked etcd if it is truly so awful now.
Yes. If you don’t, and scp a Perl script to run under supervisord that meets the business requirement in three days instead of three months of microservices strategy, you are accused of being so hopelessly junior that you are unemployable.
I have witnessed that firsthand. Despite the anecdote being backend, I’d bet my 401(k) it’s the same up front.
Being a complete newbie on the scene a few years ago, the choice of frameworks and tools was so incredibly overwhelming (coming from C/C++ land). There are strong opinion pieces both for and against basically everything too.
When you are still trying to figure out exactly what HTML and JS should be responsible for, having React, Vue, PHP, Elm and many more as options for "just making a simple web page" can slow down the learning process by an order of magnitude.
I'm not sure what the solution is here- clearly each of these tools has a niche which requires them, but making a basic webapp without a web-engineering degree is quickly becoming an impossibility.
Forking is the answer, if you don't like the where a particular group is going with an open source project. Fork an old version, and perhaps you will find some others who prefer it that way.
Nope! The original group will sue you for trademark infringement, despite that they're the ones peddling a broken knockoff version. (See, eg Python, Firefox, GIMP, etc; I don't pay much attention to webapp-related stuff, so I don't have more proximate examples.)
Unfortunately yes. I'm a freelancer so I have to keep up with the latest tech, at least just enough so I can debug issues when working on my clients' projects. You probably don't need to do this on your niche and can just keep using whatever stuff that works for you.
And here I am, making a career writing LOB web apps in Rails, and getting to MVP in weeks instead of months (or even years). Guess I'm really missing the boat!
My pet theory is developers crave challenges. If their project is not challenging enough to stimulate their mind, they will introduce more and more complexities until the project is hard enough to keep them intellectually stimulated. Thus, software development will never converge into a single tech stack as smart developers that got bored as hell working on crud apps will invent new stuff to make their work more interesting and meaningful.
In 5 years react and kubernetes will be considered legacy tech and we will march on to other new shiny tech.
Rest assured that I am firmly in your camp, and I've been tirelessly making the argument to policymakers that Big Web is not the resident innovative force in computing any more, despite the Obama administration's rapid adoption of their ideals and philosophy to salvage Healthcare.gov. In both the legislative and executive branches I have found many, many sympathetic ears who have been burned by ballooning budgets to handle public cloud and all the concerns that came with it, which folks who advocate for cloud-first deployment tend to overlook when addressing people who genuinely don't know (concerns endemic to Big Web operation of cloud actually need explaining when addressing a sector that's never touched it).
There is a watershed coming where Big Web/FAANG will lose this mantle, and DevOps, half-baked Agile because nobody ever bakes it all the way, framework engineering, and a complete fundamental ball-dropping around security and exploitation of user data are rapidly ensuring it.
At some point, Big Web/FAANG and everybody delivering Web services of some kind got the idea that the Web was computing. I think the prototypical version of this thinking led to Rob Pike's observations about systems research in the early oughts. This forum tends to operate with that underlying assumption about the lens to view computing through no fault of anyone in particular, which is why I say that.
Perhaps I disagree with the implied assumption that a big web site was the answer. I don’t have an answer for you beyond that, because step one is challenging the increasing belief that Big Web has computing figured out and we need not evaluate the decisions and foundational technologies and delivery models that have led to where we are. Step one is a mountain, as you can imagine, and it’s possible to start the conversation of climbing it with only a handful of ideas of what could be over it. Many smart people with quadruple my intelligence and foresight have toed these waters throughout computing’s history. They just lose to hype and marketing.
I realize that’s an underwhelming answer, but if we asked that everyone who disagreed with the status quo simultaneously proposed a perfect solution to fix it, we would be nowhere in any sense. I assure you that thinking about the answer to your question is a significant part of my life, and talking to people who use computers to shape that thinking is key. Government was largely responsible for setting the computing agenda and spurring all of the right technologies in the beginning, and I believe there is merit to revisiting that thinking rather than ceding computing to engineers on the West coast.
At some point in the early-mid-00s we as engineers just "gave up" on the web as it was formulated as a document/resource transfer system with hyperlinking, and turned it into this ungainly mess of DOM manipulation, rampant RPC calling, database abusing, JS bound bloat. Finding the HTML document under the weight of ... stuff... on top has become impossible.
There were precursors to this... GWT was an early example of abuse of the web browser stack to accomplish something very non-webby. But it's really gone full bore now.
Back in the mid-90s when I made web pages for a living when some designer came to us with a photoshop mockup in a PSD and said "make the page look like this" we felt perfectly fine saying "no, the web can't do that. design for the web, not photoshop"; no such luck now. And so here we are...
You should also consider that the contrarian nature of HN would set it up to exactly the opposite reaction had the post expressed the reverse of its current positions. The big irony is that it's not inconceivable that it would involve some of the same actors voicing different opinions.
> "the incentives of modern software engineering value complexity to artificially strengthen a software engineering job economy"
Not sure I would agree. Hanlon's razor comes to mind. To me, the referred complexity is reminiscent of a number of other complex patterns that somehow found their ways in software engineering, out of possibly good intentions. Regardless, an idea needs to be sold and when the marketing is well executed the community will ingest the kool-aid. Which it then tries to digest for a few years. Then begins a game of chicken that consists in not being the first to admit that you're in pain, out of fear of being branded an infidel, or out of sunk cost bias. Eventually a few courageous and influential souls finally spit it out in disgust and scream "wtf is this shit?!?" Then the rest awakes from the common stupor and a holy war ensues. I still remember SOAP. I still remember with what verve I was sold Scrum.
> " that somehow found their ways in software engineering, out of possibly good intentions"
It seems that systems and principles invented for some specific situations get rapidly entrenched as "best practice" due to the wild success of their root corporations. That's twisted much like fashion - "that dress looks great on that model, maybe if I wear that dress, I would look good too".
I'm alarmed by the way you've brought up Mikey Dickerson completely unprompted (both here and in the OP's comment section), and that your account is just two hours old. It's fine if you take issue with the Silicon Valley mindset in government IT, but there's no reason to make it personal.
The reason it seems like an attack is due to the hostile tone of your comments, the burner account, and the unfounded/unsourced accusation that USDS left its partners behind.
I apologize if you just wanted to discuss the merits of a startup-minded approach to government IT, but I'm honestly skeptical of your motivations. USDS ruffled many feathers among the entrenched IT interests, and I've seen what the propaganda response looks like first hand.
If you really believe that you're fighting the good fight, then I'd welcome a direct conversation. Even a phone call if you like. I promise that I'm not in any way similar to a QAnon believer.
Perhaps my beef with your cadre is assuming that questioning its merits is AFS propaganda coming from someone who’s never set foot south of Embarcadero and assuming everyone who mentions principals has an axe to grind. You’ve painted a lovely caricature of who you think your opposition is, but you’re looking in entirely the wrong direction. Hint: Two FAANGs.
I don’t speak on the telephone with underhanded accusations, sorry.
Perhaps because I’m a government lobbyist and consultant formerly of FAANG with 24 years of experience trying to warn the Web industry that the exact thesis of the article is turning non-Web sectors against them? Couldn’t possibly be that, no, I have to be schizophrenic or a troll. The reason I know you don’t have schizophrenia close to you in your life is how willing you are to throw it at an anonymous commenter with a dismissive gesture toward “schizo”. You’d shut up if you knew what it’s like.
And you and “a proper response to accusing you of being a conspiracy means I’m being trolled when really I have no response” wonder why I change HN accounts like I change socks. Perhaps it’s because I hold this community in the utmost contempt precisely because of you and people like you, and I have zero interest in building a reputation among the harmful, toxic people who don’t realize how wrong they have it that I am working to excise from the conversation.
Take it or leave it. The good news for you is that you’re on the comfortable side of the argument so you will never get flagged or chastised by dang. Flagrantly accusing someone of psychological defect due to not liking their opinion is okay depending on the target and beliefs.
> There are people attacking the author for a statement made about CoreOS, and for some hate towards Kubernetes.
That's not really surprising, and maybe should be a lesson to the author. If you want to make a particular point, you shouldn't make inflammatory/controversial statements about other, only-tangentially-related things. It distracts from and dilutes your point.
For my part, I know one of the CoreOS founders, and while I have no opinion on CoreOS itself, it's really off-putting to see some random armchair quarterback on the internet shitting on the work of someone you know. While I more or less agree with the actual real point of the article, the unnecessary attacks really divided my attention.
The article starts off with a cheap potshot at CoreOS, followed by a screed about Kubernetes, and then finally, more than half-way through, we get an admission that the author has "digressed", and we get to the actual point. Certainly the point required some background about Kubernetes, but I can understand why people focus on the negative CoreOS and Kubernetes aspects of the post, since they consist of more than half of the content.
While I know there are people who argue in bad faith on the internet, I am genuinely curious and am not looking for a fight. I would have reached out to you privately but you don't have an email or contact listed on your profile.
I'm mostly curious about which part of devops you're skeptical about, if it's tooling, or process, or the mentalities or something else.
I think everyone here remembers that time they wrote a simple HTTP server, that just works, and would continue to work today.
If anything, I hope that we can recognize that there has been advancements in computing. Back in the day you would be open to attacks left and right. It was the wild west back in the day. Today, you can take that tiny webserver and put it in a VM, and voila, now it doesn't matter if anyone breaks in. As long as you didn't add the SSH key to github or anything like that.
We can have complex programs, as long as every-man doesn't have to know the details behind it, or implement it or risk losing access he previously had.
I think new protocols that are extremely hard to
implement falls under that, and I don't think Qemu or Linux does, as nobody loses anything when Linux adds new complex features.
We have to make sure, at least, that we continue to support HTTP/1. I think we already lost a lot when we moved away from IRC.
FCGI controlling Go programs, maybe front-ended by a load balancer and back-ended by MySQL/MariaDB/Postgres, can get an awful lot of work done cheaply.
FCGI is an automatic "orchestration system" - it starts up more instances as needed, restarts them if they crash, shuts them down when they go idle, and restarts them once in a while in case they leak some resource. It's like the parts of Kubernetes you really need.
Go is an OK language. Not a great language, but an OK language. It comes with almost everything you need for server-side applications. Also, the "goroutine" model means you don't have to fool with threads vs async. If a goroutine blocks on a lock, no problem; the CPU gets redispatched. If you need a million goroutines, that works, too. None of that "oh, no, the async thread made a call that blocked and screwed up the whole system".
MySQL/MariaDB/Postgres have been around for decades, and they work pretty well. You can have replication if you need it. (NoSQL seems to have come and gone.)
Get the stuff at the bottom right, instead of piling more cruft on top. It's cheaper.
What if 'advancements' were anything but? How would we know whether we're advancing a nascent engineering discipline in the best way possible? Is public cloud, an optimization for a certain calculus of engineering resources, the optimal form of computing delivery?
There's an underlying presumption about computing progress that has weaseled its way into your thinking, and the devil of it is that there are vanishingly few people willing to question it (can hardly blame them, given this thread).
You seem unfamiliar with the mystical energy force that always ensures the direction of "progress" is always positive for everyone. It's very un-techy to not believe in this, you need to immediately forget enough social science subjects so you can get back to the purity of ignorance.
James Mickens' 'Technological Manifest Destiny', cited at UseNIX 2018, is a real thing. At the time, listening to it, it was easy to chuckle, but I've since had a lot of sobering conversations with people in which I realized partway through that they really believed these things, and that's frightening.
Technological Manifest Destiny:
1. Technology is value neutral and will therefore automatically lead to great outcomes for everyone.
2. If we delay the rollout of new technology than we would delay the distribution of the inevitably great impacts for everyone.
3. History is generally uninteresting because the past has nothing to teach us right now.
You jest, but there is an actual argument somewhere in this dumpster fire arguing that philosophy cannot explain the overall direction of technology, as evidenced by a particular software's choice of license and the underlying assumption that technology is "different" from any other human endeavor.
Nebulous philosophical arguments do explain things. Quite readily, actually.
> Today, you can take that tiny webserver and put it in a VM, and voila, now it doesn't matter if anyone breaks in
This is a dangerous viewpoint to hold. Containers do offer some slightly better security, but they are not a silver bullet or anything close. There are still lots of ways to cause problems whether the software is in a container or not.
Short of CVEs, which are fixed after identification, can you point out other ways to "cause problems" with a tiny webserver serving static content in a container?
> What security do you gain by being in a container instead of on a server without a container?
I'm not a container guru, but... this is obvious, right? You have to break out the container, which is an extra layer. In order to get root on the server, you need:
• Without container: application vulnerability (to gain access to the OS) + kernel vulnerability (to gain access to root)
• With container: application vulnerability (to gain access to the container) + container breakout vulnerability (to gain access to the OS) + kernel vulnerability (to gain access to root)
You make it sound like I'm missing something — what am I missing here?
Why do you need to access the OS? If you break out of the web server, you have control of the container. You can then launch any sort of internal attack at the access level of the container. You could also launch a DDOS from the container.
Most of the bad stuff you can do when you take over a web server doesn't require root access, just user level access, and you get that whether or not it's in a container.
Containers share a platform. They increase attack surface by supplying a vector to any neighbour, by being something that can be attacked at all, and by requiring another control plane that requires credentials, making it a social engineering opportunity as well as being another component that itself can be vulnerable.
They are not an additional shell of defence, they do not reduce your blast radius.
> They increase attack surface by supplying a vector to any neighbour
Could you explain what you mean by this? I'm still not getting it.
Here's the example I have in mind: I have one application listening on a TCP port, and another one listening on a different port. I don't want the first application to talk to the second application. If they're just two processes on the same machine, they can both see each other, and an attacker who manages to exploit an RCE vulnerability gets access to the port "for free"; if they're both in containers, then they can only see the ports I specify, and it's really easy for me to specify them.
This literally does seem like an additional shell of defence to me! What sort of attack would it make worse?
Trusting the isolation like this is absolutely misguided, to the extent that it’s in conflict with understanding how computers work. Those who do not remember rowhammer or spectre and their kin are doomed to repeat them.
As for your example, there’s no difference between that and OS mediated mandatory access controls, separation of uids, chroot etc. Those capabilities are present irregardless of whether you’re using containers.
Anyone selling you on containers as a security measure is pitching snake oil. They are a resource allocation & workload scheduling tool. They may help applications avoid inadvertently treading on one another’s toes e.g. avoiding library conflicts or easing the deployment of, say, QA and showcase environments on the same metal, but it’s intrinsically a shoji wall.
There’s even a cultural downside, since developers may make exactly these flawed assumptions about runtime safety, or hard-code paths for config files rather than making them relative to deployment location, make assumptions (implicit or otherwise) about uids and so on, i.e. containers can breed sloppy dev practices.
I feel like you are several metaphors removed from me right now. Rowhammer? Spectre? I am just trying to serve a couple of websites here! I'm not going to buy another rack of servers just to isolate the two. Have you seen how expensive those things are?
> Anyone selling you on containers as a security measure is pitching snake oil. They are a resource allocation & workload scheduling tool.
I agree with both of these statements — and I suspect this is where it all falls down. When you put all your applications in containers, you are not done with security, no. Nevertheless, I've found great value in having one file per version per application, being able to upgrade my dependencies one-by-one, automatically rolling back in case of failure, and taking advantage of the tools in the container ecosystem. With all that, the security features such as isolating filesystems and network ports are just the cherry on top. Before containers, I was thinking about how to give my applications limited filesystems to work with, but along came Docker, and I didn't have to think about it anymore, because I'd already done it.
This is why I feel so out-of-touch with many of the commenters here. I'm surrounded by people decrying containers for security reasons, I want to defend them because of the many benefits they have given me, and I think the people preaching some kind of True Security (where everything is 100% perfectly isolated) aren't taking this into account. I feel like I could take your comment — "Trusting the isolation like this is absolutely misguided" — and apply it to any part of the stack. You have to stop somewhere.
I've seen the salespeople. They definitely exist, but I don't think they're here on HN, and they're more likely to be learning than lying.
And developers making flawed assumptions? You know this didn't start happening with containers!
> I am just trying to serve a couple of websites here!
Don't underestimate your responsibilities as an active participant online and operator of a globally reachable computing resource. Like driving a car, if you're not fully qualified and alert then you can endanger yourself and others. This is how PHP got such a bad rap.
The fact that the container deployment tool has helped you configure security elements (such as limited filesystem access) is nice, but those elements already existed, along with the tools to manage them. Containerized software deployment did not invent them, and if you're getting procedural benefits from containers, that's great for you, but that's all it is.
Having an additional layer of restriction (be it an alloation and workload scheduler or whatever) is additional security. The fact that there are vulnerabilities that bypass layers of security is irrelevant. The power button on the machine is a vulnerability to your software. The fact the restrictions are possible with other tools is irrelevant. Grouping functionality into a singular paradigm has utility.
To be fair, more security by additional tooling adds a vulnerability in human error, but that's true of all the tools.
The problem, I think, is that the article presents all of its views as almost self evidently true. If you distill it down, the complaint is that etcd added gRPC.
I think it was a good move for an infrastructure piece like etcd to add gRPC. Now I can just grab a generated client in a language of my choice. There's certainly valid critiques of gRPC / protocol buffers but I've found things like gRPC and Thrift to reduce complexity in applications that use them, because there's a thousand interpretations of REST out there, not to mention the crazy things people do with 'simple' HTTP, such as long polling, which introduce implicitly complex semantics into something that is on the surface quite simple.
> Now I can just grab a generated client in a language of my choice.
But you can't curl the state of your infra component without creating a program and downloading the client artifacts. This is a very big step backward from an operability standpoint.
> because there's a thousand interpretations of REST out there
...but there weren't a thousand different interpretations of the etcd API out there: there was exactly one. And now there are two, and the one with the most utility (that can be used without generators and in languages where gRPC support is spotty) is being diminished.
While I agree with you that there could be design consistency arguments in the abstract for gRPC vs REST as a global architectural pattern, none of those applied to the concrete artifact of etcd client<->server communications.
Not available in any repositories and largely unknown to general masses (much like gRPC itself).
Personally I like gRPC and sometimes use it in projects. But the author is right — attempting to gradually replace an existing HTTP API via gRPC API is like replacing horses with genetically modified cows. You need a very... corrupt mindset to ever attempt anything like that.
So... being a comment from that "guy" that made the original comment :-)
To reemphasize the problem with grpcurl.... it's _still_ static. You must download the artifact. It's like needing a different phone for every person you call (protobuf artifacts) instead of calling different people on the same phone (same artifact, different address and data in payload).
Yeah I don't disagree, I was mainly pointing out that there isn't a black and white 'grpc = bad' argument.
Especially since I believe grpc does take steps to mitigate some of the RPC issues you'd have with, say, Thrift. For example having the REST gateway so you can support the curl scenario without any extra work. (For example when building my own Thrift-type services I would do extra work to have a mini REST API for status and other operational curling uses)
As to the second part of your argument about there now being a second API, and how that is not necessarily a good thing, I quite agree and can't really speak to the etcd use case, other than that I don't think it's some property of 'Xooglers' to do such a thing to an open source component. I've spent my career using open source components: I've had lone developers who change their APIs willy nilly to fit the pet peeves of their own organization; I've had large companies own an open source component and have it be very disciplined about version migration; I've had lone developers be similarly disciplined; I've had companies make open source projects really complicated.
> For example having the REST gateway so you can support the curl scenario without any extra work.
I would agree with you except for one thing--it's very clear that's a second class citizen. There is this totally extra thing to support the original use case of etcd, and it comes with all the extraordinary quirks of how gRPC protobuf -> JSON serialization works (a strong term in my opinion).
It is very very hard for me to not see an unintentional conspiracy to push this very bad standard of gRPC. It does NOT make 99% of real-world programs faster. It does not make operations easier. It is not even a "data" format at all: go try changing an enum in downstream systems.
The standard is bad. Objectively, terribly bad. Someone somewhere on HN said that "Google was a ball-pit for gifted children." I kinda laughed at that, but really, after k8s and protobuf3, I won't even grant "gifted".
Google is a ball-pit for autistic children who can be funded by their monopoly on search. I'm tired of dealing with their protocols that make everything worse.
> Someone somewhere on HN said that "Google was a ball-pit for gifted children." I kinda laughed at that, but really, after k8s and protobuf3, I won't even grant "gifted".
To your point: it's situations like this that could (likely) be mitigated if they would have just upheld backwards compatibility: keep HTTP, add gRPC as alternative, 1st class ways of interacting with the tool. I haven't used etcd directly beyond K8, so I don't know if the new versions inherently invalidated the use of HTTP. But seems to me this is just another example of breaking changes...well...breaking stuff.
> But you can't curl the state of your infra component without creating a program and downloading the client artifacts. This is a very big step backward from an operability standpoint.
You can still do this because the gRPC API is also exposed via REST.
But I also think that the original author has a point: for many solo developers it's an extra dependency that they have to learn, and they get virtually no benefit from using grpc because they aren't performance-bound. They are simplicity-bound.
> they get virtually no benefit from using grpc because they aren't performance-bound. They are simplicity-bound.
If you're talking latency, to this day gRPC in dynamic languages is still slower than most fast json implementations in _latency_.
It is simply not correct to talk about "performance" this way.
If you care about CPU allocation and not burning 70% of your CPU on JSON parsing, then sure. Most companies don't care about that, and I/O is 3 orders of magnitude a bigger problem than in-process parsing for those problems.
I think this is another reason why solo developers don't benefit from grpc the way a billion dollar company does: solo developers often aren't using languages that are optimized with grpc (eg they're using python, node, Ruby, sometimes go; they aren't using c++, Java, etc)
Obviously these aren't hard and fast rules, but Google can't just spin up 10x more data centers to run their resource intensive stuff in something other than c++ or Java. They need the efficiency in a way solo developers or small teams usually don't.
Yep! That's the nuanced argument I'd look for, as opposed to 'gRPC bad, they added it because they're from google!'
Simplicity wise my favorite scenario is where if I'm using Python I can pip install an interface (or maven, or NPM, etc. etc) which is the versioned, supported interface for a component. REST + Swagger can help a lot there. So can gRPC. But I can also get how your favorite scenario might be to explore a REST API and build your own client.
That would have been a critique of Thrift, and a valid one, but gRPC has an http gateway interface with semantics for how the RPC calls map to REST calls and JSON. As long as that is supported, then there does not need to be a regression.
Sure, if the provider of the gRPC service wishes to also offer an HTTP gateway, that's true.
But it's trivially true in the case where the provider does so wish (as it would be true of any protocol with a fully-functional HTTP gateway infront of it), and it's untrue in the case where the provider doesn't.
"Now I can just grab a generated client in a language of my choice."
Except when you can't because no client is available in your favorite language. Then it is a lot harder than a rest api would have been.
> the article presents all of its views as almost self evidently true
Not so. The article, imho, presents all of its views as unabashedly opinionated views of the author.
The complaint that simple, lucid and well-fit functional designs are getting hijacked by large-corp vested interests has been adequately taken up elsewhere in this thread.
He is talking in absolutes (Kubernetes being the worst piece of software, etc) and trying to convince reader that "megacorporations" are the faceless evil enemy and they are making everything worse for everybody. Either you are with him or against him (servant of the evil megacorps' interest), there is no middle ground.
This person is not looking for a conversation, but looking for a fight. He seems to be angry and cannot direct his anger to "make good art" (Neil Gaiman - Make Good Art), instead trying to insult people and pushing them to a place where their emotions takes control and can be dragged into the mud.
Sorry but even though I disagree with most of the points from the article, I prefer to read this instead of a shy and unconfident rant from someone that's afraid of insulting your favorite tech.
What about a third option, where the author makes focused criticisms supported by clear explanations instead of the shy and unconfident strawman you've constructed?
Here is one from me with the solo-dev/small-team in mind - https://github.com/imaginea/inai . Good or not I don't know. It came out of some reading I was doing and I'm having fun with it due to the "deploy first and then dev" mindset it encourages.
There are some irony on the text, but to understand this is very naive.
The critic is at malicious/dumb people who latch to technologies without understanding them.
For argument sake, see how many people are defending coreOS etc, and how they claim to have decided to like it based on their superb expertise, yet they will likely fail to explain to you chroot or anything else containers are based on. Or how they will proclaim that LXC is pure garbage and Docker is better (i will not explain the joke, let them downvote :)
Bottom line: we are being sold bad ideas from the past, in new clothing, by the incompetents/malicious people, which megacorp is made of.
So your response to him pointing out that the author talks in absolutes, and tries to shit on any tech he doesn't like...is to talk in absolutes and shit on tech you don't like, with little jabs and dumb in jokes? Very convincing.
Read my comment again and you will notice there's not a single comment against or in favour of tech. Not even an adjective next to tech, only to people.
The comment is exactly that some people can't tell the difference between two completely identical techs, just because they buy the marketing of one of them, despite it being a brand on top of the first.
>> It's almost like there's no good representation in the open-source world for the solo developer or small team.
As a solo open source developer who maintains a popular OSS project, this is 100% the case. Sometimes it feels like big corporations are actively working against us. It's almost impossible to get any coverage at all on any kind of media. No matter how much organic growth your project has.
My project has been growing steadily and linearly for almost a decade. Now it has almost 6K stars on GitHub so you'd think that this by itself would draw some attention? Not so.
It's almost impossible to find it on Google search results. Most users find the project through direct word of mouth from other users of the project. We've never received a single consulting contract from any corporation.
You’re going to be bitter and resentful for a long time if you believe that the fact that you haven’t received a single consulting contract is due to big corporations working against you.
It sounds like you’ve built a useful piece of software. However, you don’t get handouts because you’ve built something. Those who are successful actively promote themselves and their products. There’s a reason companies aren’t comprised of just software engineers.
However, i’m not quite sure what it does (it obviously does something really well, but I’m probably not precisely the target audience)
It might help adoption and re-share value if your readme starts with some problem and shows how your tool solves it. The cadence/temporal people did this really well: https://www.temporal.io/
A lot of the complexity is just intrinsic to working on distributed systems, and if you're doing that kind of work, then these products really can help reduce overall complexity by giving you an admittedly complex solution that is at least robust and pretty well tested; that tends to be better than constructing your own.
I think where a lot of the pain comes from, and the article hints at this, is there are a LOT of problems that do not require a distributed solution, and while you can totally solve those problems with a distributed system, it's not really useful to do so in a way that dramatically increases complexity.
This is a known issue, and there are simpler models for these simpler cases. I think the problem is all the hype around k8s has driven a lot of square pegs into round holes.
Yes distributed systems are complex, but IMHO at least a large part of that complexity is caused by people who think they understand the problem, and go off and create some "great" new technology. Which then gets a lot of press, and other people building systems on it, only to discover that the edge case being ignored as inconsequential starts to create lots of failure modes. Sometimes these critical problems are "human factor" ones like with kubernetes configuration, or they are just a lack of proper engineering hygiene. Then you get another layer of crap on top. Repeat that a few times and what you actually have is a giant mess that frequently only works through the sheer effort of a large operations team.
To put this another way, if you read the Jepsen reports overwhelmingly what comes across is a lot of hubris. Its overwhelmingly old stodgy technology these new systems are meant to replace (posgres for example) that are satisfying the promises being proclaimed on high by $NEW_TECHNOLOGY.
I don't look at the Jepsen reports as being a representation of hubris. Most of what Jepsen finds may have been preventable problems, but they are also fixable problems, which is why I don't see hubris.
There's just a reality that we don't really build these systems on top of a distributed operating system. They're starting with an platform abstraction that is already terrible imperfect for the job. There is a ton of pressure to get something out quickly and make it accessible to a broad set of developers rather than to do it correctly (MongoDB is perhaps the most classic example of this). I was building big data systems in the early days, and the whole mentality was that you were building tools that were riddled with flaws, limitations, and outright bugs, but in the right context would make capabilities accessible to a broad set of developers that were otherwise completely off the table.
The truth is, if developers were already experts at distributed systems, they wouldn't need or want most of these tools; a LOT of the value is in accessibility and having something that is "good enough" rather than correct. The products reflect that more than they reflect hubris.
What I do see is a combination of marketing and customer ignorance that does believe in silver bullets and belief that these tools don't have to be understood to be effective. It's an unfortunate byproduct of focusing on that accessibility over correctness.
Then the author should have focused the article. If their goal was to address that, then don't take a left-turn into hating on Kubernetes for no reason, before even reaching the point trying to be made.
> Funding and adoption (in the extreme, some might say hijacking) of open-source projects from large corporations dictates the direction of all major software components today
This is because large corporations are the biggest users of these pieces of software. Its literally that simple; money only plays a smart part of it.
Etcd's biggest user is Kubernetes. I would estimate that 2nd place isn't even close. I would estimate that Etcd wouldn't even exist today if not for Kubernetes. The direction of open source projects is usually some approximation of a democracy; its users contribute the code they need. If they need gRPC, it gets added. gRPC wasn't added for no reason, or because politics; its a sister CNCF project, and would increase the performance of the API surface, which greatly benefits all of its users.
Lets do a little experiment: Go start an etcd clone. Just go do it! The code is still there! It would be four clicks in the Github UI. Strip gRPC, add the HTTP API, and maintain it. I'd bet a thousand dollars the author won't do it, for two reasons. First, the author hasn't actually used etcd. I know this because, as I said, startlingly few people outside of Kube use etcd directly, and the author isn't providing any substantive technical reasons why the gRPC change was bad; just philosophical ones. And Second, because maintaining large open source software projects is soul crushingingly, destructively, enormously hard. Its way too easy to armchair-quarterback decisions like these based on your own personal flavor of ethics, then cherry-pick the evidence you want to say "corporations are evil! they destroyed etcd by getting rid of HTTP!" Meanwhile, people out there do develop it, and use it, and service billions of requests, and deliver value to their users, and sometimes, turn off their computer and cry because despite all of the hard work and positive results people with no skin in the game still pipe up to say what they're doing is horrible.
Did the author join in on the discussion when these changes were made? Did they voice their concerns? Or did they just read about it after-the-fact and say "jeeze, that piece of software I played around with two years ago really took a turn, I'm going to write a post about how evil corporations are."
> gRPC wasn't added for no reason, or because politics; its a sister CNCF project, and would increase the performance of the API surface, which greatly benefits all of its users.
gRPC (Google RPC) was absolutely added because of politics. Yes, it's performant, but let's not kid ourselves that gRPC wasn't picked because it falls nicely into the orbit of Kubernetes and the Google ecosystem. It's also why etcd can even threaten to remove support for HTTP+JSON (!!!) in the API, when it costs nothing at all to keep that support. This is exactly what the post is bemoaning.
> Did the author join in on the discussion when these changes were made? Did they voice their concerns?
Did you? Did anyone outside of a few people in a SIG or an RFC? This is also the complaint the author is obliquely getting at. Google may be 95% of the etcd _volume_, but they are basically 0% of the institutional users. That a small group of people can push a project in a direction that uniquely benefits them while adding an incidental complexity tax for everyone else is worth complaining about.
I feel as if I have to repeat this during most discussions about Kubernetes and its halo technologies: Kubernetes is not developed by Google. gRPC is not developed by Google. etcd is not developed by Google.
These are all projects under the Cloud Native Computing Foundation; its Platinum-level sponsors include: Alibaba, AWS, Apple, ARM, Cisco, Dell, Fujitsu, Google, Huawei, IBM, Intel, JD, Microsoft, NetApp, Oracle, PaloAlto Networks, Red Hat, SAP, and VMWare. Also included are 20 Gold-level Sponsors, 415 Silver-level sponsors, 3 Academic sponsors, 13 Non-profits, and 100 End-user supporters. That's a total of 570 organizations with a vested interest, and voting rights, on the direction of their projects.
But Google does a lot of technical oversight, right? A non-zero amount. Among the eleven people of the CNCF Technical Oversight Committee, Google has ONE representative (Saad Ali. Learn their names. These aren't just faceless mega-corps we're talking about; they're real people). Microsoft has the most, at two. Also represented is Apple, Intuit, Docker, American Express, Aqua Security, Lyft, Rancher, and Alibaba.
But, ok, Google "sneaks in" a lot of code, right? They're super-evil, that's the message we're trying to convey here. Of the top ten contributors to etcd [1], there's only two people who seem like they work at Google. The top contributor, by far, works at AWS.
Alright, fine. Kubernetes was created at Google, this much we cannot deny. Started by Brendan Burns (now at Microsoft), Joe Beda (now at VMWare), and Craig McLuckie (now at VMWare), all Google engineers a decade ago. Yeah, the thought leaders behind the project sure are still making that sweet sweet Google money, sure seems like it.
Its just tremendously incredible to me that anyone still believes Google has a large say in CNCF projects.
Cloud Native Computing Foundation was spearheaded and initiated by Google as a competitive alternative to AWS's dominance.
Google knew they couldn't catch up to the AWS offering by selling service alternatives, so they intelligently and astutely zagged, and went all-in on an "open source cloud" approach.
Kubernetes is the Trojan horse - it gives people hope that they can develop with open source, and be cloud agnostic. But when the complexity overwhelms and drains them, they invariably look for a "managed" kubernetes solution to take the load off. And, wouldn't you know it, Google happens to offer a managed Kubernetes solution! Sure, so does AWS, but EVERYONE KNOWS (whether you like it or not) that Kubernetes was developed by Google, so surely they would be the ones best at managing Kubernetes ops in production!
> I feel as if I have to repeat this during most discussions about Kubernetes and its halo technologies: Kubernetes is not developed by Google. gRPC is not developed by Google. etcd is not developed by Google.
And your condescension and repetition would show that you don't know what you're talking about.
These were all--100%--developed on by Google and moved into CNCF, which is uncomfortably influenced by Google. Big providers have made peace that they lost the standards war. (And while etcd was not originated from Google, its adoption into kubernetes was obviously influenced by its use of golang.... developed at, where was that again?)
Kubernetes and gRPC started at Google, were exclusively developed by Google, and then much later moved into CNCF as a form of legitimacy.
If you have never tried to get a change into gRPC, I would encourage you to try. It took 3 years for Google to allow an upstream change that would not require rebuilding PHP protobufs from scratch on every request, because it would change the way their C library did things.
> maintaining large open source software projects is soul crushingingly, destructively, enormously hard.
Are you making a case in this package that complex open source projects are impossible unless completely controlled by very large, highly opinionated companies? That can be proven wrong by endless examples.
> Did the author join in on the discussion when these changes were made? Did they voice their concerns? Or did they just read about it after-the-fact and say "jeeze, that piece of software I played around with two years ago really took a turn, I'm going to write a post about how evil corporations are."
You don't know this to be true, so why make this accusation? Why not address the argument as it is instead of inventing ad hominems?
No. I'm making the case that nebulous philosophical reasons for why technology is the way it is are nearly always unproductive, and is certainly unproductive in this case, which involves a very liberally-licensed open and libre-source project.
Its startlingly easy to write five-hundred words. Its startlingly difficult to accomplish what the etcd team has; I therefore give the benefit of doubt to etcd.
I don't fully understand why etcd switched from HTTP to gRPC. It seems weird to me. I wouldn't do that for any project I run. But I'm not going to write a blog about it. I'm not going to get angry about it. I'm not going to spin a conspiracy about ex-Googlers spreading gRPC everywhere they can. Instead: there are a ton of people developing and using etcd, who aren't me, who know what they're doing, and their results speak for themselves; They probably made the right call.
> You don't know this to be true, so why make this accusation?
Those are questions. But, yes, my tone was accusatory. Generally, if you have skin in the game, you don't write posts like this. That's why having skin in the game is so important; even if you lose the fight, you're left with positive remnants of the work you've done and the people you've interacted with. There are positive ways to have a negative discussion about technology. Instead, this article goes obscenely negative about etcd based on one thing the author disagrees with, then spins it into being negative about Kubernetes, Electron, Docker, corporations, and finally goes personal by directly criticizing the "expats from a megacorp ... who just want to build big ungainly architecture"
But also, you can look at their github and see their lack of contribution to etcd, if you really wanted to go full-stalker and demand evidence.
> I don't fully understand why etcd switched from HTTP to gRPC. It seems weird to me. I wouldn't do that for any project I run.
At a (very uneducated guess) I'd imagine that the query payloads would be smaller with gRPC since it's binary which would save money when running in the cloud for things like traffic egress, cross-zone, or cross-datacentre billing. HTTP can be unbeliveably verbose.
It's also probably quicker to deserialise which means you get more bang for your buck on your machine spend.
> At a (very uneducated guess) I'd imagine that the query payloads would be smaller with gRPC
> It's also probably quicker to deserialise
These are pretty educated guesses :-). This kind of reasoning has been behind a lot of Google initiatives, such as protocol buffers. If you're slinging enough data around, smaller payloads and faster serialization/deserialization does have a significant payoff. Once you're marinated in that mentality it probably sticks.
Moral reasoning premised in attachment theory(which has been a very productive field in the past decades) and extrapolated to the inanimate would draw a different conclusion.
If you are that tied to your work you may be treating it as a child, and as a child, your moral reasoning will proceed with the underlying premise: "what is good for the child is good", leading towards a parenting of the project ahead of other needs.
But a software system is not a child, in fact. It's just an expression of ideas. "Skin in the game" signals that you've been dragged into being a parent. Is being a parent for software the right thing? Perhaps, if the software's premise draws upon a strong justification. But most of these tools have not been emerged from standalone justifications, but from solving something else, which neatly ties back into the author's argument: the corporate needs are what are complex. And here attachment-based morality recurses: if the corporation is the child and you are tending to it, once again, you will put it ahead of other needs, and hence will develop the justifications for software complexity.
But if we zoom out a bit and look at the world generally, it's operating from a point of indifference: if the output of the corporation is pragmatically convenient, it is good, if it presents an obstacle then it is bad. And ideas - and hence software - that persist in the indifferent world, outside of the attachment relationship, are the ones that survive.
Which perhaps means that all of it is wrong, which isn't a very stunning conclusion if you still have to work with it. But that is philosophy for you.
You said: "Its startlingly easy to write five-hundred words. Its startlingly difficult to accomplish what the etcd team has; I therefore give the benefit of doubt to etcd."
That's a VERY confusing comparison and conclusion to draw from it.
If your project is sufficiently popular, that's safe; otherwise maybe not.
As someone else mentioned, when a giant corp absorbs a free project, you can reasonably expect them to maintain it. If you GPL the project, that won't happen. But if it's a popular project, then if it's GPL it will attract more maintenance from people who can't free-ride on the corp's efforts.
(Note that the relevant measure of "popular" is not number of users, but number of people enthusiastic enough to actually contribute.)
I love k8s, think it’s the best thing that happened to distributed computing since .. sliced bread? Anyway, it’s good, but I don’t disagree with the author I just have a different perspective. I see k8s as a vehicle for learning about distributed computing not just for me but the industry as a whole. Same thing about Linux, it’s just a stepping stone for node management and has served us well in figuring out a bunch of problems. K8s removes a lot of duplicated effort in the learning process for the industry, just close your eyes and think back to the past decades of ESB wars.. that was a waste!
“Under the radar” comes a new generation of better systems and components that learns from these learning vehicles of computing and will change how we deliver services fundamentally, but without k8s and Linux we would never have gotten there. Open source RISC-V co-evolved with seL4, WASI, webassembly and NATS comes to mind, but I’m sure the future will take its own path.
Personally I see this as coming from K8s having effectively two parallel uses.
One is for making it easier to manage large, complex infrastructure. The other is as a quasi-OS for containerized software.
Instead of using a new technology to move infrastructure complexity away from developers we now run development environments on k8s, piling even more complexity and layers of abstraction on.
Disclaimer: I'm was part of the Apache Aurora PMC before the project went into the attic.
Not sure what you mean by declarative infrastructure but eventual reconciliation has been implemented in at least one other project (now retired) called Apache Aurora[1] which runs on Apache Mesos[2].
Twitter ran (and probably still runs) a combo of these to huge scale and great success so I don't think it's fair to say that there's nothing else around that does that :).
K8s is great, but we should be careful not to rewrite history.
> vested interests from large companies are able to introduce huge complexity into simple, well-designed projects.
In this case, the vested interest are interested in making it as easy as possible for real world customers to reliably deploy their applications to the Cloud.
That's not an interest that favors unneeded complexity. The complexity comes from the need for a container orchestration system that satisfies the needs of large organizations. If you could satisfy those needs more simply, the incentives are to simplify.
Undoubtedly there is space to build a simpler tool with a subset of Kubernetes' features. And sure, you can pick at specific flaws in the way it's evolved.
But I'm not so convinced by the claim that the whole thing is a mess because tech employees like replicating the stack they are used to.
When I've worked on small teams we avoided complexity by hardcoding locations and using GUI management tools and all these other things that still exist - and honestly, are better today than they were in the past, when you'd also need to get yourself a rack in a datacenter to run stuff.
I think there's a big swath of people adopting fancy tools to try to solve problems they don't have and won't have for years, but I don't see that having destroyed the ability to do things the easy way.
Definitely true. The author could have chosen differently and it would have probably led to more constructive comments.
However, I'll also point out that focusing on the snark and ignoring the rest of the article is also a choice made on the part of the commenters.
Given a choice, I personally prefer to take the strongest and most charitable interpretation of someones ideas before choosing to criticize them. At least the discussion is interesting that way...
That things I write for the web usually go through a "lazy comment" filter where I pull out references that would lead to lazy comments. Things that often get stripped are one reference to a specific example when I'm talking about something general–because invariably the comments will focus on "how I don't get the software" or "how I singled it out" or "how what I wrote isn't even true if you look at this way so my point is now invalid". Either I mention no examples, and have the commenters come up with their own, or I have a few so that I don't get attacked by a specific group of people.
While I agree with you, I also think that snark is what makes the blog personal to the author. If it's not a personal reflection of one's thoughts, some may not consider it worth doing.
Perhaps the people who treat CoreOS being called bullshit as dispositive of the entire thesis, and an inclusion worthy of disregarding the author and thinking of him in any way other than making an argument, aren't the intended audience.
Pull the snark out and there will remain any element of this post that kicks off a violent HN centithread that conveniently avoids the core issue. You know it, I know it.
Snark directly related to the main point is fine. The CoreOS comment is unrelated.
In fact the author liked etcd in the time period when it was directly associated with the "bullshit" project. And even mentions "but that doesn't really matter".
Bashing K8S is at least related to the main point.
The author may or may not have the correct conclusion about over-engineering coming from big-co alumni, but if they can't get their one example correct, why should we discuss their post? Are their insults that novel?
> It's almost like there's no good representation in the open-source world for the solo developer or small team.
Availability heuristic: large companies get more press. Partly because they're large and therefore more likely to attract readers for stories about them, partly because their scope is vast and so journalists know about them, and partly because they intentionally seek press for their work.
The scratch-an-itch period of mass-market OSS is long gone. If indeed it ever truly existed.
I don't get it, isn't that the point of open source and free software, why doesn't the author, or a group of like minded people simply fork the old software and release a SimpleEtcD?
Of course, you could argue that the people who made it more complicated in the first place could/should have forked instead of taking over, but that ship obviously sailed a long time ago.
The problem, as I pointed out elsewhere, is that the post makes very few constructive comments and presents a lot of opinions and assertions without justification or reasoning, making it extremely hard to actually make meaningful comments about. As someone else pointed out, if it were an HN comment, it would be flagged.
I don’t agree that this is a statement about complexity, so much as it is a rant about the author being inconvenienced by technology they personally deem pointless and unnecessary. If the CoreOS comments aren’t a sufficient red flag, the HTTP/2 ones really ought to be. Also, some etcd users debate their assertions about newer versions being more complex internally. On the other hand, its difficult to argue with overly cynical type attitudes.
But that's just the "circle of life" no matter what. Things start very simple, and then inevitably get ambitious and try to add new gizmos. Sooner or later you end up with a monster.
Historically FOSS has been far more resistant to this cycle; not all FOSS projects, but many. It's much easier for FOSS projects to just say no, and they can spend years or even decades refining things. In the commercial world you can never say no. Not only that, but you have to constantly add features.
Now that corporate interests dominate in many corners of the FOSS sphere, this benefit of FOSS is diminishing. Which is a shame, because it's a big reason why FOSS was so successful, and why so many old commercial projects collapsed under their own weight, even though they always had more features than their FOSS counterparts and thus theoretically should have only grown their user base.
This is true - but I think the interesting (and understated) part of this argument is Big Tech's influence on the process.
Say I want to fork etcd or build a greenfield alternative - I now have to compete with (or support) the Hivemind's preferences in direction/progress. Who's to say their way is the "best" way?
I think this tension is much greater than it was 10 years ago.
> In the meantime, the simpler version of the software is long gone
The thing is that with free software the old software is almost never really long gone. It's still out there, free for your perusal. That is the beauty of free software, it's always additive process. But it does also put responsibility on the user to pick the patches they want, either themselves or by proxy. Ultimately you are responsible for the free code you are running, not some vague upstream dominated by fnaag et al.
While this is technically true, using old, unsupported versions of software (such as etcd) in production is a recipe for disaster, especially when you start running into problems with it.
I think, in the case of etcd, it would have been better for the k8s folks to fork the project and add whichever features they felt were needed to be able to use it for k8s. Instead, it seems like they swarmed the original project and added a ton of features that made it a behemoth of complexity rather than the simple tool it once was.
> I think, in the case of etcd, it would have been better for the k8s folks to fork the project
This would only make a difference if someone was willing to maintain the non-k8s fork. From what I understand, nobody was willing to take that on, so the k8s version became the only version by default.
It doesn't seem to me like the comment section is weird at all. This is kind of a perfect object lesson on pitfalls of writing a persuasive blogpost.
Basically, if you want to argue for a somewhat controversial conclusion, you shouldn't back it up with even more controversial and unsupported (or weakly supported) premises. Particularly if "even more controversial" means "in disagreement with the vast majority of your industry."
When you fine some cool tool you can:
1. Use as is.
2. Make behind the scenes changes that don't alter functionality.
3. Make changes that add functionality.
4. Make changes that remove existing functionality.
4 is the problem. It's like joining a car club then keeping pushing to have petrol cars banned. It is really shitty to find a cool tool that you can use, then try to force remove features that existing users use.
> The key point of the article is not really being addressed here:
Perhaps if the author wanted their key point addressed they could have tried focusing on it, rather than on an ill-considered spray across a group of largely unrelated bugbears.
Why can't solo developers or small teams represent solo developers or small teams? If an etcd that has a thin, inflexible API is of value, a small team should be able to fork and manage it (the notion it's smaller and simpler ought to imply it's a lot cheaper to maintain than a multi-API etcd, right?).
If the economics of the situation is "large teams have to maintain tools for small teams and individual users," that doesn't actually scale well, does it?
I think a large problem is open source projects actively marketing themselves, and seeking large user bases. This leads to a wasteland of zombie frameworks, and developers selling out (so to speak) in exchange for the reputation.
It is important to note, too, that etcd is permissively licensed and anyone can run the old version in whatever way they please.
The final, and biggest, takeaway? Hackernews apparently has drank the koolaid and will reach for any convenient justification to write off an opinion they disagree with. Stockholm syndrome all through the comments. Conservatism and pragmatism apparently are no longer current in the cloud economy.
A major problem that leads to dysfunctional bloat is the inherent ego traps of software developed by teams of well-adjusted people.
Even if it's for the good of the quality of a product, it is not often easy telling your team mate that they wrote an unnecessary component that should just be deleted. I have done it lots in my short career (and in terms of engineering outcomes, it's been spectacular), but it's easier for me I think, since I'm more disagreeable than the average person.
The author is not aware that etcd is still offering the JSON API he mentioned. https://news.ycombinator.com/item?id=23835651 so his main argument doesn't hold in the first place. The rest of the post is "Google's technologies like protobuf/grpc influence people too much, everything should be a json api".
He's aware of the old API, but believes it will be killed soon. From the article:
"The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely."
These open source projects you talk about were developed by profit seeking enterprises for solving complex distributed systems problems at scale.
Because they chose to make them open-source (not out of altruistic motives but out of the same profit seeking perfectly good business motives), these solo developers (who are also profit seeking) can benefit from those projects for free and can afford to build to serve lot more users than they would be able to otherwise.
I don't get the entitled attitude of the author. If he wants a
'simple' system by his standards, he can build one easily enough with the modern tools like golang, node.js and rust.
Btw, the web development tools were never simpler than today to operate a safe and secure production web application to serve any number of users. I'm saying this with about two decades of experience doing the same.
> the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys
I maintain a (/the only?) etcd3 library for Node.js[0], and used etcd extensively on my former team.
None of these things are new to etcd3 API. All of these are present on v2 as well[1], whose API the author extolls, or are built within clients on etcd's base APIs (e.g. there's no 'lock' API, only leases). However, with etcd3 we get stricter typing, better performance, and better semantics (e.g. watch streams and lease streams over polling) thanks to GRPC.
In general these rich APIs allow 'average' engineers to build complex distributed apps more correctly. I've built reliable sharding, hash rings, elections, and so on based on etcd's API--none in more than a hundred or two lines of code (more in Go, less in Node.js). All of these are classic hard problems that etcd makes easy. Sure, there's innumerable standalone services for each of these things, but often there's no need to take the cost of many tools when one would work.
I feel like the blog post could largely be summed up as "stop threatening to remove the normal HTTP API, some of us find that a lot easier to debug and ease of debugging is essential for a core operational component of a distributed system" - which is, I would argue, an entirely reasonable thing to want.
I quite enjoyed the rantiness on a "being entertained" basis but it did rather work against effectively making the core point.
> stop threatening to remove the normal HTTP API, some of us find that a lot easier to debug
I've seen the trend toward complexity in other projects too, and it harms not just ease of debugging, but ease of hacking.
Take, for example, Swagger UI[0]
v2 was so simple. It was vanilla JS using jQuery. I, as an embedded systems developer, was able to easily hack it so it could read in the OpenAPI JSON from a database and I even added a little search box so you could filter down the APIs you wanted to see. Super fast and easy and worked just the way I wanted it to!
Starting with Swagger UI v3, it became... extremely labyrinthine by comparison. It was completely re-written in React JS and now I need a bunch of new tooling to make changes and everything was broken out into dozens of different modular files so I couldn't find where I needed to make a given change, also not to mention I've never used React so it felt like the barrier to hackability was dramatically increased.
I'm sure full time React folks love the new architecture because it's so much <cleaner/safer/scalable/etc>, but for me the change was extremely confusing and made the tool unhackable (I tried for a few hours to get it to do what I wanted, but it started looking like I was just going to have to learn all of React and I threw in the towel), and so I'm permanently stuck on v2 for now.
> Complaining about an open source project changing technologies because you can't be bothered to learn them is a bit rich
Web technology is its own beast. I invested a long time learning and mastering AngularJS only for all that work and knowledge to be flushed down the toilet over the next few years. Web tech has terrible ROI so that's why I "can't be bothered" to learn the latest one. I'm salty specifically about Swagger jumping on <insert latest framework> instead of sticking with simple vanilla JS that everyone understands.
Author is really good at discrediting themselves straight off the bat in one sentence:
> for a ~~bullshit~~ unsuccessful project called CoreOS Container Linux that was EOL'd several years ago
CoreOS was actually quite successful to me as an outside observer. It had decent paid user base as well as people using it without paying. It has showed people that Chrome OS can be used to build atomically updating host OS with a readonly fs for container hosting. It has inspired many other container hosts to come like RancherOS and Google’s Container-Optimized OS. Similarly, it was probably one of the reasons why Red Hat was interested in the acquisition.
Furthermore, CoreOS was EOL'ed last month; not several years ago.
I can keep going on why etcd API was switched to gRPC and what benefits this offers, even at small scale. But I don't think it's worth anyone’s time convincing the author otherwise. Based on their tone, it's clear to me that they have trouble with using software when things get a tad bit complicated. Usually, there's community decision-making behind these decisions, and they're often deliberated for months, backed with prototypes and data. I'm pretty sure the author doesn't care, however.
My team ran about 50% of a Fortune 500 software company on CoreOS up until a few months ago when we migrated to Flatcar. We even paid a lucrative support contract while we ran it. I can count the number of major OS issues we had with it over several years on one hand- and we've had even better success with Flatcar so far.
Calling CoreOS unsuccessful is a massive misunderstanding of the market.
Chris from Kinvolk here. Happy to see you're having success with Flatcar.
We, of course, agree that CoreOS Container Linux was a huge success. The uptake that we've seen in Flatcar usage, especially since the CoreOS EOL date on May 26th, has been extraordinary. So from what we a can see, the market is there for a minimal Linux for containers and and we're happy to continue filling that need with Flatcar.
I'm not exactly sure what you're asking. But Flatcar Container Linux is completely open source. In fact, everything we do at Kinvolk is. We do not build open-core products.
That sounds really interesting!
I currently work on a side-project which is a distributed application and I plan on incorporating kubernetes/cloud-native into some of the core concepts, so I will check it out once I'm at this part of the implementation!
1. I repeated an argument made in the article itself. That cannot be considered flamebait.
2. I've always expressed strong opinions in my posts. Sometimes this results in multiple upvotes, sometimes in multiple downvotes.
3. About being "unsubstantive" - I underscored the main concept/rationale for Core Linux, and linked to its Wikipedia page. You could argue that it's not substantive _enough_, but when you phrase it this way it's as though you merely dislike the fact that I repeated the inflammatory term from the article.
I want to make constructive comments about this blog post, but the way it conducts itself makes doing so extremely difficult because nothing about it is constructive. This is troublesome for HN - I almost wish I could flag it for being impossible to discuss without turning into a flame war. It feels like several disjointed complaints based on a desire to jam any and all inconveniences into a unified narrative about Modern Software and Google. Dismissing CoreOS is the opener, but there’s almost nothing on the page that doesn’t wave red flags in some way, and I’m trying desperately to be open and objective.
I have worked with (and under) big names in system software and so many of them think saying "terrible", "garbage", and "trash" will motivate people to do productive corrections or improvements. Nope.
Times have changed and the very people he criticizes are infinitely better at branding and inspiration. There's a reason they're the ones gaining traction.
Like you, I want to be constructive, and I do agree with the author's core complaints. He just doesn't realize he's actively self-sabotaging and harming his own reputation.
It is essentially a rant with not much substance. If this was a comment on HN it would be flagged probably. Like to hit at so many things in 3-4 paragraphs you aren't making an argument for or against something, you are just making a scattershot blast on like most of systems stuff today.
Many reasons to switch from JSON to gRPC: * gRPC uses HTTP/2 which means you can concurrently make multiple requests on a single TCP connection, while on a typical JSON API which probably uses http/1.1, you can't. * Type safety of protobuf types. * Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane. * Client libraries are automatically generated, and not error prone. * RPCs are already optimized for bytes on the wire efficiency, whereas JSON is not. This matters a great deal as Kubernetes objects get large in size/quantity over time but controllers still work effectively by not spending too much CPU on encoding/decoding like they do on JSON. * Similar to the previous point, most json decoders don't reuse objects, so every decoded object is a new alloc, whereas gRPC can Reset() and reuse the same object while decoding/encoding. * Builtin authentication primitives (such as JWT/tokens or even adding TLS to client and/or server). * gRPC has support for interceptors which are like middleware functions you can inject to requests/responses on both client and server-side for logging, authorizing etc.
The list goes on, but something to note is that etcd was not developed by Google (and as far as I know, not by ex-Googlers). Both etcd and gRPC are owned by the same open source foundation, so it's natural that they make use of an existing technology.
> gRPC uses HTTP/2 ... while on a typical JSON API which probably uses http/1.1, you can't.
What are you actually talking about? An HTTP API by itself doesn't actually care about which http version is used, that's something only the HTTP server that serves the API should care about (nowadays, pretty much any http server in any language supports HTTP/2 and 1.1).
> Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane.
Even HTTP/1.1 has mechanisms for that, like long polling and chunk encoding (which allows infinite streams to keep a dual channel of communication open where each chunk can be treated as a message - with "headers" and all).
> Client libraries are automatically generated, and not error prone.
This is true, but we've had similar technology since the SOAP days... you could easily do the same with a XML-based HTTP API.
By the way, most of your points are against JSON, not a HTTP API per se, which usually would allow a number of formats , including XML and JSON at least.
> Builtin authentication primitives...
HTTP also has that when you include cookies and something like OAuth/OpenID.
> gRPC has support for interceptors which are like middleware functions
This kind of thing is better done by using the specific platform you're running on (depends on the language) so I don't see something like this as something desirable on the specific RPC framework you're using. With HTTP, caching, logging etc are trivial to do and one of the strongest advantages of using HTTP in the first place, so I think you're confusingly making a point for HTTP here, unless your focus is on the RPC side of things? In which case, you would have to make a case on why RPC is a better fit than HTTP for etcd, which you haven't.
So all in all, I found your points utterly unconvincing, but presented with so much conviction that you're actually right that I could not resist to respond (even if I don't want at all to get into a pointless discussion on the merits of HTTP VS RPC or JSON VS Protobuffers)!
By using gRPC, we get much more without thinking too much. Also, bi-directional streaming is instantaneous and it is not like afterthought long polling.
It is pretty straightforward and not too hard to understand the API. Designing APIs with protobuf makes things convenient and it brings lots of already written tools with it.
Since gRPC built on top of HTTP/2, we thought gRPC as easiest and performant way of writing a HTTP API with good defaults.
"we get much more without thinking too much" is precisely one of the reasons for the original push to REST (and against RPC at that time SOAP and CORBA and RMI etc.) was made in our industry 10-15 years ago. By falsely representing a _remote_ resource as if it was a _local_ one we open a whole can of worms; not just performance, but resilience, infrastructure issues, etc. Transferring documents over HTTP with a common language of HTTP's verbs was supposed to get programmers to model their applications as the sets of resources that they are, discourage them from making excessive round trips, make debugging easier, and make use of standard HTTP load balancing hardware.
Yes it's easy to make that RPC call. But should you?
I've been off doing other things in the intervening period, but while I had my back turned the industry seems to have turned its back on REST and gone whole hog on RPC. Again.
At first I was thinking this was just internally here at Google, where protobufs and gRPC reign supreme. But it seems to have taken hold everywhere.
What did I miss? Why have we swung this way. Again. Is the pendulum going to go back?
> By falsely representing a _remote_ resource as if it was a _local_ one we open a whole can of worms
I've heard this argument before, but how does gRPC itself cause this issue to manifest? I'm curious to hear what your opinions are on a better alternative, and how not to represent a remote resource as a local one.
But the fact that it presents the remote resource in an API which resembles a local object means that programmers often get lazy in the manner of which they think about these things. The REST semantic is supposed to make this more explicit.
A remote object is not an object in your program or even your computer. It's something you're taking from something that is computationally miles and miles away. Compared to the microseconds it takes to dispatch a local call, it's an eternity away, even on a local network of the highest speed.
Accumulate those latencies over thousands of dispatches, and trouble can ensue.
I am reminded of an observation from when I first joined Google, coming out of their acquisition of the scrappy awesome ads company I worked at (Admeld).
We had a little service that kept track of ad impression caps / budgeting. We didn't want to serve an ad a single time more than the customer wanted us to, etc. Serving many thousands of ads per second, a process distributed across multiple machines in multiple data centres, this is a bit of a tricky shared state problem. The people who came before me had designed a rather clever solution which used a form of backoff to trickle down the number of ads served as they got closer and closer to budget cap, and to synchronize this state across clusters (this was before there were Rafty services to make this kind of thing easier, BTW).
We had a bit of a show and tell with Google when we first joined. I wanted to know how they were handling this problem, since their scale was many times ours, so I asked the question and got a puzzled/annoyed look:
"Oh we just make an RPC call to our budgeting server."
Summary: if you're at Google you don't have to worry as much about these problems. You still do, but there's an insane amount of infrastructure and horsepower and an army of SREs to help make it happen.
So, yeah, my point is -- just because Google does something or has invented something doesn't mean it's the best way to do it, especially in a smaller more cost conscious organization.
Thank you for taking the time to write this up, and the interesting anecdote.
What I don't see though, is how is making a (g)RPC call any different from making a REST call? Like you said, the REST call is supposedly more explicit, but at the end of the day, it seems more of a convention and some hard underlying difference. What's the difference between `httpClient.get("...")` and `grpcClient.foo(...)`?
The latter encourages you to model things as remote procedure calls, the former, well, in my experience it's open to incompetence and abuse, too, so... ehh... but done properly... well, go read the Roy Fielding paper :-)
I mean, internally at Google we have load balancing for grpc (I'm sure the outside world does now, too, but it was new to me when I joined) -- but load balancing HTTP requests containing readable JSON or XML documents, that's far more sysadmin friendly, wouldn't you say? Off the shelf infrastructure, nothing exotic, easier to monitor. Same goes for caching, for proxies, etc.
Being able to just stick a URL for a given resource in your browser, or hit it with wget/curl to read it, that's a serious bonus.
In general URLs follow conventions similar to those down by our Unix forefathers, when they designed the filesystem API. We are all familiar with this model. And in some ways REST done right is very Unix philosophy -- provide a consistent model upon which a bunch of little tools can interoperate.
I could go on... have to go put my daughter to bed tho
I don't think that's accurate... as the article claims (and it seems believable) it has only appeared in etcd because of Xooglers interfering with the project in the name of Kubernetes (also from Google). Or do you actually have examples of gRPC being used in many other non-Google-related projects?
Could you share some info about the switch to gRPC, not for the authors benefit, but for the rest of HN? As someone not familiar with this story, I’d be very curious for your insights!
Entirely agreed, CoreOS Container Linux was a great project that served its purpose well. Yes it went EOL last month and we just moved to one its successors, Fedora CoreOS, also works well. The authors tone is incredibly off putting.
Yeh I don't even know what I'm doing and kept a 3 node cluster running with 100% uptime of small/jokey internal apps for 2 years and everything up-to-date. Replaced one node due to failure and often pull the power from one at random as a party trick.
> HTTP/2 a.k.a. SPDY is a comically bloated Layer 4/5/7 mega-combo protocol designed to replace HTTP. It takes something simple and (most importantly!) comprehensible and debuggable for junior programmers and replaces it with an insanely over-complicated system that requires tens of thousands of lines of code to implement the most minimal version of, but which slightly reduces page load time and server costs once you reach the point of doing millions of requests per second. I am filled WITH RAGE just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web
This resonates with me, and I'd like to add HTTP/2 can only bring advantages if you actually go all the way to push/bundle resources into responses and have a strategy/priority when to push eagerly vs serve lazily; I'm even much more worried about upcoming QUIC (and DoH) because there's no impl in sight.
> HTTP/2 can only bring advantages if you actually go all the way to push/bundle resources into responses
Compared to well-optimized HTTP/1 (e.g. using minified CSS and sprite-sheets), sure. Compared to most HTTP/1 deployments, though: no. HTTP/2 gives you tons of advantages "for free" that you need build-time processes to attain in HTTP/1. With HTTP/2, you can do "the naive thing" that'd you'd have done on the 1995 Web in Notepad, and it'll be the optimal thing.
Also, HTTP/2 means less OS packet-switching overhead server-side if you have ancillary connections (e.g. websockets) open against the host, since those also get muxed into the same carrier socket.
Also, mobile clients wake up less, because there's only one TCP socket to do idle-keepalive on.
HTTP/2 also means that TCP's Nagling has more to work with, and so is less likely to end up needing to waste bandwidth on emitting many undersized packets—it can just pack N short requests into the same TCP jumbo frame, since they're all going to the same place.
I would also point out an indirect "advantage": HTTP/2 makes it cheaper to serve ads proxied through the first-party host (as HTTP/2 flows) than for the client to hit the third-party ad servers directly. People can still block the ads/trackers either way, but served inline to the origin like this, the people who don't block ads, will get a better experience.
it shoves everything down a single TCP connection. This means that if you loose a packet (and 4g is lossy) it stalls the _entire_ queue.
Instead of addressing the main complaint (that multiplexing everything down a single TCP connection is a fundamentally stupid idea) they come up with an entirely _new_ protocol.
It also fails to understand what HTTP2 has turned into: a file transport protocol, with some RPC built in.
So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
To be clear, it's not like there are any HTTP/2-only servers in the world. HTTP/2 is something a client connection upgrades to. One might think of HTTP/2 and HTTP/3 not precisely as versions of HTTP, but rather as variants—HTTP/2 is "optimized binary HTTP, fixed-station profile" and HTTP/3 is "optimized binary HTTP, mobile-station profile."
(Mind you, they are versions relative to each-other, because HTTP/3 is strictly better at doing what HTTP/2 does than HTTP/2 is, so there's no reason to use HTTP/2 if you have HTTP/3.)
But as I was saying: nobody forces mobile devices to use HTTP/2; and by-and-large, they don't. HTTP/1.1 still has its use-cases; neither HTTP/2 nor HTTP/3 was designed to obviate HTTP/1.1's entire set of use-cases.
You know how SSH has sub-protocols, e.g. SFTP? HTTP/2 and HTTP/3 are to HTTP/1.1, as SFTP is to SSH. It's a mode you can put the connection into, that will optimize it for a certain profile of use-cases. No more, no less.
(You know what else is a mode you can put HTTP/1.1 into? Websockets! Again, a complementary use-case.)
> So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
If you think you're so smart, write an RFC (or even just, build a reference server) that's competitive in production performance for its use-cases over similar links.
There comes a point when explicit OSI-style encapsulating layering becomes an active hindrance; and for the use-case of "a server serving a million concurrent requests"—the FAANG-scale problem—they passed that point long ago.
Yes, we mere mortal client users and backend developers might not find such use-cases relevant; but take the perspective of e.g. an internet backbone ops engineer. By volume, most of the traffic going around the Internet goes to these sites. Optimizing just the traffic going to these sites—with a special protocol clients only use when speaking to these sites—makes the whole Internet faster, by reducing in size the huge blob of bandwidth-contention the requests to these few sites create.
Also, a design hint, if you actually want to try to build something competitive to HTTP/3: most of the problem being solved by HTTP/3 is that certain things are known to be optimal, but despite that, nobody can just force their corporate overlords to mandate a switch to those things right away. So HTTP/3 needs to be optimized for the case where you do the most fundamentally-performant things (e.g. using zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access); and also for the cases where you do less fundamentally-performant things (like generating and streaming a JSONL or CSV stream line-by-line as the output rows gets computed by an in-process dataflow engine.) One might call HTTP/3 an arbitrary custom protocol, that happens to have semantics such that it can be downgraded to HTTP/1.1 by a link-to-link proxy. And that is a hard constraint to optimize under.
> HTTP/2 is something a client connection upgrades to.
to which I as a normal person on a normal phone has no control.
> Also, a design hint, if you actually want to try to build something competitive to HTTP/3
I'd need to control a major browser's code base.
Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
> perspective of e.g. an internet backbone ops engineer.
They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
> If you think you're so smart, write an RFC
I already work for a FAANG during the day, I'm not going to put free hours in at night. And I'm certainly not going to get involved in cross company politics.
> a point when explicit OSI-style encapsulating layering
OSI has never been correct, its a fib told to students to make them think there is any kind of order past tcp/udp. I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms. yes its very neat, but
> zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access
from the protocol level, its not actually that much different, unless you are doing some sort of multiplexing down a single TCP pipe.... now, if you trying to balance throughput and latency on the same connection, thats can be really tricky. An answer is not to use the same connection. You do what FASP does which is use UDP for bulk and TCP for control/accounting.
> like generating and streaming a JSONL
you can easily use TCP at one packet a minute, or maxing out a 10 gig link. (I know, I've written a FASP replacement.)
> semantics such that it can be downgraded to HTTP/1.1
you can downgrade 10gig-t to 10m/s that has no bearing on its performance, unless you terminate it badly. Its just a preamble, then for the life of the connection, an utter irrelevance.
> you can downgrade 10gig-t to 10m/s that has no bearing on its performance, unless you terminate it badly. Its just a preamble, then for the life of the connection, an utter irrelevance.
You misinterpreted "downgrade" here (though maybe I chose a bad word for this.) I didn't mean that the proxy itself can force both sides to speak HTTP/1.1 to it. I meant that a proxy can be speaking HTTP/{2,3} to one side, while speaking HTTP/1.1 to the other, by translating ("downgrading") HTTP/{2,3} messages into their equivalent HTTP/1.1 messages. This is the reason that HTTP/{2,3} are HTTP in a real sense: the messages that flow over them still interoperate with the rest of the HTTP ecosystem, through bilingual gateways. The messages are a different bijective encoding of the same HTTP application-layer message-flow semantics.
(Nginx, by default, is such a gateway: it speaks HTTP/2 to clients if they offer, but speaks HTTP/1.1 to upstream backends even if they know how to speak HTTP/2. This is because HTTP/2 is useful to carry flows over the Internet, but has few advantages within a low-contention low-packet-loss LAN.)
This also has other implications, like the fact that browsers and caching proxies don't need to change the way they store cached HTTP resources, just because the resources come in over HTTP/{2,3}. They can just transcribe the messages to disk as if they had been the equivalent HTTP/1.1 messages.
> Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
The point is to take a stupid thing that people will keep on doing no matter what (for better or for worse), and help them to do it as fast (or as un-slow) as possible, without trying to make it non-stupid in the process (because that won't be the same thing any more, and so people won't use your smart thing, but will continue doing the stupid thing.)
You know Formula 1 racing? In F1, there are rules against doing most things that could make cars go fast. The point of the competition is to make the cars go fast anyway, without breaking those rules. If you make the car go as fast as it can, it's not an F1 car any more. To win, you've gotta make the fastest F1 car, not the fastest car.
That's HTTP/{2,3}: the cars are the flows, and the rules are HTTP's must-be-all-things-to-all-people semantics and manifold optimized-for usecases. The goal of adding HTTP extensions/versions/etc. is to make your server (or client) talk to clients (or servers) as efficiently as possible, without breaking any random peer from 1996 speaking to you through a bilingual gateway. If you did break those peers, you might have the fastest protocol, but you wouldn't have the fastest HTTP protocol.
> They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
That's an infinite regress, as the people who you're handing the problem off to are also Internet backbone engineers. If your company owns 30% of the Internet's traffic, your leaf-node AS is part of the backbone, and that AS's problems are the Internet backbone's problems. Just like the major universities were part of the backbone until ~1990.
Though, keep in mind, this isn't about fancy application-layer-aware routing or anything. HTTP/{2,3} have bandwidth savings, and that savings is passed directly onto the Internet backbone those messages travel along, in the form of more headroom for other stuff. It's not about needing to pay peering fees; it's about e.g. packing more clients per head-end fibre switch for an ISP. Or, to put that another way: it's about making more money out of the same aggregate delivered traffic volume, by having that volume represent more customers.
That's why the carriers are interested in participating in the development of these protocols: not because they need to do anything with them in their operational lifetime, but because the effects of the choices made by the HTTPWG have the potential to ripple out and create huge (e.g. 10+%) reductions in their aggregate traffic load (which they can then re-oversell); and they want to do what they can to argue for whatever designs best make this happen.
It took me a bit to understand what you were saying here:
> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.
...but I think I got it now. I was arguing with the wrong thing here.
Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)
But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.
However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.
In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.
But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.
And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)
And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!
And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)
well you do not upgrade your http/1.1 connection. http/2 works by using ALPN on TLS connections, that actually try to get the right protocol.
yeah there is h2c, which does upgrade the connection, but that only works in backend scenarios without tls.
I think if you're making this trade-off and "optimizing just the traffic going to these [big] sites—with a special protocol clients only use when speaking to these sites—makes the whole Internet faster", then I guess you shouldn't be surprised to be criticized by the majority (according to your own words) not having these problems, yet having to shoulder additional complexity for little or no benefit, and even only serving to increase asymetry on the web. Why can't the "big sites" (it's just Google who's behind QUIC anyways, isn't it?) not then create their own network and take their enormous traffic there? The web was created for easy self-publishing after all.
> yet having to shoulder additional complexity for little or no benefit
They don't, though? HTTP/2 and HTTP/3 are voluntary to all parties concerned; whether you're a client, a server, or a gateway, if you don't speak those protocols but choose to just speak HTTP/1.1, then it's the peer that has to cope with that, not you.
(There isn't even any fancy forward-compatibility needed in TLS to support the semantics of the ALPN extension. If you just use an old TLS library that ignores the unknown extension data in the TLS stream, the other side will continue on assuming you didn't understand the question, and therefore aren't an HTTP/{2,3} server.)
HTTP/{2,3} are like a local language of a culture, spoken by immigrants when they run into other immigrants from the same culture. If either party is not an immigrant from that culture, it just doesn't come up.
> Why can't the "big sites" not then create their own network and take their enormous traffic there?
That's called circuit switching (i.e. the thing telco and cable services do that's not the Internet), and it's the thing the packet-switched Internet effectively obsoleted. From an Internet engineer's perspective, if you have two data streams, it's strictly better engineering to just feed them into a switch that linearizes those packets onto a high-bandwidth line "as they come" (and upgrade the bandwidth of the switch+line as needed, so that no signal is ever starved of line-time), than to try to time-divide or frequency-divide the pipe; let alone to keep those packets isolated on two separate networks of pipes. Then you'd need to maintain two networks of pipes! (And the people working at FAANG are still fundamentally Internet engineers who believe in Internet principles, rather than telecom principles.)
But besides that, how would that network be delivered into people's homes? Unless you're proposing that these services take the form of their own additional cable going into your house/SIM in your phone, this network has to merge into the regular Internet somewhere. And it's exactly at that point when that traffic once again contends with the rest of the traffic on the Internet. Even if it's only on the last mile, it's still getting in the way.
> it's just Google who's behind QUIC anyways, isn't it?
SPDY and QUIC are the names of "prototype standards" developed by Google. HTTP/2 and HTTP/3 are standards inspired by SPDY and QUIC, developed by HTTPWG, with Google as just one participant in that conversation.
The other backers of the standard are, of course, the groups whose interests are aligned behind having more-efficient HTTP: carriers, bigcorps, switch/NAT/WAF hardware manufacturers, cellular ISPs, etc.
But I see your deeper point—you're saying that this is all Google's solution to Google's problem, so shouldn't the onus be on Google to solve every downstream problem as well?
Well, it is and it isn't. Google is solving this problem for us right now, but it's not a Google-exclusive problem. TCP was created by DARPA, but maintaining a consistent stream over packet loss/reordering is not a DARPA-specific problem. They just happened to be the first group to need a solution for that problem.
The reason HTTP/2 and HTTP/3 are public standards, rather than things going on secretly only between Google Chrome and Google's backend servers, is that other parties see value in them—not just present value to themselves, but also future value.
New big uses of internet bandwidth arise every day. Netflix started sucking up half the Internet ten years ago, and it's already dropped down to less than 15% because other even larger use-cases have eclipsed it.
HTTP/2 and HTTP/3 are engineered to allow small businesses a path to grow into the next big bandwidth-sucking businesses (i.e. things so many people find useful that the whole Internet becomes about using them) without having to solve a thousand little scaling problems in the process.
Would you rather we live in a world where TCP/IP was a proprietary thing DARPA did; any company that needs TCP/IP semantics, has to re-invent it (probably poorly)? No? Then why would you rather live in a world where any company needing HTTP/{2,3} semantics in the future has to re-invent those (probably poorly)?
I agree these things can be useful, but again none of these come out of the box (if I haven't overlooked or misunderstood something). Including "doing the naive thing"; I mean how do you expect your web server to automatically push CSS or SVG sprites/fonts unless you're relying on the server to intercept/parse your HTML for delivery-time optimizations a la PageSpeed and make heuristic scheduling decisions? Unless you're putting in the effort to optimize your payloads you will just result in as many roundtrips with HTTP/1.1 + keep-alive.
Web browsers are limited in the number of concurrent socket connections they'll open to a given origin. This matters not-at-all in HTTP/2, since everything is going over a single socket; while mattering quite a lot in HTTP/1, since dependent resources being loaded in parallel must be loaded on separate sockets. If you only have six parallel sockets to work with, then if your page is, say, an image gallery, then the Javascript file that makes it work (loaded at the bottom of the body) might be blocked waiting behind the loading of e.g. some large image higher up in the body. The previous requests need to entirely finish (= a round trip) before the next requests on the same socket can start. Keep-alive does nothing to fix that.
HTTP/1.1 pipelining partially mitigates this, allowing the client to "queue up" a list of all the dependent resources it wants from each socket; but it suffers from head-of-line blocking. Which sounds like some arcane thing, but in practice it means that big things might block the loading of small things. (The browser doesn't know how big things are, so it can't effectively schedule them; and the server must dumbly queue results up in the same order the client requested them, because that's the only way the HTTP pipeline's implicit flow sequence counters will match up.)
HTTP/2 is a full mitigation for this problem, since—even without a heuristic "prioritization strategy" for the delivery of dependent-resource flows—the "oblivious" strategy is still a good one: if you attempt to deliver all the resources in the queue concurrently; and you do so by delivering one fixed-size chunk of each flow per iteration, in a round-robin fashion; then you'll end up finishing delivery of resources smallest-to-largest—which means you'll usually deliver the most-critical resources first, no matter where in the dependent-resource queue they started.
Or, in short:
HTTP/1.0 = O(N) required roundtrips for a page with N resources.
HTTP/1.1 with pipelining = O(1) required roundtrips for a page with N resources (followed by O(N) bytes streamed half-duplex), but the page can still take nearly the same amount of time to become interactive as if it were O(N) roundtrips, because of effectively worst-case scheduling.
HTTP/1.2 = O(1) RTTs + O(N) half-duplex bytes for N resources, loading "intelligently" such that the page becomes interactive in O(log N) time.
I really appreciate your going into these details, but I still don't understand the "oblivious" strategy thing which supposedly improves baseline performance OOTB when the criterion is the time to first render of above-the-fold content given otherwise same conditions. You say the effect of delivering all resources concurrently is that "the most-critical resources [are delivered first], no matter where in the dependent-resource queue they started". But this isn't proven; it's just a different heuristic to apply to traffic shaping assuming small resources are needed early vs assuming dependent resources (+ HTML markup itself) are loaded asynchronously in the order the browser parses an HTML DOM (the default behaviour, and the one authors can influence directly). You're not magically increasing the bandwidth by multiplexing so something has to give.
The "oblivious approach" is "just a different heuristic to apply", yes. It's just one that happens to work especially well for HTML rendering, given that large resources tend to be depended upon in a way (e.g. an img or video tag) that gives them a pre-download bounding-box size, allowing the rest of the page to render around them and to not reflow once they're loaded; while the types of resources that tend to be small (like CSS or JS files) are mostly depended on with the semantics that they can potentially reflow the page entirely when they finish loading, which means the browser completely inhibits interactivity (and/or rendering, depending on the browser) until those resources finish loading.
The only things that kind of break this heuristic are:
• large single-file Javascript SPAs. These are usually just marked with the `async` attribute, such that the initial DOM of the page can first render, then be gradually enhanced when the SPA loads. But with HTTP/2 + ES6, you can also just not pack the SPA into a single file, instead relying on ES6 modules, which will each be individually smaller and therefore will end up being delivered first by the content-oblivious round-robin chunk-delivery strategy.
• web fonts, which are large and will necessarily cause a complete reflow of the page once loaded. Currently, browsers make a special loading-precedence exception for web fonts; though it doesn't matter as much right now, as they're still mostly served from third-party CDNs rather than as first-party HTTP/2 flows.
AFAIK, for regular web-page retrieval, these are exactly the total set of things that currently benefit from being server-pushed along with the first response; everything else just gets handled well even without server-push.
(If you're curious, server-push is really designed for the use-case of pushing secondary API responses along with an initial API response; to allow for GraphQL-like "single-round-trip resource-hierarchy/-graph walking" in a way that's more friendly for caching layers than comingling the results into a single resource-representation. It makes the most sense in a Firebase-like system, where un-asked-for server-pushed resources can be obliviously accepted and dumped into the client-side in-memory set of synced entities asynchronously to the parsing of the initial response; and then, once the dependency is parsed out on the client side, the client can discover that it already has the entity it wants in its in-memory entity store, and doesn't even need to make the request.)
Apologies if I've missed some part of the explanation that answers this. The way your previous reply described it, the benefit of HTTP/2 is loading smaller things earlier. But given that goal, couldn't the browser just prioritize JS/CSS files over images/videos for the same effect, without any new protocol that sets in stone some heuristic purely based on content size?
> the benefit of HTTP/2 is loading smaller things earlier
No, the benefit of HTTP/2 is a lack of head-of-line blocking. Head-of-line blocking can be easily seen when big things block small things, but that's not what it is. What it is, is when something doesn't make progress because another thing is being waited for.
Imagine a multimedia container file-format where you can't interleave audio frames with video frames, but rather need to put the whole audio track first, or the whole video track first. This format would be unsuited to streaming, because downloading the first chunk of the file would only get you some of one track, rather than useful (if smaller) amounts of all the tracks required for playback. Note that this is true no matter which way you order the tracks within the file—whether the audio (smaller) or video (larger) track comes first, it's still blocking the progress of the other track.
HTTP/2 is like a streaming multimedia container format: it interleaves the progress of the things it loads, allowing them to be loaded concurrently.
This doesn't just mean that small things requested later can be prioritized over large things requested early (though it does mean that.) It also means that, for example, if you load N small Javascript files that each require a compute-intensive step to parse+load (GPU compute shaders, say), then you won't have to wait for the compute-heavy load process of the previous files to complete, before you begin downloading the next ones; but rather you can concurrently download, parse, and load all such script files at once. Insofar as they don't express interdependencies, this will be a highly-parallelizable process, much like serving independent HTTP requests is a highly-parallelizable process for a web server.
One benefit of HTTP/2's lack of head-of-line blocking, that would be more talked-about if we had never developed websockets, is that with HTTP/2, you get a benefit very much like websockets, just using regular HTTP primitives. You can request a Server-Sent Events (SSE) stream as one flow muxed into your HTTP/2 connection, and receive timely updates on it, no matter what else is being muxed into the connection at the same time. Together with the ability to make normal API requests as other flows over the same connection, this does everything most people want websockets for. So the use-case where websockets are the best solution shrinks dramatically (down to when you need a time-linearized, stateful, connection-oriented protocol over HTTP.)
> new protocol that sets in stone some heuristic
Note that there's actually no explicit specification of the order in which HTTP/{2,3} flows should be delivered. What I'm calling "content-oblivious round-robin chunk scheduling" is just the simplest-to-implement strategy that could possibly meet HTTP/2's non-head-of-line-blocking semantics (and so likely the strategy used by many web servers, save for the ones that have been highly-optimized at this layer.) But both clients and servers are free to schedule the chunks of HTTP flows onto the socket however they like. (They can even impose a flow concurrency cap, simulating browsers' HTTP/1.1 connection limit and starving flows of progress. It'd make the client/server a non-conformant HTTP/{2,3} server, but it'd still work, as what progress "should" be being made is unknowable to the peer.)
It's a bit like saying an OS or VM has a "soft real-time guarantee" for processes. Exactly how does the OS scheduler choose what process will run next on each core? Doesn't really matter. It only matters that processes don't break their "SLA" in terms of how long they go without being scheduled.
The advantages listed above all come from the fact that HTTP/2 runs all traffic over a single multiplexed TCP connection, rather than HTTP/1.1, where clients are forced to open many independent TCP connections (because each is a single channel where responses must be returned sequentially, in the order they were requested, and so block one another).
That happens totally automatically, there's nothing special you need to do to enable that.
There are also other potential advantages once you look at push etc, but using TCP as it was meant to be used really does give you many advantages for free.
That's not true at all. HTTP/2 is hugely impactful without using push at all—multiplexing beyond 4/8 streams enables radically different, performant bundling strategies, all "for free" with zero configuration, just by the act of using the protocol.
The misconception is the idea that anyone can easily implement a compliant HTTP 1.1 server, or even client. The protocol may appear simple on paper but parsing it correctly and securely is quite tricky. IMO, one of the benefits of HTTP 2 and 3 is they force people to use libraries and remove the illusion that interacting with HTTP bytes on the wire is a reasonable thing to do.
The value of being able to build an HTTP 1.1 server is not in putting a hand-crafted server on the internet and serving traffic. It's in the experience of building the server and later applying that detailed byte-by-byte knowledge in real work. It's also in teaching young engineers that yes, they can learn something by digging into the layers that surround their own code. Part of the dysfunction of "enterprise" programming is the programmed helplessness of people who are surrounded on all sides by complex, opaque technology that turns curiosity into a liability. We are replacing a "whoa, I can see how it works!" experience with a "fuck, I guess have to leave this part to those bigcorp engineers" experience.
(I'm not agreeing with the author, just pointing out why he might care about the ability for an individual to implement the protocols the web is built on.)
I agree it's incredibly important to dig into layers under (and over) the one you're working at! I'm very grateful for the experience of building a simple unix-like kernel and a simplified TCP & IP implementation as school projects. And I'd encourage anyone to do the same with HTTP 1 or 2 or 3. The difference is that no one thinks that their educational/toy TCP implementation is suitable for production use, while people do seem to (mistakenly) think that about HTTP.
> IMO, one of the benefits of HTTP 2 and 3 is they force people to use libraries
What a strange comment. "The benefit of this protocol is that it's so complex that nobody understands it, so they have to use an existing implementation."
What a weird slippery slope argument. There is a world of difference between "so complex that nobody understands it" and "I can telnet to port 80 and fetch a webpage using keyboard and fingers"
That's pretty much in the "cut out future generations from system programming for the web" territory, and complexity for the sake of complexity. Also, you have to implement HTTP/2 in addition to HTTP/1.1.
It's a big jump to go from "this protocol is more complicated than the previous one" to "cut out future generations from system programming". It's not _that much_ more complicated that a competent programmer who's familiar with sockets and bytes can't ever accomplish it.
As discussed at length in another sub-thread here, the complexity solves real problems, it's not for its own sake. But if you're not convinced of that, nothing I say here is going to change that.
·
If you've worked with me, you'd know that I take simplicity as a virtue to an extent that's almost detrimental to getting stuff done. I despise complexity for its own sake. I wish as much as anyone that we could start over and redesign IP, TCP, TLS, HTTP, HTML, JS, and a few more things, with all our accumulated knowledge, into a simple and clean application and content distribution system that's easy to understand and observe. But path dependence :( We have to work with what we've got.
It's the picking out of HTTP 1.1 as some sort of golden age of clean and simple and observable that bothers me. It isn't.
What I don't get with the complaint is that HTTP/1 still works fine. If you are hosting (or writing) your own server, you can stick with that and be fine. I have heard ZERO talk about browsers deprecating HTTP/1 support.
For the companies implementing http/2, those improvements DO matter. They have different needs.
>but will completely cut out future generations from system programming for the web
There's so many things where my competence is based on understanding the simple preceding system plus a good intuition of how the class of systems tend to be bungled up over the years. We don't pay down the accumulated complexity, and newcomers don't get the same luxury I do.
It really becomes noticeable with highschoolers and I have to chose between teaching useless but illustrative groundwork, or just explaining and expecting a combination of taking it on faith and memorization.
Early employee of CoreOS here. I wanted to float a few ideas out there for discussion.
First, while etcd did get a bit more complicated over the years, it stayed relatively stable compared to Consul. This was a point of pride of the etcd team and why it ultimately was picked for Kubernetes. Yes, a gRPC API is not as easy to use as HTTP, but this gained massive scale for Kubernetes when it was released. Something like a 10x increase in the number of nodes in a cluster, without changing any Kubernetes features. Just enabling etcd v3's gRPC API. That's pretty awesome.
As a member of the product team, I think we could have made different choices that would have massively impacted the ecosystem. Consul might not exist today. We did not want to add in a DNS server to etcd in order to keep it lean and focused. We had an integration with SkyDNS, but it wasn't on by default. This birthed Consul and its built in DNS server for service discovery. Consul is very popular so clearly this was the right move for some consumers. Kubernetes didn't need this, nor did Container Linux for its update coordination. For us, having this possible on top of etcd was the right trade off. Consul has added a complete service mesh into the later version. Again, etcd has stayed lean and focused. I think you are focusing on pebbles when you take in the overall ecosystem.
Would you rather etcd have turned into Consul's breadth? Would it have been forked?
The second point is around the success of Container Linux. I only include this because I am proud of it. We updated and secured a fleet of machines that measured larger than some cloud providers. That is a huge impact on the container and security ecosystems. Flatcar Linux and Fedora CoreOS remain as successors, along with other projects like Google'c container optimized OS and I think AWS announced one as well.
This work continues at Red Hat on OpenShift, etcd, Fedora/RHEL CoreOS, and tying Kubernetes more closely with the OS to keep the combined whole secure, up to date and successful for engineering teams shipping software.
> In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers). I would go so far as to say that Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd.
This, I think, is the key to understanding this whole rant. It is entirely of a piece with the anti-systemd crowd, and I think understanding what was really going on there helps explain this rant, and a lot of other agita in the community these days.
Like, if you followed the drama on the Debian mailing list, it could sometimes feel very surprising that they were willing to go to war over the least sexy thing ever, an init system. Part of the reply is that systemd is more than that, it's a bunch of low-level plumbing all lashed together. But, who cares about low-level plumbing that much?
The answer (other than 'the exact sort of people who run Debian') is this. Linux started off as being written by hobbyists, for each other. It was very often a labor of love. Even the people making money off it weren't doing so in Big Business sort of ways. Linux and the associated stack around it is being taken over by a lot of big corporate interests: Intel, Red Hat, Google, even Microsoft. Are they bad? That's a matter of opinion. Are they inept technically? Probably not, but a lot of it revolves around how you measure things.
But what is absolutely happening is that people who are paid to maintain these projects on the behalf of large corporate interests are more plentiful, both in headcount and in personhours, than people who are maintaining these things for each other. For people who were used to Linux as this anti-corporate space, that's a huge and distressing change. And they can feel powerless against it, because it's really hard to reach a critical mass of people who feel the same way as you, and are willing to do all the things to not just support a fork but to keep the rest of the ecosystem they live in open to the fork.
It's not about any one piece of software, and it's not _really_ about pure technical merit. It's about the culture and the system of values and the way that decisions are made and it's about who's in control. The author's complaint isn't so much that the Google style is bad (although I'm sure he believes that very strongly), but that it's harder and harder to find somewhere to escape its reach.
I too am sick of people telling me to accept ubiquitous containerisation (snapd), locking down of filesystems (macOS), overly complex init systems (systemd (1)), lack of server access (manage it via ansible/k8s, not ssh). Usually the arguments given are "it's necessary 'for security'", "this is where everything is going", "you have to trust the vendor". This is not the spirit of Linux or open source. I totally see the effect you mention of paid maintainers forcing projects into their employer's direction. As such I found it sad when even Arch fell to the systemd camp. And it's one of the few things it really enforces, it supports 6 different network initialisation methods for example, where all other distributions just pick one and go with it.
(1) I'm not all anti-systemd by the way. I agree init was bad. But I think other systems like OpenRC are way easier to work with.
> As such I found it sad when even Arch fell to the systemd camp. And it's one of the few things it really enforces, it supports 6 different network initialisation methods for example, where all other distributions just pick one and go with it.
Arch has only supported one init system at all times. We could still have stuck with the sysvinit with hacks to look like openrc, but nobody liked that. The initscripts where a mess to maintain and systemd enabled us to not having to care about these details.
If we stuck with the old system, what would the argument be today? "Arch only supports sysvinit! That is not the UNIX philosophy!"? You don't see any complaints on distributions that only support one init which is either runit, s6 or openrc. But when it's systemd somehow people get sad?
> You don't see any complaints on distributions that only support one init which is either runit, s6 or openrc. But when it's systemd somehow people get sad?
Runit will just run 3 scripts then supervise your service directory. _That's it_ and no other choices are made for you. You can compose runit with many other different types of systems according to your own tastes.
Systemd, ostensibly also just an init system, ties you to so many other non-init choices that I can understand the feeling of unease people have towards it.
I agree sysvinit was no longer sufficient for today's needs.
The issue I have with systemd was mainly related to its heaviness not really fitting in with Arch's simplicity. And with the products of corporatism seeping through even one the most noncommercial variants of Linux. Basically what cwyers mentioned above.
I've been putting up with systemd as all the distributions I use have moved to it, but I never really liked it.. I don't like the way the scripts are so far hidden in the filesystem and I don't like journalctl, I prefer having simple logfiles.
I discovered Alpine linux recently as a server distro (previously I thought it was mainly developed for docker containers) and I really took a liking to OpenRC. It reminded me that things could still be simple even in a modern init system. I think it would have been a good choice for Arch, for me it would have been better than systemd. I don't even know the other systems you mention, I'm not an expert on init systems.. My opinion is just that of a user.
>The issue I have with systemd was mainly related to its heaviness not really fitting in with Arch's simplicity.
Arch is a pragmatic distribution first and foremost. If we can build systemd, and ship it as-is to have a complete init, and more features along with it, that is much more enticing then the alternative.
> I think it (openrc) would have been a good choice for Arch, for me it would have been better than systemd.
It would still entail maintaining some form of initscripts. I don't think most users realize how much of a burden this is, they don't really deal with that part of the system.
I understand, and it was not my intention to drag the systemd discussion up again. It was just one of the many things I thought of after reading cwyers post which really resonated with me, and the parent article as well.
And there are things I do like about systemd, I also have to say. Especially the way you can pass a variable to a server with @, like run 2 instances on different ports.
I know very well that the rc scripts were a pain to maintain as I've written some as well in the past. So I really understand the benefit of upstream-provided service configs.
The transition to systemd for me as a user was just a bit more difficult. It deviates so much from earlier conventions, and adapting it I found difficult. It enforces a lot more than just the init system.
For some reason I always end up having to compile some of the software I use myself, and having to make scripts then. I found this a lot easier with OpenRC than with Systemd (having pretty much no knowledge of either).
But anyway I'm just one user and you support millions, I know you're making the right choices for the platform. Thanks for taking the time to reply to me in fact!
I don’t think it’s even that. Most software developments in the last 10 years are vanity projects, fashion shows or death marches built to sell support, IPOs and conference tickets.
We forgot what we were doing somewhere. Oh yes, solving problems.
It's because majority of us have no real problems to solve, so we do what everyone else is doing in the corporate world and make ourselves useful that we can keep collecting paychecks. Best way to do that is by over-engineering solutions to problems that are already solved.
Or we can replace perfectly functional systems simply because replacing legacy systems with modern solutions has some intrinsic value on its own, even if the modern system brings no additional value.
There are bunch of real problems that could be solved with software, but there's no money in it so why bother? It's better to roleplay technical expert and collect consultation rewards doing nothing.
How about a new kind of free software, individual and small-organization supported free software?
Policy proposal: Accept contributions only from people who have not done paid work for a large organization in the last 24 months. Large organizations are entities with 50 or more paid engineers. Contributions include bug reports, feature requests, pull requests, and donations of money or equipment. Anyone may contribute documentation and training materials.
The counterpoint is that there's a lot of people and organizations out there that are happy to take advantage of the labor of these large organizations, and view loss of control over some of these things as a valid price to pay for all of the contributions in code that they get. So the people left to contribute to this are a small portion of the overall base, and quite frankly often don't agree on much except their dislike for BigCo decisions. (systemd was able to consolidate its position as the top Linux init system by adoption; the contention for the number-two spot is split between sysvinit holdouts, OpenRC, runit, S6, GNU Shepherd, and probably others I'm forgetting.)
So you end up in a situation where package and distro maintainers face some work to support systemd, and because everybody else is supporting it it lessens the burden. There's this sort of gravity to these big projects that's hard to escape, and frankly a lot of people don't feel the need to; they view it as something they can get in a stable orbit around, not as something they're being sucked into.
I run k8s in production and I think this is a really bizarre axe to grind that sort of smells like someone who got upset by how steep the kubernetes learning curve is. Which, in a way, is understandable.
> 1) Add hundreds of new failure modes to your software
In my entirely anecdotal experience, it removes error modes. It turns out that just because k8s offers a feature (it offers many!) doesn't mean you're required to use it.
> 2) Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
Portability, put another way, is a requirement to maintain n different deployment mechanisms because you have no meaningful control over the system on which your software is deployed. This is unavoidable depending on what it is you're deploying and where, but for those of us who have the ability to make those decisions the old notion of portability described by automake and others is actually a bad contract to agree to.
That said, the YAML but is horrible and should be abolished. This is a great criticism.
> 3) Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking
The author isn't considering that I am not, and never will be, Google's entire SRE team. I consider it a general win to be able to apply some subset of their encoded knowledge to the problem of keeping my services running with minimal downtime even if you pick up some k8s-specific baggage along the way.
It's always worth noting that k8s isn't designed to give outsiders the power of Borg the way Google SREs have the power of Borg.
it's designed to give outsiders something better, based on learned experience of where the toil and cocked-foot-guns are working with Borg. Borg is, by modern standards, extremely legacy infrastructure and even Google is trying to replace it with something better internally.
the problem is that k8s is a half arsed implementation of borg/tupperware.
Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.)
The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale.
Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.
> The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.)
Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure.
Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.
Of all things I would definitely not call it easy to read and write. It has super-weak (see this for examples: https://github.com/cblp/yaml-sucks) dynamic typing, which coupled with requiring awful templating to get anything non-trivial done it has been the cause of many outages.
For real world Kubernetes needs to be addressed I'd argue we need Dhall or something with Dhall-like semantics and goals to have first class support.
YAML actually has one of the worst default behaviors I've seen: automatic conversion of certain strings to bools.
This wouldn't be a problem if YAML required that all strings be quoted...but it doesn't, and there's no "strict" mode to force quotes, so users are extremely likely to end up doing something that behaves unexpectedly.
If you're lucky, you'll spot the problem quickly instead of wasting half a day on it.
YAML can be validated using JSON Schema/OpenAPI, and somebody does maintain Kubernetes schema definitions, including Kustomize, which can then be used in an IDE for live validation. For example, the Kubernetes extension for VSCode does this.
If you use Helm, writing and validating schemas for Helm value files is trickier, and not something I believe anyone is doing. It should be easier to do. Ideally you should simply put this at the top of a value file:
$schema: ./schema.yaml
But from what I know, there's no editor/extension, that supports validating through such a declaration. Secondly, Helm doesn't have a way to statically analyze value access. Something like {{ .Values.missTypedKey }} will only be caught when you generate the manifests.
You know i actually agree with you but, as a manager of other engineers, people never stop complaining about having to edit large amounts of YAML that describe complex objects. At a certain point im forced to acknowledge that its just not ergonomic. Now, I will say that perhaps whats actually happening is the objects that the describe are so complex and numerous that the problem is actually that you need better tools, not that YAML as a format is bad. YAML as used to configure, say Open API specifications, is something people always make my problem. I don't really know why this is because I don't have an issue hacking this stuff up in VIM but people really just don't seem to like it.
The features intended to make it easy to use, in the end lead to surprising behavior, often with values having non-string types where you expected strings. There are some examples in https://www.arp242.net/yaml-config.html.
Nothing at all. Until you start 'templateizing' it (looking at you, Helm).
It's definitely better than JSON (it can even have comments, imagine that). Most people don't bother reading the spec or even examples though, and don't realize how good it actually is.
In the Kubernetes and Ansible/Salt world it's frequently templatized, and then you need to worry about indentation that you place or your template engine places. Every day I pray that Helm will incorporate support for jsonnet so I never need to touch YAML again.
I made a script for expressing similar deployments (ie homepage-en, homepage-de) in a variety of contexts (test, prod) and their permutations in a single, succinct json file.
I agree with, and have felt the same pain as you here. But this isn't YAML's fault, IMO, it is that the templaters, like Helm, aren't safe, and don't escape their outputs.
If have a string, and I use that in the middle of a YAML template, I expect a string — I don't expect "300" to become (integer) 300; give me a function for that conversion.
Any HTML templating system that behaved this way would be frowned upon due to the potential for XSS. There's little risk of XSS here, but it's still annoying to get the wrong results.
Why not parse the YAML- manipulate it like you would any other data structure in your language of choice, and then generate the YAML from there?
I don't understand why you would want to directly edit markup language without parsing it. YAML's goal is to represent a data structure so that a human can interact with it. It accomplishes that goal well. If you want to edit that data structure programmatically- no markup language makes sense to use, represent the data structure using your programming language's native representations.
Your position here seems to be that YAML is a reasonable configuration language. It kinda is.
But that's not how it's used in Kubernetes. It's used as a declarative DSL that you use to specify your deployments, and it is HORRIFIC. Extremely nested things where the equivalent actual code to do that would be shorter. Things that are inexpressible. Backwards-incompatible changes to your YAML that actually make it impossible to deploy changes that bring your YAML up to spec if your cluster's master node upgraded first.
YAML gives nothing here. jsonnet really isn't a drastic improvement, IMO, except that it lets you express some repetitive parts without having to figure out how many times a template engine is going to indent your YAML.
YAML is easy to read and write in tutorial documents. It becomes a complete mess in real life and whitespace based syntax is horrendous to debug. We have had endless heartaches due to minor yaml issues and today employ tooling to minimize the amount of yaml engineers write to avoid these issues.
Mostly that it's a single projection of an underlying data model. Where the projection is a snug fit, great. But typically folks want sets of things and suddenly it's a bit chatty.
It's mostly a rant about someone not accepting that extra performance can come at the cost of complexity.
While I could argue with all the points that he's making, my main counterpoint is this: junior devs don't care that their http/2 server uses way more "complex" code then their http/1 server, it's just a flag away (or in most cases, automatic). Senior devs worth their salt also don't care, if I design an application that is going to run on kubernetes I now know it will run in the big cloud providers and on premise without major changes. It forces you to accept that your app will die and needs to be able to run from a cold start without any issues. I can't count the number of machines I've encountered over the years that couldn't be rebooted because the maintainers monkeypatched the crap out of it while running EoL distributions, libraries and web servers.
And now I've realized I've become a geek yelling at the cloud as well.
The performance benefits are aimed squarely at the large, entrenched interests who can absorb the increased costs as a rounding error. HTTP/2 and SPDY/QUIC look, to me, to be about decreasing costs and increasing efficiency for large web hosts and erecting barriers-to-entry for competitors.
HTTP/2 is more complex to implement from scratch (but still quite doable, even as an individual), but it has built-in support in every language & framework worth its salt now, so you don't need to do that.
If you're using an existing implementation, that's usually just as easy to do as HTTP/1.1, because they have almost exactly the same semantics (that's an explicit goal from the spec), and so most implementations have almost exactly the same API.
In practice, it's a syntax & connection management change on the wire that's mostly invisible as a developer building on top of it, plus a set of optional extra features (like Server Push) that you can use if you want or ignore if you don't.
Can't speak for QUIC/HTTP3, since I haven't touched them yet, but I'd be a surprised if that's a hugely different story.
I mean, who are these newcomers we're talking about? There's basically never a reason to roll your own web server. I guess if you want to compete with Nginx, sure, the barrier of entry is a higher now? But that's an incredibly niche case.
if you wanted simple, there was redis or memcached.
it's an arbitrary cutoff to call etcd simple (it implements Raft, for fuck's sake, it implies a distributed system, multiple odd number of nodes to avoid split-brain!)
systemd is simple, fixed config format, comes with every distro, default works, has extensive documentation, compared to undocumented distro-specific init scripts written in sh (or bash, or worse).
sure, if someone just wants to hack on their 8bit toaster, then they might find a simple shell script simpler.
Heroku also forces you to accept your app will die and needs to run from a cold start. Surely there's more to this k8s thing than "it's cheaper than heroku" ?
> the worst thing to happen to system administration since systemd
Cool, I need to look into Kubernetes way more then! There hasn't been any single development in the Linux world that has made my job as a sysadmin easier than systemd. Services, Timers and Networking that just works, is concisely defined and can be used across all relevant distros? Hell yeah, I don't ever want to go back into init.d/ifupdown/crontab world filled with bad configuration files, strange footguns and bugs that won't ever get fixed because someone's legacy system might depend on them.
But it's not about kubernetes. It's about simple API's becoming complicated for no reason (according to the author, I have no experience on etcd) using etcd as an example
The author doesn't detail _what_ is more complicated about the gRPC API, other than the fact that it's gRPC. One could implement the exact same API in HTTP or with gRPC; so without specific examples, it's kind of a meaningless critique.
Surely, most consumers of a database like etcd are using a client library, in which case why does it matter if the API is HTTP or gRPC?
Tangential: after reading some of the comments, I was surprised that the blog post was only like 250 words; the author really says very little.
why does everything with a negative valence attract this criticism? surely there are plenty of times in one's life where he or she says negative things about something (a coworker, a boss, a partner, an acquaintance) without any intent to foster a "productive discussion". as though that is something owed to products / services / people / experiences / whatever that we hate for whatever reason. as though those things would somehow magically improve with any amount of "productive discussion". how about not everyone take it personally when someone dislikes something purely because of the experience that they have had with it? given the comments section, the only way for this "discussion" (some call it a post) to have engendered this tone of discussion while still being on the front page is either (1) everyone disagrees with the post, but hate-upvotes it anyway or (2) the points being made strike a chord with people. and regardless of which it is, this whatever the opposite of saber rattling is that causes this kind of reaction clearly misses the point.
Curious why the author declares Container Linux bullshit/unsuccessful? I thought it was a wonderful project that got better over its (brief) lifetime. The active/passive upgrade was an absolute blessing. If the declaration is it was unsuccessful because it no longer exists, Flatcar and Fedora CoreOS are pretty straightforward successors. fcos is basically functionally equivalent if you don't use rpm-ostree layers and if you use podman in place of any rkt containers. If the complaint is "systemd," that's fine—it's not the OS for you—but I don't think that makes it bullshit.
I didn't like having to move cl machines to fcos but it really wasn't that bad and I still get coordinated, active/passive upgrades. ¯\_(ツ)_/¯
There's a lot of inflammatory buildup to the chief complaint:
> With the massive influx of Kubernetes users came, of course, a large number of Xooglers who decided to infect etcd with Google technologies, as is their way23. Etcd's simple HTTP API was replaced by a "gRPC"4 version; the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys. etcd 3.2 added back a tiny subset of the HTTP API through the "gRPC Gateway", but not enough to implement any of the rich applications built on top of the original API. The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely.
This is the interesting part of the post to me. I can understand the disappointment - but also, nobody's making the author upgrade to the gRPC-based etcd v3. This reminds me of other "traditional" sys-admin folks I've worked with who seem to want to stick to 2010-style administration rather than work with newly available abstraction layers that eliminate some of the overhead (e.g. Docker, k8s). If they want to continue linux administration in that way, that's their choice, and there's some merit in that - but they shouldn't expect their tools and other developers to necessarily follow them.
> nobody's making the author upgrade to the gRPC-based etcd v3.
They're stating that the v2 API keeps being slated for termination. Surely once that happens they won't be able to use it anymore, and will have to either use an unmaintained etcd, fork it entirely, or be forced to use the v3 API?
That actually raises the oldest question in open-source engineering:
If the HTTP API has value, why not fork etcd and maintain a version where the HTTP API is the primary interface and gRPC is an afterthought or missing?
>The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 [appendex a] is just a list of the 61 most popular headers from Google properties.
I thought this was moderately-funny cheeky banter, but I wanted to see what implementation decision they were making fun of with this silly misrepresentain. Quote RFC 7541: "The static table was created from the most frequent header fields used by popular web sites." Oh.
Why do you think that is a bad header compression scheme for a static protocol where the majority of traffic contains those headers/values repeated over and over? That indexed table shaves about 30% off header size.
> This just reads as a systems engineer who is angry about new tooling/processes replacing parts of their job.
but it doesn't its needs a bunch of work to upgrade/migrate replace. which is what they are complaining about. having to replace perfectly good stuff, with _new_ stuff.
Unless you are bringing teams together onto one platform, k8s is never going to save you cash/engineer hours.
Lamenting about complexity just for the sake of lamenting about complexity and then adding a few potshots at Google/FB/etc. doesn't exactly make for a thought-provoking, nuanced, or interesting argument, though
I think he/she made it clear it is needlessly replacing parts of the job.
The question is, should one feel any one way about that. I’d imagine it would be the most annoying thing on earth, so it’s a valid emotion being expressed.
> I think he/she made it clear it is needlessly replacing parts of the job
Maybe? There weren't any actual reasons for why the author so vehemently disliked all of the technologies mentioned, and there was certainly no acknowledgement of the potential benefits of them.
"This is systems. You are to be trying to be clever. Please stop." is a damn good rule of thumb and I think lots of people tend to go all-in on k8s without considering whether their use of the features will actually justify the level of additional clever so introduced.
To me, that being the 'actual reasons' was implicit from the rant - I don't disagree that it was a rant rather than an argument.
The author is arguing that simple, easy to use, and easy for hobbyists and upstarts to learn tools are being replaced by complex systems designed by and for the 1% of companies who will actually need the complexity.
They are arguing that "House Builders Inc" has invented an automated robotic nail gun and is encouraging standards industries to recommend the only things built with such a system are compliant.
Agreed. The writing style is so full of bile that any actual point just drowns in it and makes the author seem petty and bitter about the world. Not a good read, even for those that agree with the author about unneeded complexity.
How tedious to have to constantly refute FUD when it's easy to find the answers with Google.
Anyone involved in Kubernetes, near CoreOS at the time, or really anywhere in the space at the time (instead of looking back at it with anger), knows this all to be false. CoreOS was setting direction for etcd, and understandably adding features for one of its bigger users (and in fact, some of those features are used by things of larger scale than k8s).
Kubernetes itself was started by Googlers, many of whom are still there or left to go... do Kubernetes at Red Hat (IBM) or as a startup, or at Microsoft. But to act like it was an outside project started by people who had previously quit, or are somehow unqualified to work on an orchestrator, is just an an angry untruth. Every major committer to Kubernetes besides a handful of RH folks were at Google when Kubernetes 1.0 came out. I'm happy to be corrected but I know it's hip af to hate k8s (just like two days ago https://news.ycombinator.com/item?id=23807556)
Thanks. I just wanted to know what the author meant; it wasn't obvious that I needed to Google the quote to figure out the meaning. I certainly hope that I wasn't adding any FUD to the discussion.
How about that Kubernetes was released in 2014, not 2015? It's the first thing that shows up when you Google "Kubernetes" - Initial release: 7 June 2014; 6 years ago.
Evidently, OP couldn't even be bothered to gather basic facts about the tech he trashes.
Thank you. I'm sorry that you feel like I was trashing the tech. I really didn't mean to come off that way by asking a question about a statement that the author made.
According to the others on this thread it's related to the date when Kubernetes was released. It wasn't clear to me, but thank you, I think you raise another good point.
I'm mostly a happy systemd user. Unit files are much easier to work with and more consistent than init scripts, the faster bootup (at least on my arch and Ubuntu systems, maybe you could make Gentoo even more minimal with initscripts), and the container/isolation features won me over.
That said, I can understand the objection to software that would previously support BSD or macOS now only providing systemd units or depending on other parts of the systemd project that in turn depend on the init making at a pain for portability across unices. Or software like gnome taking up huge amounts of time for the BSDs to support.
Unit files are great. SystemD should have stayed there. But then it handles networking(including configuring network interfaces), and time synchronization, and name resolution, and the kitchen sink.
From an administrator's perspective it's the same problem with every declarative run control system: there is a layer of indirection between what's written in the configuration files and what the machine does (as opposed to an imperative system, where it's a bunch of shell commands that are literally the actions the machine takes).
With systemd in particular the indirection is notably (needlessly) opaque, with a big spaghetti mess of unit files symlinked in 18 different places, some of which have semantic meaning to the init system and some of which do not.
Branding perhaps. If 50 tools had been released, called "newInit", "newDns", "newNTP" etc, with each tool succeeding or failing (upstart) on it's own merits, that would be one thing.
There is a perception that systemd has taken over the entire system as a cancer - replacing things that have worked for decades just because it's cool.
One annoyance I get is this: I've just taken an Ubuntu 1604 server running apache and b0rked the config file.
$ sudo systemctl restart apache2.service
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.
$
That's crazy. "Something broke, now go look here or there for why". Compare under an old ubuntu 1404 machine I have lying around
$ sudo service apache2 reload
* Reloading web server apache2
* The apache2 configtest failed. Not doing anything.
Output of config test was:
AH00526: Syntax error on line 1 of /etc/apache2/sites-enabled/sd.conf:
Invalid command 'blah', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
That is so much more helpful.
You run the first and it outputs the last few lines of log. Why not output that when I call restart in the first place?
Another one is taking over DNS from resolv.conf. Muscle memory from years of "cat /etc/resolv.conf" (with a tab somewhere in the middle) has been replaced with the ungainly "systemd-resolve --status".
Now all of this is great if you're running servers as cattle - if you have thousands of machines doing web-scale stuff you don't want to see what's happening on a single machine, your layers of automation abstract it all away, your system can cope with machines going down, you probably don't even have an ssh account.
If you are looking after individual machines though, systemd came in and turned everything upside down to presumably make things easier for cattle owners.
There is https://blog.darknedgy.net/technology/2020/05/02/0/ which puts systemd in its historical context, explains some of the criticism, and then goes into the technical details. It's a long post and the tone makes it difficult to continue reading at times, but I learnt a lot from reading it.
I for one think the team behind it and especially Poettering have incredibly bad taste in software, which bleeds into their designs, and possibly the worst attitudes and culture of any major OS initiative, making Linus look tame (hell, at least if he got bent out of shape he was usually right, and not just throwing a tantrum because he could get away with it). I don’t necessarily disagree with it’s goals, even, I just think the implementation’s not great and the team are, largely, self-righteous jerkasses. See links others have posted here, their attitude and demeanor’s pretty out-there.
I have used systemd successfully in production and have a positive opinion of it. I like that it is integrated, documented, and self consistent.
None of the negative ideological complaints against it resonate with me, as none of them seem to impact me (yet, anyways). My experience with past init systems is that they are cryptic and error prone.
Ditto. my personal experience with systemd has been "wow, it's easier than ever to write an init script that runs when it's needed".
Looking at the components that make up a modern UNIX system, it's definitely time to think "how can we make a new operating system, evolved from UNIX, but with a more coherent core running the majority of run time orchestration".
Similar opinion here. I put it off for when I could learn it better, and so far it's been fine... sometimes nice too when it has a feature I want that's already built-in (depending on version level and all that).
You may not realize you're inciting a flame war. If you google for this there's no shortage of arguments going back 10 years and continuing to this day
"You can't discuss this topic because of a flame war" is ceding territory to the assholes. I'm willing to accept that when it comes to talking about political topics here, because this isn't a political forum, but I refuse to accept that we can't talk about technical topics here because some people refuse to keep a civil tongue.
One practical problem I've had is with journald. The log files are useless outside of the system they were generated on. They could have just used a simple text file and avoided major headaches.
Almost like with etcd, it started simple, but got expanded into a huge incomprehensible monster by the original developers. Politically and business-wise, systemd is a clear win for Lennaert and his clique, as well as the whole Redhat/IBM.
But I am sure that future historians will view the impact of systemd on computing as largely negative.
The systemd/RH relationship is complicated. Red Hat doesn't use most of systemd's features--they don't even ship systemd-networkd on RHEL8, preferring NetworkManager instead. On the flipside, I see more use of systemd's features on Arch or Debian.
I don't think you can frame systemd as some kind of RH trojan horse when so little of it makes it into RHEL/Fedora.
So while Arch ships everything, and being Arch it doesn't really have a default, its recommended networking system isn't systemd-networkd but rather netctl, which is basically everything systemd-networkd should have been.
`netctl` was explicitly written because systemd didn't have a network daemon. The past 3-4 years the community has in general recommended against using `netctl`.
It was never removed from the ISO because the releng maintainer didn't put that much thought into it, but I'm happy to tell you `netctl` was replaced with `iwd` on this month ISO release.
> But I am sure that future historians will view the impact of systemd on computing as largely negative.
I don't really think so. They probably will point out a lot problems with at some point somehow got fixed but it should be net-neutral or positive.
The thing is todays linux is running manny services and to do so nicely you want to have some form of service manager which does startup/shutdown/restart/circicute braking and helps with interconnect. Which is what systemd does and what mainly differentiates it from many previous systems which where mainly "just" start-up helpers. Through there are IMHO a large amount of problems with systemd the general approach is IMHO good, just the implementation isn't so much.
Sorry, no. Systemd doesn't add anything for service management that didn't previously exist elsewhere other than the unit file syntax for service definition. What it did is pull as many disparate aspects of "how do you run a thing" under its umbrella as it could, so it controls as much of the environment as possible and doesn't have to care about playing nice with anyone else. That's the general approach.
It's a logical fallacy to assume, because systemd was the thing that came along and resulted in the problems in sysv being fixed, that systemd's approach or implementation were necessary, desirable, or the best option available.
It violates the old school UNIX Philosophy of (1) simple standalone tools (2) text as the universal interface.
(1) systemd is a monolith that aspires to do almost everything from booting to networking to sound. It's in many ways a Kubernetes sans distributed systems.
(2) systemd uses binary logs, abandoning "text is the universal interface"
systemd is also a giant single point of failure.
Personally I don't see the big deal. The actual reason people hate systemd is that it's the type of design a software company like Microsoft would come up with rather than one that a bunch of gritty basement hackers would create. They have found refuge in OSes like Gentoo and OpenBSD, so systemd served as a kind of explusion of this type of programmer/user from the Linux ecosystem.
i went trying to learn how to make my linux box do something at startup.
as somebody learning how to use my linux, systemd clouded the discovery process of learning how to linux in the intended way.
i ended up using systemd, but now that's the thing i'm familiar with if i want to add some service to my computer... so it has infected me with some non-linux-y knowledge that won't go away soon.
Also compare it to doing things the Windows way: Something to run on startup? Put it in the "Startup" (magic) folder. Want something to do the things a service does, where it gets auto-restarted and auto-run and even logs success and failure? Like three clicks can run arbitrary anything using the scheduled task GUI and things get auto-logged in the Windows Event Log
i guess if i wanted windows, i could use windows... right?
edit for tone:
sry, that sounded saltier than i wanted. i actually went to linux because i was convinced that was the better way to do things, and i wanted to do things the better way.
I feel like this page actually comes off as a pretty good recommendation for systemd. Like it's a piece of software with normal bugs and the usual crop of hard decisions.
I don't experience the sadness that the writer of the blog mentions. However I do understand what they mean. Personally, I've decided I don't really like huge projects, and instead have focused on smaller iOS apps. They don't have dependencies, except for what the Apple platform offers. There's a backend, but it's really abstracted away for me through a RESTful API. And I'm able to continually and without fail, deliver something of value to my customer.
I don't know anything about what's happened with etcd in particular, but I think there is a genuine problem in this area with the model of "the people doing the work get to make the decisions".
If you have a piece of software which is basically finished, and a group of people come along who are interested in extending it far beyond its original purpose, then it's very easy for those people to end up as the official maintainers of the software simply because they're going to be the most active.
So when it comes to deciding whether expanding the software so much is a good idea (where the alternative is to make a fork to add all the new stuff in), the people who are happy with it as it stands don't get as much voice as perhaps they should, because for obvious reasons they aren't contributing many patches.
Hi, thank you for admitting in your first sentence you're not qualified to comment on the subject at hand, yet felt compelled to share your two cents. Here's the context for your totally uninformed opinion:
I was a major contributor for etcd 2.3-3.2 at CoreOS. The people who extended etcd to support gRPC included the original etcd author (hi Xiang). gRPC support was necessary for good performance; we had benchmarks to justify the decision. Likewise, v3 brought about a key-value model change that was incompatible with v2 to better support binary data, ranging over the keyspace, transactions etc. A v2-style gateway for v3 with a pretty JSON API was planned but never completed due to lack of resources; the ugly gRPC json gateway turned out to be good enough for most people. Similarly I wrote a proxy to run v2 requests over v3 instances, which does support the v2 JSON API. This isn't as if a new group of people showed up and ruined the software without caring about existing users. None of us were "Xooglers".
It seems what you're proposing is what the author both argues against and wrongly believes is what happened. We constantly pushed back against k8s influence. If we didn't, etcd would be a k8s sub-project right now. I'd also like to point out that removing the people who do the difficult work of actually writing the software from the decision process is incredibly insulting and devalues their labor. How dare you.
The problem is not with the Kubernetes but with management/architecture teams of companies that decide to use k8s for projects that have no real scaling requirements or when overhead of just having a constant fleet of nodes would be less than overhead of maintaining working k8s deployment.
Kubernetes is a hugely complex piece of software that is handling hugely complex cases that arise when you try to deploy and maintain a large number of applications with hugely differing scaling needs.
In my experience, k8s deployment absolutely requires a dedicated team with top notch k8s knowledge and debugging skills. You also need to understand k8s is a huge constant overhead as it requires actual knowledge and experience to use and has complex faults that require ability to debug really complex and interdisciplinary problems.
Google could do that because they invest in their teams and because they have exactly the type of problems that greatly benefit from a single complex solution that can handle all of them (meaning your engineers can migrate between projects but they still can use same tools they already know).
The trouble with management/architecture teams is they do not understand they are not Google. They are Google-wannabees. They like to promote how great they are and how great their projects are but are either deluding themselves or are being deluded by their lower echelons and have not set up themselves to understand what is actually going on.
And that's how you land at a situation where a team that has simple application that would require just two nodes (second for redundancy only) and no knowledge of Kubernetes is required to deploy to k8s instance and deal with a host of problems they never had to deal before that are beyond their capabilities.
It is. I even mentioned it is nice to have engineers be able to move through the company and be familiar with the tools (this is also called standardization).
But if you have only simple applications you will be surprised how complex it is sometimes to debug k8s problems compared to what you would expect from a simple application.
If some of your applications already have very complex deployment needs then you are replacing one complex homemade process for another complex GP solution.
> In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers). I would go so far as to say that Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd.
Oh for crying out loud. This Kubernetes bashing is getting so old.
Frankly, I think it's the best thing that has happened in the past few years. Boohoo it requires some YAML. Yeah. And then you get lots of value out of the box.
Systemd bashing is a bit more deserved, but this horse has been dead for a while now.
And there's also some CoreOS bashing too. CoreOS is/was spetacular.
I'll pile on: I'm working through Marko Luksa's "Kubernetes in Action", where he introduces this quaint line of code:
etcdctl ls /registry
He notes in an aside that you may have to do
etcdctl get /registry --prefix=true
for later versions of the protocol. But here I am with k8s 1.16, and etcd has been locked in a pod (seriously, why?) so before you contact it you need to "kubectl exec ..."
We're not finished. Apparently the pod was insufficiently secure, because now we need to decorate the etcdctl with a couple certificates, a key, and do SSL encryption to make contact, so in the end I have this wrapper (names changed to protect the guilty):
$ cat ~/bin/etcdctl
#!/bin/bash
declare -A EtcdHost=(
[clust_prodA]=clust1-leader
[clust_prodB]=clust2-leader
[clust_qualA]=clust3-leader
[clust_testA]=clust4-leader
)
if [ ! -z ${EtcdHost[$cluster]:-} ]; then
export KUBECONFIG=$HOME/.kube/$cluster
exec kubectl exec etcd-${EtcdHost[$cluster]} -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --endpoints https://${EtcdHost[$cluster]}:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt $*"
exit 0
else
echo "No known etcd host for cluster $cluster";
exit 1;
fi
The entry points for beginners have been nearly barricaded, and the ladders charred and dangling high.
> If you are running a truly enormous system and want to have off-the-shelf orchestration for it, Kubernetes may be the tool for you.
Not even then. If you're an enterprise using EKS, you still have to work around the pain points, missing features, incompatibilities, and do a metric shit-ton of custom integration. And that's before even touching CI/CD, and ignoring how you build the infrastructure.
It's like a Formula 1 engine. Really complicated, you still have to assemble the rest of the vehicle, and you need a team of engineers to understand what it's doing when you drive it.
I think the author is right. gRPC is bullshit that you only use if you're dedicating your whole organization to one protocol. They actually re-added that wacky backwards-REST API-gateway because people really needed a REST API. On the other hand, Consul not only has a REST API since the beginning, but it incorporates more useful (some would say necessary) features so you don't have to write them from scratch. So you can go with Consul and get more compatibility and features, or go with Etcd for the opposite.
But at the same time, a "lightweight database written around Raft consensus" is like saying "clogs for runners". Most people shouldn't use it, and those that do won't be happy with the consequences later.
etcd switching to grpc makes complete sense. The author seems to have limited experience with the downsides of http long polling and all the implementation and operational complexities it comes with.
As a counter example of Xooglers making things more complicated, I wanted to bring forward the removal of the protobuf exposure format for prometheus 2.0 after it was discovered that it's actually more performant to parse prometheus' text format.
It's free open source software. Take it or leave it. Fork it if you want. Use an old version if you want. At the end of the day the maintainers (presumably) decided to do this to etcd. They could've rejected the changes but didn't
Funnily enough, the part that this author seems to hate the most - moving to a well-defined protocol that has an actual typed description rather than some ad-hoc pseudo-plaintext format - is exactly what's wrong with systemd/kubernetes/etc - reinventing the wheel with new protocols rather than reusing what already exists. You don't have to like gRPC - I don't particularly like gRPC - but it gives you an interface in a well-defined format that's easy to play around with. That's head and shoulders above having to figure out the edge cases in yet another custom pseudo-plaintext protocol.
And how was it Xooglers? Wikipedia certainly seems to be saying this is a Google-initiated product, through and through: https://en.wikipedia.org/wiki/Kubernetes
> This guy used to run infra at uber and was incredibly salty about every single new technology. There were a lot of bad ones, but every conversation was about as constructive, free of evidence, and bitter as this blog post.
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
The apprehensiveness of sysadmins May have been justified in the world 5 years ago but today it sounds somewhat out of place. Note that critically evaluating new technologies is still an important skill and many infrastructure people I work with are extremely cautious about adopting new tech without spiking/getting to know it. But the sysadmins with the penchant for saying no is probably one of the reasons the devops movement actually kicked off: developers and infrastructure folks could see the immense productivity gains from being able to ship code quickly to production and got onboard the technologies that enabled this after being frustrated with the amount of time that traditional software deployment processes took.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
Almost everything in your comment is wrong. Kubernetes has enabled a whole host of observability tooling (eg opentracing), promoted a culture where app logs are easily and always accessible, enabled 0 downtime deployments for teams without dedicate infra specialists and so much more. It has made deploying reliable applications a lot more easier than ever before.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
Kubernetes does indeed have advantages but also brings a whole layer of complexity, overhead and moving parts. From my experience, in many cases the theoretical advantages don't end up being worth the tradeoff and/or don't even end up being implemented. Furthermore the particular things you mention (tracing, centralized logging & no-downtime deploys) can be done just as easily without Kubernetes.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
Your answer to everything I said is “actually, it’s not that hard without kubernetes”. Maybe it’s not for you. For most developers, it absolutely is. And that’s why kubernetes is popular.
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
I am approaching this from a developer's perspective. Kubernetes and similar (even local Docker) introduces an extra layer of indirection that you often have to fight with. Sometimes it's worth it, sometimes it's not. For me, if my application is misbehaving in production, I prefer being able to just SSH into the machine and figure out what's wrong than fight with the container layer, its authentication system, command-line syntax just to obtain a shell inside the container. When I am developing locally, I prefer having all my files on the local filesystem instead of having to worry about volume redirection and "docker exec".
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
I'm an infrastructure type and everything you said rings true for me.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
It is and it isn't. I remember the first time I thought "this software is just X that everybody forgot about, reinvented again". That was 2005. Not coincidentally that was a decade after I started working as a sysadmin.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
I find it hard to parse anything meaningful from your comment, sorry. I don’t mean to be rude, perhaps blunt.
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
> A decade is a long time in software engineering.
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
That’s pretty cool, really. I can’t really find studies on average software life time, so all I really have is anecdotes. I can’t imagine software running for that long on eg public clouds without any kind of major refactor considering how quickly technology changes today, and how aggressively cloud costs push companies to change. Specifically, I can see something like “Serverless” coming to maturity and completely changing the paradigm (again!) on how software is architected. But it’s possible I’m wrong and like you say, k8s will be around for much longer than I anticipate.
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
If you dislike the article, go ahead and say why. Leaving negative personal comments about the person who wrote it approximates your own comments: unconstructive, free of evidence and bitter
I haven't even read the article and hence don't really have an opinion either way, but "[not] constructive" and "free of evidence" seem like pretty specific criticisms to me.
Given that Uber is infamous for its extreme NIH syndrome, unnecessarily reinventing the wheel and spending enormous amounts of dev effort to create internal versions of products that already exist, I think it's relevant to the article to point out that the author almost probably had a key role in that culture, and what his philosophy leads to if taken unchecked.
Its safe to say the unix crowd recreated the Windows Registry. I guess it turns out the Microsoft guys weren't idiots its just a messy problem that requires a messy solution.
etcd is not a windows registry- it's more similar to active directory, but with consensus protocols to implement the multi-machine requirements for consistency.
As far as I can tell, the only thing this article is saying is that the author doesn't like gRPC, preferring hand-rolled APIs. (The rest of the rant doesn't really talk enough about the problems to respond to. The author doesn't like Kubernetes. The author doesn't like systemd. The author doesn't like software-defined networking. No reason is given as to why, so there is really no way to have a constructive conversation about it.)
Hand rolled APIs are easier to understand, but harder to maintain. It's great if you're only ever going to have one client, but once you need more than one, it's sure tedious to write and rewrite it for every language you want to support. Using gRPC means that you can auto-generate the client, and while they might not be as wonderful as writing each one of them by hand, at least you can get a client for whatever language you're using. And, the clients all behave the same way -- trying to figure out how to add interceptors to every bespoke client you need is quite tedious. (Look at how long it took AWS to get contexts in go, or how hard it is to add OpenTelemetry to the random hand-rolled HTTP client, etc. With gRPC, you just do those things once!)
Using protos as the transport layer lets you make backwards-compatible changes smoothly; adding fields is safe, renaming fields is safe, etc. The same is not true of using JSON -- if you call something "foo", you can't just one day rename it to "bar". Clients won't know what "bar" is. So you have to update clients and servers at the same time, and you can never "make before break". You see this all the time when someone rolls out a client/server update for a browser app -- your browser cached the Javascript, and it can't talk to the server anymore until that cache expires. It's nasty. I don't understand why people do that to themselves.
gRPC also adds defined semantics for TCP connection length; with HTTP/1.1, maybe you can reuse your connection, maybe you can't, it depends. You can't have multiple requests in flight on the same connection, even if you can reuse it. HTTP/2 fixes this, but gRPC has first-class channels and behavior is well-understood for request/response, streams, etc.
It is unfamiliar and not as easy to debug on the command-line as "curl http://api.example/foo", but once you get up and running, easy things are easy and hard things are possible.
As for Kubernetes, I dunno, it hasn't been bad for me. I tend to use the managed offerings, so I don't have to spare 5 machines for an etcd cluster / masters, or maintain them. I build a container and Kubernetes ensures that it runs forever. If it dies, it's restarted. If more replicas are added, they start receiving traffic.
I can manage 100% of the configuration in Git, so if my cluster or cloud provider blows up, I can re-apply somewhere else and have a 99.9% chance of it all working within 15 minutes. Before containers and k8s, production felt very much like a "yolo" thing to me. Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release. You needed to distribute root credentials, hoping that you could fully trust everyone on your team to not mess anything up. It mostly worked, but through sheer brute force rather than any system working behind the scenes to make things run smoothly. With k8s, you can delegate this tedium to software. A developer can update a config file, have the PR approved, and software will ensure that the new version of the software starts, is assigned some load, and the old version is shut down. It's smooth, hard for someone to manually mess up, and quite productive.
I get that people have made their own ad-hoc orchestration and they like it. I guess that's OK. What I like about k8s is that it's a common language for this sort of thing -- I've used it for 3 projects, and they've all looked about the same. I've never had to learn anything company-specific; I can just run my software. I don't think that's a bad thing.
(The solid alternative to k8s, it seems, is just paying someone else to run everything but the one application that your company makes. Can't figure out how to run Prometheus? Just buy a Datadog subscription. Can't figure out how to run a frontend proxy? Just buy an Application Load Balancer. Can't figure out how to search and retain application logs? Just buy a Splunk subscription. Can't figure out how to install MySQL? Just buy it from your cloud provider. That is all great, but you end up spending a lot of money because you can't efficiently manage software and instead spend all your time writing rants about orchestration frameworks. Sometimes I wonder.)
Whenever I see articles like this, I have to ask what the transition from "big UNIX" to Linux would have looked like on HN. I am sure some people hated it, and I'm sure some of those people had good reasons. But things got smoothed out and it turned out that Linux and its ecosystem was pretty good. If you maintain a production application today, it probably runs on Linux, and that's good because that's one less thing you have to teach your team. I kind of see k8s in the same place. Lots of people have trouble running multiple pieces of software in production. k8s is a common language for doing those things. You can build it yourself or you can buy a managed service. You can extend it to do the crazy things you need it to do. It's not a bad thing, and I think it's pretty disingenuous to compare it to systemd.
thanks for the succinct explanation of gGRPC over HTTP / REST / whatever.
I've been trying to figure that out for a while.
Think your analysis is spot on overall.
The author strikes me as the person who was saying "why can't I just keep writing Assembly?" in the 90s .
The answer is you can keep using <X technology at a lower abstraction level>.
But don't be mad when people are using the new abstraction layer to build interesting stuff because they don't have to worry about the lower level as much.
> Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release.
that's not really true, configuration management on mutable systems was pretty widespread before everyone started doing immutable infra.
One should not forget that a big corporate is not made out of great people, but actually only of normal sized people... just, well, a lot of them!
If you hire a lot of people, they all will need to occupy a space each. To deserve that space, everybody needs to be either quite good... or good at faking being good.
Truly innovative things are done by great individuals. And corporates are not the natural habitat for great individuals.
People think it's easy to create a durable fork that will gain enough traction to live on many years (in a stable manner) and attract contributors. It's not. Just look at the tons of forks on github for abandonned opensource projects.
Let's say there are 10 people willing to fork it to keep it simple as it was previously. Where is the go-to place for that to happen in the open source world?
In your 10 people example, it seems that there are a few problems to me:
- finding like minded peers
- getting critical mass (are 10 enough to maintain the fork?)
- getting them all to agree on what subset of features to keep
If I were in the market for a fork, I would search Github first before forking my own, so I think the first item is basically solved. The second varies on a case by case basis, of course. The third seems more critical to me, probably dispersing the effort more than anything else.
When I read these anti-kubernetes articles, of which there is one every couple of months, I think of an analogy with tailors. I mean, a made-to-measure garment is strictly superior to what you can buy from Old Navy. If you take a tailor into Old Navy you would probably get a similar rant about how badly made the clothing there is. I also recognize that everything that Kubernetes/Docker is doing could be replicated more simply and with greater craftsmanship.
I think the unnoticed problem is that the tailors (old school system administrators and dev ops) are being asked to become the managers of Old Navy (kubernetes cluster administrators). Their entire skill set is opposed to this new role. I can't really blame them for being frustrated.
To keep stretching my already thin analogy, there are still tailors in this world. However, most people buy their clothes at outlets like Old Navy. Whining about the quality of the clothes and the crummy manufacturing process at Old Navy won't change that.
> When I read these anti-kubernetes articles, of which there is one every couple of months, I think of an analogy with tailors. I mean, a made-to-measure garment is strictly superior to what you can buy from Old Navy. If you take a tailor into Old Navy you would probably get a similar rant about how badly made the clothing there is. I also recognize that everything that Kubernetes/Docker is doing could be replicated more simply and with greater craftsmanship.
But down in reality, most companies are probably running their own made-to-measure deployment and operation schemes with lower quality and consistency than Old Navy.
Of course, just as many tailors back in the day were likely to make clothing worse than Old Navy makes today. We fetishize the top 1% of masters as if the average quality is anywhere near that.
Again, it reminds me of a carpenter friend who hates Ikea. Yeah, I get it, Ikea is really bad compared to custom built furniture. No one who buys it expects anything else.
This story is older than the industrial revolution. Craftsman being replaced by technology. We even instinctually know this is going to happen to knowledge workers but we seem blind to it when it happens to us.
I don't really use k8s nor etcd, but one sentence in the article really stood out to me:
> I would go so far as to say that Kubernetes is the worst thing to happen to system administration since systemd.
I know it's popular to shit on systemd, but as a casual user, I honestly don't get it. I've written about a dozen systemd unit files over the years, and each time it's been a pleasure. Write a few lines describing what to launch when, one command to make systemd aware of your new file, one command to launch it.
In contrast, I've written a couple sysvinit files, and the experience was horrible every time. Having to manage pid files and logging myself, setting up the proper symlinks, etc. It'd always end up being a ~100 line script that'd be 99% copy-paste.
There might be some use-case where systemd is worse, but for my (admittedly simple) use-case, it's far superior.
I personally have no beef with systemd and completely agree with you that having a simple and standard way of composing services/daemons is a wonderful thing, and one that was sorely lacking for a veryong time indeed.
My impression from the systemd debate is that those that are against it feel that way because of the breadth of systemd - it's much more that "just" about service units.
It's almost that every month there is an article like this popping up in HN.
The main ideas are all similar: "I don't like complexity, simplicity is king. React, K8S, SAP and bluh bluh are bad. "
Maybe I just don't really fit in here. I almost always feel the opposite.
I don't find those new, so called "complex", software difficult. I can pretty easily see why that we need those new features and what problem are they trying to solve. I'm really grateful there are ppl out there solving my problem one-by-one for free. I feel those new softwares enable me to do much more and better than I could do 7~8 years ago.
I feel sorry for the fellows who just can't grasp new technology quickly but this world is cruel.
I think this is the problem for open source software. It's not a one size fit all.
We all do complex software and you all know its complexity. Sharding, failover, retry, back off, schema etc...everything in a high scale systems.
The problem is user who used it but doesn't need other features feels frustrated. At the same time, user who used it at scale need feature others don't like? What can we do? I don't know.
I can understand that people who are used to HTTP based APIs will feel more comfortable being able to tap into a rich fauna of tooling for HTTP.
However personally I am not really interested in raw network APIs as long as they work. I'm interested in programming - not poking around low level stuff.
I'm interested in libraries for speaking to services. When I use services like Twilio and Stripe, I'm not interested in rolling my own. I'm not going to implement my own unless I really have to. It isn't nice to just publish an API and not have libraries for talking to them.
gRPC, at least for me, makes both initial bootstrap, development and maintainance easier.
This post sums up the last decade. Shouldn't tools and protocols become simpler and easier to use? If megacorps can use complexity to reduce democratization of the web, they will because of profit motive.
> Anything that has a simple and elegant feature-set ends up coöpted by people who just want to build big ungainly architecture and ends up inheriting features from whatever megacorp the coöpters came from
I feel the author's pain. Earlier today I literally said "modern software makes me wish I was dead". Things are so unbelievably bad now, and nobody in a position to do anything about it seems to care, and the kids growing up with this garbage think it's normal that it sucks so bad. No, it's worse than that, they celebrate it! It is now at the point that I no longer want to work in this field and am actively seeking alternatives. I just can't deal with the bullshit anymore.
I can 100% agree regarding the evolution of etcd as I used several versions of it and the amount of workarounds I had to use on every version went from 0 to "too many moving parts to know whether I will get a call over the weekend".
With gRPC came as well came as well incompatibility of many clients with less popular architectures and as the article describes, being that a super specialization (because that's the only thing that it is), we are losing the qualities that made OSS legendary.
Joe Armstrong spoke of "the mess we're in" and I think he captured it well. He was referring to the symptom in general, where-as this essay is about one of the root causes.
Having recently learned bits & pieces of Kubernetes, I can see the author's perspective as it's one I held for some time. Kubernetes is useful and makes many things easier. In exchange you give up finer-grained control and you have to work with an abstraction. Such is the nature of all abstractions.
My favorite excerpt from this article and one that sums it up nicely can actually be found in one of its footnotes:
> I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol [HTTP/2] pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.
> That's it. That's the story. Popular modern technology is taken over by expats from a megacorp and made worse in the service of a hyper-specialized (and just plain over-hyped) orchestration platform
If you hate the direction the open source project has gone in, why not fork it and have the v2/HTTP version live on forever? The project does not owe it to you to stay crystallized at a moment in time.
When v2 is removed, I'll be shocked if etcd isn't forked because it still has enough value to enough people to maintain it.
But it hasn't been removed yet. It might be cynical, but I think a v2 fork now won't accomplish anything because the people most interested in one would simply continue to use v2 upstream.
Unrelated note / self plug - your comment section is hard to read. I built a comment tool that will auto adjust to dark sites like yours, and it supports importing from your comment tool:
This a rant post that backs up very few of its assertions. Though, the author may not have been trying to write for serious consumption. Sometimes it's therapeutic to have a good rant.
I'm not sure what there is for us to discuss. Nice rant, I guess? The post does not attempt to persuade or change opinions (which, again: cool. sometimes it's nice to have a good rant).
> I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server,
Ahahahahahah, yes, anyone can implement a HTTP server, just badly. HTTP/1.1 is quite complex, the spec alone spans over eight RFCs: if you can implement all of that I doubt the HTTP/2 serialization is much of a concern. :P
I stopped reading at "Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd.". These are the best tools to happen for system administration in 20 years. Also, etcd is a terrible db. You cant back it up and can easily lose quorum (and all your data) with a network outage.
As systems get more complex, whether they need to be or not, the issue I believe is no one takes the time to explain or document why. They just plow ahead building complex stuff at times seemingly so they can say “look at this complex system I built to solve that really hard edge case”. When 99% of the use cases were already solved.
OP's piece is heat not much light except maybe replacing HTTP with gRPC (if true). Should have added second path and made it configurable. Storing Kubernetes stuff in ETC is the raison d'etre for multi PAXOS systems: it stores stuff. Also ETCD seems to be an innocent by-stander: the real whine seems to be about K8s.
The big missing piece in this article: if the author has a real need for the old simple etcd, he can always fork it. If nobody has done that, that means nobody has enough of a need for a different etcd from the one the Xooglers took over. In which case it's hard to work up a lot of concern.
> If you are running a truly enormous system and want to have off-the-shelf orchestration for it, Kubernetes may be the tool for you. For 99.9% of people out there, it's just an extra layer of complexity that adds almost nothing of value.
This is probably even true for 99.9% of HN readers.
In a world of space architects and ignorant cargo cult practitioners, this post is a breath of fresh air. While some might find the style vitriolic and “toxic” it’s nothing but honesty peppered with salty experience.
Having said that, the guy represents the old guard of tech, the kind of people who ask “why do you want to do this” instead of trying to help you, and usually follow this up with a useless suggestion for an alternative that sort of does what you want but not really. Thanks, I guess. They have a very concrete idea of how things should be done, new ways of doing old things are frowned upon, and anyone challenging their “authority” is seen as an imbecile.
I had the dubious pleasure of working with such a man a few years back and while I couldn’t but admire his technical expertise and competence, I grew to hate him to a degree that would rival that which the author of the blog post hates Google.
Get off our lawn, old geeks! You’re not the only ones with opinions around here.
Why on earth did this garbage make the front page? For shame!
The author is a total jerk trashing everything in his path with hyperbole rather than sound logical arguments. Read his rant on protobuf. Why would anyone waste their time tearing down the contributions of others?
At some point huge complexity are introduced to simplify a problem and solve for a gap or feature. Don't look at this as a burden, look it as a use case you don't require currently and this may not be the right tool for you to use.
Meh, author certainly has a point about K8s being overkill for 99.9% people out there.
Unless you need to operate at a certain scale (that most people never will) then Kubernetes is like taking an F1 car onto a regular road. Recipe for pain.
OK, there seems to be a parallel universe of computing where systemd, Kubernetes, gRPC and Protocol Buffers are considered bad. Time will tell if they have been right, or are sore dinosaurs.
Well can't say I'm surprised at these reactions. Random internet dude releases opinion, followed by everyone else releasing their opinions all over the internet. Discord ensues.
1. The project leaders agreed to this; it's not just the Google people. If it were just Google, they would have had to fork the project and call it getcd or whatever, then ruin it at the expense of their own reputation.
2. Speaking of forking, it's still quite possible for a few interested developers to fork an clean older version, call it freetcd, and rebuild the reputation. It worked for LibreOffice after all - and that's when OpenOffice wasn't terrible, just somewhat mismanaged.
All that said - I don't quite see why you need a specialty database for a simple key-value store for /etc settings. Aren't there plain-vanilla kv-stores which would do?
I might not agree entirely, or even mostly, but I have seen how trying to emulate google can seriously harm applications or open source projects. Guess etcd is an example.
> The software development world would prefer to use their multi-gigabyte IDEs running on ElectronJS to build thousand-dependency Java applications targeting ungainly APIs on hard-to-operate systems than support something simpler and better.
I think that developer tools have always taken up a significant proportion of the resources available on a developer machine. Back in the 90's, Visual Basic was criticizes as being bloated for requiring a 4MB of RAM (the exact number may be off).
Now we have vastly more powerful computers. I think using those resources to have easier, more extensible, and more capable developer environments is a good tradeoff.
First time I’m aware of that reading the footnotes provides at least as much meat and interest as the article itself. There’s even footnotes in the footnotes.
Google rolling their own replacement for TCP and HTTP is a disappointment. Their own benchmarks indicate they get maybe 10% more performance on a good day. All this comes with a huge increase in complexity. Which means more bugs.
The "solution" to bugs today is forced updates. Vendors love being able to forcibly impose whatever they want to do on the user. Turn off features, put in more ads, whatever. If software was reliable enough, nobody would upgrade, which damages the business model. It's so convenient that software seems to need constant "security updates".
I think that author is more dev than ever ops, a year to try to support the project not on dev but on prod in highly available configuration could make eyes open
It all seems like a position of a dev not ops.
I suppose a year in supporting his projects in prod in highly available configuration can change his oppinions.
I have not found etcd to be 'an absolute pleasure to work with', to put it lightly. It has been a plague of stability issues and it sometimes seems better to roll one's own than continue tracking down issue after issue in etcd (yes I realize DIY has its own set of probably bigger problems :) ) I don't know if this experience is due to the changes on the original etcd implementation that the author is describing.
i always cringe when the tech megacorps buy out an up and coming start up that shows promise. good for the start up employees as they cash in, but i can’t help but think that overall these actions stiffle future innovation moments that could have been if the start ups matured more on their own.
There is an element of "craftsman blaming their tool" here. I don't mean that as a criticism of the author, who is clearly frustrated with some choices they didn't make but nonetheless have to suffer the consequences of, but rather of the larger context. Clearly, someone, somewhere is craftsman blaming their tools.
I actually block all QUIC traffic, the only use I have seen for it so far from my OSX machines is chrome phone home traffic. At the weirdest moments, suddenly little snitch pops up, hey chrome wants to connect to ip x.y.z.1 443 QUIC
He is partially right because i think containers or vagrant are good tools for development in 80% of all cases...but on the other hand i like having monolith's in testing and production...so there's that ;)
Too much hate when the solution is quite easy: Fork it from the version that you like and maintain it.. I'm quite sure lot of people would love to support you on that.
There may well be evil at google, but it's not here.
My take on this is that CoreOS and google have different use cases, and the beset solution would have been to fork the project. Since google is big [citation not needed], most development effort would probably end up there, but people who wanted the simple version could stick with it.
The difference between a HTTP API and gRPC is one of scale. If the HTTP one is a few bytes more per message, and takes a few more clock cycles to decode, then at google-scale that might mean extra energy use on the order of a small town. RPC makes complete sense for google here.
Then there's reliability. If something fails one in a million times, it's a minor inconvenience to someone who uses it once a day but if you're running it four billion times a day then you start to notice. gRPC is strongly typed so it actually removes the malformed HTTP failure mode - and you can bet that google's SREs are taking care of the other ones too. When the OP says "Quality is, alas, a dying art", I'm not sure whether that's an honest description of just how many "nines" the google SREs are building into their projects. It's not like they invented gRPC to make their systems less stable!
I can recommend rachelbythebay's comments on this, she's a SRE who has done the rounds in silicon valley - appropriate posts here are "Some Items from my Reliability List" [1] where she famously says "there is far, far too much JSON at $company" and ends that section with "Why aren't you using some kind of actual RPC mechanism? Do you like pain?" - at the scale she's operating, RPC is less pain it would seem.
In "We have to talk about this Python, Gunicorn, Gevent thing" [2] she criticises the use of a popular python framework at the scale she's operating at (I'm sure this was shared on HN in the past).
This doesn't mean that python, "RPC over HTTP", JSON or etcd are in any way bad. In fat their simplicity makes them great things for beginners to learn and medium to large companies to use - they're not just prototyping tools, people can and do build a lot of real stuff with them. It's just that at some point before you get to the ridiculous scale of google's infrastructure, there's a tipping point where going full RPC starts saving you a lot more than it costs; that's not a bad thing and it has nothing to do with trying to "capture the market" or anything like that. A corollary of this: precisely because google works at such a scale, there will be a lot more developers working on grpc-etcd than the "consumer" one.
Summary: developer builds great tool for A; big company has use case B.
6. HTTP/2 a.k.a. SPDY is a comically bloated Layer 5/6/7 mega-combo protocol designed to replace HTTP. It takes something simple and (most importantly!) comprehensible and debuggable for junior programmers and replaces it with an insanely over-complicated^7 system that requires tens of thousands of lines of code to implement the most minimal version of, but which slightly reduces page load time and server costs once you reach the point of doing millions of requests per second. I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web. ↩↩
7. My favorite HTTP/2 interaction has been finding and reporting this bug in haproxy. The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 S: Aa is just a list of the 61 most popular headers from Google properties. ↩
8. HTTP/3 is all the badness of HTTP/2, but run over a worse layer 4 protocol named QUIC that totally fucks up networking for everybody in order to get a tiny bit more optimization for Google. That's all it does. It makes the Internet strictly worse for everybody but slightly better for the hugest of huge web properties. Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to work. ↩
"Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to [continue to] work."
These protocols do not make it more difficult for advertisers or advertiser-funded companies like Google to serve ads and optimise serving ads, they make it easier. For example, they allow for faster and more efficient serving of third party resources, e.g., ads, and siphoning of user data without any affirmative interaction from the user, and cramming more data into more headers that are more difficult to monitor. As an end-user I aim to send the absolute minimum headers needed in the particular instance. 1-2 usually works. From an end-user perspective we should be trying to decrease headers not increase them. Websites should only collect the information they actually need, nothing more. That is not the sort of web envisioned by the "new" HTTP protocol advocates.
I actually use HTTP/1.1 pipelining with a user-agent I wrote myself and this original HTTP pipelining has worked great for me for over 20 years. I am not an advertiser or an ad-supported company; I am just an end-user retrieving text in bulk from the web. Thus in addition to state-aware firewalls and load-balancers, I want HTTP/1.1 pipelining to continue to work. The proponents of these protocols want their audience to believe HTTP/1.1 pipelining doesn't work or has serious problems. But that is only for their use case. In fact it does work for end users doing efficient bulk data retrieval without retrieving any ads or other cruft. But these Google-sponsored protocols are not written for end-users wanting to avoid ads. Those are not Google's customers.
There's a lot of justified hatred for kubernetes, but for fuck's sake why did anyone adopt it in the first place, if they didn't understand what they were getting into? It's like the startups from 10 years ago who absolutely had to use MongoDB because "scale", and then they never get off the ground because they're trying to implement ACID from first principles.
It's possible that many people (like me) had been trying to solve a similar non-business problem for years, including rolling our own solutions and the like, and when an open source option widely backed emerged and looked like a possible standard, we accept some warts in exchange for a broad, general purpose, flexible, automatable, and well-thought out (yes) solution.
Of course K8s is not perfect, and it's overkill for small to medium apps (I think hype train convinced a lot of people they would need to scale to massive cloud levels when really they didn't), but if you have ever needed K8s (especially for a complex microservice system at big enterprise level) then you know the value and you remember the proprietary vendor-locked era of sadness before K8s emerged.
Engineering-driven-development, where the purpose of engineering is engineering and business problems take a backseat in favor of hiring lots of engineers, giving them toys to play with and having buzzwords to on your careers page. In this case, your objective is to fill up your stack with as many shiny things as possible regardless of whether they are necessary or appropriate to solve the business problem at hand.
This strongly correlates with bullshit VC-funded companies that wouldn't be profitable even on a good day so their only objective is to raise as much funding as possible (the business problem is secondary, because even solving the business problem perfectly wouldn't make them profitable) and chasing fads and buzzwords is sadly a valid strategy for raising capital.
There is a cost for not getting on a hype train. CEOs rather waste a couple of billion dollars on that machine learning/AI hype train, because if they don't hop on that train, Wall street folks see these companies old-school, not innovating. So, they bring in Chief AI officer and write about about how machine learning gonna help their companies in the Quarterly reports. Even if they lose $2B in this AI venture, just treat as insurance premium to keep the stock price going up.
This. I cannot tell you the number of times where I've seen teams use Mongo and then reinvent a schema as part of the development contracts. It's enraging.
Since hackernews is mostly junior devs or senior principle architects (formerly junior devs), I expect there's a lot of stockholm syndrome going on in these comments.
Your statements are overly hostile and hyperbolic. There's been plenty of great software out of Google and lots of good ideas taken for granted. I'll give you folks trying to build things the Google way without Google tools or scale is more often than not the wrong thing to do, but that doesn't invalidate the good.
This is much more about your opinion, and your axe to grind against google, rather than an intelligent discussion about the merits of software.
Personally, I really like JAX and XLA. These are two really powerful systems that work together to implement high performance machine learning and theoretical physics/biology. JAX can basically take derivatives of python functions (useful for training) and XLA optimizes the underlying compute graph to execute quickly on different physical platforms.
I'm serious. Say what you will about the high complexity of the web platform, or the rising dominance of Chromium. It's a seriously impressive piece of engineering. Have you ever watched the Chromium build process in action? It builds hundreds of static libraries, then statically links them all into one monster executable or DLL (depending on the platform). BTW, part of what enables static linking at this massive scale is the gn build system, which AFAIK is inspired by Blaze (the internal ancestor of Bazel). So IMO that's a major point in favor of that type of build system.
BTW, to answer your point about throwing tens of millions of dollars at a problem: some problems are just unavoidably that big, especially when you factor in things that are required for a piece of end-user-facing software to be usable by the whole world, such as internationalization, accessibility, and in general, complex but usable UIs.
Bazel is fucking awful. Most miserable years of my career is when I was tasked with maintaining a build system based on it. Same for gRPC, which TFA also does a good job of shutting down.
Blaze/Bazel has its flaws (including poor integration into the world outside google), but what are you comparing it against? Any general purpose build system I've seen that is not bazel based and not nix is a flaming pile of garbage: almost nothing else can even figure out a correct dependency tree, let alone which parts of it have changed and need recompilation. Bazel also uses a familiar, readable and yet concise subset of python for build description, whereas most other build systems use customer languages with syntaxes and semantics that are comically bad if you don't have to use them and tragically so if you do.
Again, gRPC has its flaws (and I'm not a fan of HTTP2), but what are you comparing it to? For internal services REST is acceptable for extremely simple, stable and low volume APIs. For everything else it is an obvious disaster: terrible CPU/network/memory performance, you need to write more code than you would have to with gRPC and get zero type safety unless you add some json schema monstrosity on top of it at which point your performance will likely take a further nose dive. Also, good luck evolving the protocol – gRPC is carefully designed to support this. Far from doing a good job, "TFA" quotes some extremely clueless article on why protobuf sucks that was ripped to shreds when posted on HN previously.
The only compelling alternative to gRPC I am aware of is capnproto (which is derivative, but in some ways nicer), but it has far less eco system maturity and mindshare.
Yes, sorry I was imprecise, it would have been better to have written "derivative of protobuf". To the extent the RPC part is derivative, it's probably mostly derivative of E?
I'm going to recommend at this point, you take a breather from your invective, cool off for a bit, and return when you have something valuable to contribute. You're going to have to do better when commenting on Hacker News: you're just attacking stuff because you don't like it, rather than making technical explanations.
> Bazel is fucking awful. Most miserable years of my career is when I was tasked with maintaining a build system based on it. Same for gRPC, which TFA also does a good job of shutting down.
Counter-anecdata: my most miserable years of my career were when dealing with bespoke build and API systems that weren't Bazel and gRPC.
I thought each of these were good until I tried something different. I think people moved from legacy things like Windows Live Mail, MapQuest, and Word 2000 to these services, and then never noticed the rest of the world got better outside of Google-land.
Gmail operates what is probably an industry-worst mail client. I have helped people using AOL Mail in 2020, and it is a superior experience. It performs better. FastMail is a paid service, and it's quality matches that: FastMail can run in circles around Gmail every single day.
Google Docs/Sheets is Office without all of the reasons businesses use Word. You can try to patch in some of it's holes with add-ons (I had some experience with a Mail Merge add-on for Gmail/Sheets, and it was fine, I guess, but a paid service from a third party) but with a lot of added jank. Collaborative editing was it's claim to fame, but you can do that just fine in Office 365, with a far more capable app suite.
Google Maps, Drew covered the point pretty well: Google just has the money for more data. It's prohibitively expensive to use that data yourself as a service, and others who have access to less data still manage a comparable and sometimes better product than Maps. Usually while better respecting your privacy.
Gmail sucks. They use it as a vehicle for anti-competitive "standards" and it's needlessly difficult to deliver mail to. Their new JavaScript-riddled frontends (which they seem to develop and then shrug off every couple of years) have awful performance and are terrible at doing what has ultimately a very simple task for 30 years.
Google Maps is fine. OSM is superior technically but Google Maps only wins because it has access to better data. It's not an example of great engineering, it's just okay.
Google Docs is pretty good. At least it was, the last time I used it was several years ago. I can't comment on the code, which is mainly what I'm commenting on in this thread: their engineering quality. Throw enough monkeys at a typewriter and maybe Google Docs comes out, but who knows what it looks like on the inside.
Engineering quality is about nothing other than meeting the spec- how well does google accomplish that?
It sounds like you have some issues with the design itself, maybe some issues with the business practices. Those are issues with the spec, not the engineering. The engineering team delivers the spec, they do it smoothly, they do it at scale
Highly used? So what? Might does not make right. They're definitely not cost efficient. Google's standard play is throwing tens of millions of dollars at a problem, which is not good engineering, even if it works.
Edit: you changed your comment out from under me and I fail to rouse the enthusiasm to reply to your new comment.
Go was made by a small group of smart people with debatable loyalty to Google. Google just ran with it, and they haven't done a great job with it since.
I'm not sure what you mean by debatable loyalty. The creators of the language have all worked at the company for more than 10 years and support the product for internal users.
Can you put your grinding axe away and stick entirely to facts and technical merits?
Look, Go has way more to do with the experience of its creators than with their contact with the engineering culture of Google. The influences from Plan 9, Inferno, Limbo, and so on are all extremely apparent in Golang's design and were all born outside of Google. Very little of Google's spirit made it into Go.
I don't know how you got from "debatable loyalty to Google" to "the language was mostly inspired by Bell Labs tech, rather than Google", which is perfectly fine. But it doesn't explain your animosity and refusal to discuss based on technical merits, rather than just (what is clearly) axe-grinding.
Note that Go ended up being a major language within Google for a wide range of systems (even replacing systems built previously by the Go developers, such as sawzall), so rather than saying "debatable loyalty to google", might it not be better to say that the developers worked to transform Google from the inside to be less dependent on C++ and Java?
I retract "debatable loyalty to Google". I meant something more along the lines of "debatably attributable to Google's engineering", i.e. a statement directly in support of my main argument.
Bazel is the most complicated garbage I've ever had the displeasure of using. It takes tens of thousand of lines of support code just to add a new tool to your toolchain.
> It takes tens of thousand of lines of support code just to add a new tool to your toolchain.
No, I've added custom compilers and languages to Bazel, and it was not nearly 'tens of thousands of line'. Sure, it's complex, but such is life with inversion of control. It's difficult to get around this complexity if you want to follow the design goals of Bazel.
> gRPC, k8s: see TFA
The article says nothing about why gRPC or k8s are bad. It just uses very emotional language to handwave into the general direction of 'bloat' and 'too complex'.
I initially thought this was serious hyperbole and I still think it may overstate things a bit, but when I thought about it I actually had a really hard time coming up with counterexamples. Meanwhile it’s easy for Microsoft and Apple. Huh. Best I can do is products that were great at launch... like 12-15 years ago.
There are people attacking the author for a statement made about CoreOS, and for some hate towards Kubernetes.
The key point of the article is not really being addressed here: vested interests from large companies are able to introduce huge complexity into simple, well-designed projects. While the complexity may be good for some end that said vested interest has in mind, they are also in a position to absorb the cost of that increased complexity.
In the meantime, the simpler version of the software is long gone, with the added complexity placing a large burden on the solo developer or small team.
It's almost like there's no good representation in the open-source world for the solo developer or small team. Funding and adoption (in the extreme, some might say hijacking) of open-source projects from large corporations dictates the direction of all major software components today. Along with the roses come some very real thorns.
Just my 2c.