Im not a backend guy. I assure you that I’m not a magician. As I never had the fortune of being blessed with the knowledge of modern backend stacks like k8s. All I know is at a surface level. This probably makes me naive. At the same time, I see these problems people deal with and wonder why make it so difficult.
Nats is a at its core a pub-sub system. Everyone (clients and services) connect to the Nats server (or cluster but same principle). Then you can either publish or subscribe to messages, whose payload is typically simple json.
Anyway, the point here is that with such a setup, you don't need to worry about networking between services. How do I stand up new service, let’s call it Galactus? You connect to the nats server, then you listen for messages on a subject space like “galactus.*”. Now anyone else can talk to Galactus, without proxies, api gateways - Galactus doesn’t even need a URL, only a subject (this is what subject based addressing means). As a result, you only worry about networking once.
These systems have been around and greatly improved for all kinds of use cases, yet it seems like nobody is paying attention.
I don't think nobody is paying attention. Message oriented middleware has been around for decades and pretty much every company with a service oriented architecture is using it somewhere, whether as ActiveMQ, RabbitMQ, Kafka, SNS/SQS or whatever else.
The issue is that moving from synchronous API calls to asynchronous message passing introduces new complexity. It's not suitable for every problem. Most places I worked use a combination of synchronous APIs and asynchronous messaging to share information between services, depending on requirements.
Of course in some cases you can just share access to the same database or filesystem too.
> The issue is that moving from synchronous API calls to asynchronous message passing introduces new complexity. It's not suitable for every problem.
This has been a problem I've been trying to solve for years! Surprisingly, it's a fun problem space, mostly because I disagree that it isn't suitable for every problem.
The solution I've been converging on is quite interesting, essentially a non-linear programming language that you write linearly (similar to how CPU branch prediction runs code that hasn't executed yet ... just on a more abstract level). It's pretty neat in how it works under the hood, but on the programmer level, it feels like just another programming language; but like erlang/elixir, the execution can be spread all over the world, or just your machine.
Probably one of the coolest things was when I got optimistic mirroring working, where local code will "predict" what distant code will do and then allow your local code to run to completion but not commit the result until the remote code agrees with the prediction -- and if it disagrees the code has to rerun that portion.
Fun stuff, but still probably a few years away from a HN post about it... it's just a fun side-research project atm.
> Fun stuff, but still probably a few years away from a HN post about it...
Remember that partial results are very useful too. Personally I like reading posts that have unfinished ideas, with commentary. Eg “I needed something to achieve X and I settled on Y. That causes some annoying side effect Z, so I’ll keep looking for a better solution”. That stuff is even more engaging than “look at this almighty finished and perfect system”.
Heh. Most progress is done in error messages at the moment: “if I get this error, then it didn’t work, but if I get this error it did”. If I solve distributed systems unit testing, that will be nice and worth a post all by itself, but not quite there, yet.
> The issue is that moving from synchronous API calls to asynchronous message passing introduces new complexity. It's not suitable for every problem.
Well yes but request-response is simply a pattern that can be implemented on top of pub-sub extremely easily. Nats implements it with a temporary generated inbox subject for responses, so dead simple. This keeps application logic conceptually identical to traditional rpcs.
The gain though doesn’t come from the messaging patterns themselves though, but the addressing. Going from “I wanna talk to X” to “I wanna talk about Y” is an extremely powerful shift of mindset.
