I used to think this, but now that I know a bunch of these cloud tools; I think these frameworks standardise things between companies.
Kafka is well documented and understood. Your custom postgres solution is not.
Best case you write high quality maintainable code and have a single dependency. Worst case is you create a complete mess that takes months to onboard people.
Right now I'm contracting on a project that uses AWS BATCH, docker, kubernetes, terraform, Nextflow, Django and postgres. It took me 2 days to on-board myself and start delivering features because everything was standard.
I've worked for a while now in the field and I never worked with kafka. Nor most of the peers I've worked with. And I wouldn't even call it the most used message queue or data stream solution, so I don't understand why people treat it as ubiquitous software. SQL, on the other hand, is. And while it's certainly not built for message queues, that's not the point of the article.
That's not my argument. I'm saying it'll be easier to onboard developers onto a standard message queue (like Kafka) than to onboard developers onto some custom thing you wrote.
The original article is rebranding "Not invented here" as "Minimise dependencies"
But kafka isn't a standard message queue, and therefore won't make onboarding easier except for the ones who already know it, which is not a significant majority, right?
Or at least we'll have to agree on the definition of standard here. For the equivalents of SQL, I'd say we have AMQP or MQTT, and kafka does not seem to support any (my goggling tells me it has its own protocol); deployment-wise, it doesn't seem easier to set up than, let's say, SQS; while used and supported in most common languages, most queue/job frameworks do not seem to support it OOTB (looked into ruby, python, php, javascript), so you'd have to write your own custom kafka handling code from scratch. So what would make it easier than just riding postgres?
For me standard means used by more than one company and well documented. It also means that most companies use it in the same way. Basically all 3rd party Frameworks and tools are "standard"
Custom means that you wrote a unique solution for yourself. No-one else uses your solution. New hires have to learn it from scratch either by reading your code or your own custom documentation.
There is obviously more work to onboard people onto a custom solution. You can hire people who understand the standard solutions.
Postgres isn't a queue or a message broker. You are writing your own queue that depends on postgres. Your custom queue will have complexity that new hires have to learn. No new hire will have used your custom in house queuing system before.
If you want to use SQS that's a great idea. I support using standard solutions.
Kafka is well documented and understood. Your custom postgres solution is not.
Best case you write high quality maintainable code and have a single dependency. Worst case is you create a complete mess that takes months to onboard people.
Right now I'm contracting on a project that uses AWS BATCH, docker, kubernetes, terraform, Nextflow, Django and postgres. It took me 2 days to on-board myself and start delivering features because everything was standard.