Some things cannot be tested. I work in distributed systems a lot and you can unit/integration test simple functionality, but there is nothing which can test how your system will behave in prod outside of just trying it out.
I find having really good metrics and a tight development cycle allows for quickly iterating on distributed systems problems. Obviously the best situation is to have all of the above: unit tests, integration tests, and a tight development cycle in prod.
If I had to pick one because I am time constrained, I would choose testing in prod with good metrics which is maybe what the article is getting at.
You can get quite close to reality with really good integration tests. There's all sorts of real life scenarios ive hit like flaky networks, email appearance in various clients, etc. that can be integration tested with a bit of creativity but most people wouldnt even think to do it.
The investment in this stuff can be quite high...unless you've got premade tooling for all of this that you can just drop in.
Yes, unless you are doing a clean room implementation of paxos or the raft protocol it probably isn’t worth the effort to create harnesses to simulate packet loss, thundering herds, split brain, out of order responses, etc. Even then, if you are writing some distributed synchronization primitives you might be better off with formal proofs than some sort of test harness.
To give an example: You can unit test your OBD response parser all you like, at some point you have to actually get in the car and see what gibberish the OBD adapter spews at your app, what timeouts it needs, how reliable it is, etc
I find having really good metrics and a tight development cycle allows for quickly iterating on distributed systems problems. Obviously the best situation is to have all of the above: unit tests, integration tests, and a tight development cycle in prod.
If I had to pick one because I am time constrained, I would choose testing in prod with good metrics which is maybe what the article is getting at.