My experience is that this is difficult without the system under test being designed to operate deterministically.
So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both.
Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically.
A while ago I built a simulator testing harness for an early version of the Knative autoscaler[0]. I eventually ran aground on the difficulty of keeping up with changes, but it may be rearchitected and repurposed to assist Kubernetes HPA development instead.
FoundationDB's approach is to pretty much bake optional determinism into the language with some macro magic. It makes me wish that were easier to do in other languages.
I used to have a similar opinion, that it requires a special language to leverage such power. But recently I've discovered that Rust, with its latest advancement in async/await design together with swappable runtime, can provide a decent story for building determinism tests like described in the video. Yes of course one still needs to mock all the IO as well as clock part, but the deterministic core is already there, and with a carefully designed runtime and surrounding library, we might have pretty good support in Rust for this.
Disclaimer: I'm never a Rust zealot, I do understand Rust has its tradeoffs, and it's not the panacea for every problem. In fact I do have side projects which are perfectly suited and written in Go. I'm just saying Rust's design turns out to be suitable for such a deterministic testing structure, and Rust's target for system programming, can also benefit A LOT from this style of testing.
And also a shameless plug: I do have some initial work exploring this area: https://github.com/xxuejie/diviner. It's still quite rough and a lot of work is needed but I do believe this is something that is worth exploring.
So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both.
Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically.
A while ago I built a simulator testing harness for an early version of the Knative autoscaler[0]. I eventually ran aground on the difficulty of keeping up with changes, but it may be rearchitected and repurposed to assist Kubernetes HPA development instead.
FoundationDB's approach is to pretty much bake optional determinism into the language with some macro magic. It makes me wish that were easier to do in other languages.
[0] https://github.com/pivotal/skenario