How do we set up tests to simulate this? He mentions Emulab testbed but essentially says it is unavailable.
Disclosure: I work for http://gunDB.io/ and we want to start doing tests like this. Willing to pay people who know how to set these things up. Please contact.
Intercepting system calls seems like the easiest way (I expect most of these failures are in I/O). Use LD_PRELOAD to easily hook the libc wrappers, for (almost) anything C-based.
Deeper simulation is possible, but this should get you the low-hanging fruit.
Disclosure: I work for http://gunDB.io/ and we want to start doing tests like this. Willing to pay people who know how to set these things up. Please contact.