> reproduce the code -- ideally, from scratch, so that his results are (hopefully) unaffected by authors' bugs
This rests on a common false assumption that programmers make: they think it's easier to write bug-free code when starting from scratch. The reality is that it's almost always easier to start with something that's nearly working and find and fix the bugs.
What really happens when you do a clean room reproduction is that you end up with two buggy programs that have non-overlapping sets of bugs, and you spend most of the effort trying to figure out why they don't match up. It's a dumb way to write software when it can be otherwise avoided.
I wonder though, maybe non-overlapping sets of bugs are actually better for science? That is, it could avoid systematic errors. Of course, one bug free implementation is clearly better!
True, but this is research, not business. Getting 2 programs that don't agree, even when your post-doc has cleaned up all the bugs, is the point of reproducing the research. Ideally, you want to know if you both ran it the same way so you know it's 'true'.
True, but how would one discover those bugs in the first place? In commercial software a client might run into them and report them. But for science papers? Algorithms that go into production might work the same way, but analysis papers such as the OP is talking about don't.
Worse, with the attitude that the OP has, do you really think they will take extra time to verify the entire code base or look for bugs?
Often the best way to find errors in these sorts of analyses is to do a clean room implementation then figure out why the two programs don't produce the same results.
But you're not writing software for production, you're writing software for understanding/advancing research. Every thing that doesn't match up is either a mistake on your end, or an undocumented (and possibly dubious) assumption on their end, and it's really valuable to find out either way.
Reimplementation matter hugely (in ML at least). But that doesn't mean having the original implementation available isn't a huge advantage, obviously it is.
I think there's exceptions to this: most modern attempts to verify software has involved design decisions that are only practical when you're starting from scratch. Similarly, rewriting in a language with more static safety guarantees may lead to fewer bugs (or at least different, less critical ones).
It can also be used to flesh out unconscious biases. I did this at work and this helped to find several issues that might have become bugs in the future.
This rests on a common false assumption that programmers make: they think it's easier to write bug-free code when starting from scratch. The reality is that it's almost always easier to start with something that's nearly working and find and fix the bugs.
What really happens when you do a clean room reproduction is that you end up with two buggy programs that have non-overlapping sets of bugs, and you spend most of the effort trying to figure out why they don't match up. It's a dumb way to write software when it can be otherwise avoided.