As a PhD student in CS, this is an incredibly good idea---even if it stops short of verification and stays somewhere at "reproduce". It can be a lot of work to reproduce a single publication, and often requires very careful reading and attention to detail.
I've done it a few times myself (partially because my work requires it and partially because I wanted to convince myself of the work I was using), and it was an incredibly valuable learning experience.
In the case of CS, replication should become a matter of 'git clone' and 'make'. (Or running a prepared virtual machine, etc.) Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult.
> Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult.
I don't understand what you're trying to achieve by saying this. Did I somehow imply that people should be excused for not making their work reasonably reproducible?
It my sub-field, it is utterly ridiculous to expect to reproduce work by running `git clone && make` or by getting a prepared VM. Computational biologists are not known for their systems or software skills.
Well, the context here is discussion of the problem of unreplicable research. I do get the impression (including from your post and reply) that computational biologists need to get their act together on that front; see http://ivory.idyll.org/blog/ for one researcher in that field who sometimes blogs on this theme.
Indeed, we do need to get our act together. I'm still just starting, but I have a couple of ideas on proactive things I can do, but it's an uphill battle.
Thanks for that link---it's always nice to see other people in the field concerned with this. There aren't enough of them.
I'm not convinced that's "replication" in the sense that's important in science. Bugs in the original test will be reproduced in the attempted reproduction. It's useful for "this person didn't flat out lie about their results", which is probably also a good thing, and for finding issues when replication fails, so I don't disagree that it should happen. I just disagree with calling it "replication".
Fair enough -- maybe we should find a different word. It's a reasonable bare minimum standard in the computing world, where it's practical: it makes it possible to trace back from questions and problems with the claim.
Sure - it's clearly a good idea, where feasible (and git repo + whatever-special-hardware should allow the same thing, for the one reasonable case I can think of where it wouldn't be feasible).
I've done it a few times myself (partially because my work requires it and partially because I wanted to convince myself of the work I was using), and it was an incredibly valuable learning experience.