Although every scientific primer says that replication of scientific experiments is key, to quote this tweet, you’ll need luck if you wish to replicate experiments in computational science. There has been significant pressure for scientists to make their code open, but this is not enough. Even if I hired the only postdoc who can get the code to work, she might have forgotten the exact details of how an experiment was run. Or she might not know about a critical dependency on an obsolete version of a library. The current state of experimental reproducibility in computer science is lamentable. The result is inevitable: experimental results enter the literature which are just wrong. I don’t mean that the results don’t generalise. I mean that an algorithm which was claimed to do something just does not do that thing: for example, if the original implementation was bugged and was in fact a different algorithm. I suspect this problem is common, and I know for certain that it has happened.
- Computational experiments should be recomputable for all time
- Recomputation of recomputable experiments should be very easy
- It should be easier to make experiments recomputable than not to
- Tools and repositories can help recomputation become standard
- The only way to ensure recomputability is to provide virtual machines
- Runtime performance is a secondary issue