Here's a practical solution I have proposed in my community (autonomous robots):
1. package code and data into tarball or VCS repo.;
2. place package on long-lived website;
3. compute SHA1 hash or similar from package (if git is used, this is the revision ID, conveniently);
4. publish the URI and hash in any paper that makes claims based on that code or data;
5. as a reviewer, prefer papers that follow this method, all else being equal;
6. as an editor, suggest that submissions use this method.
(Edit 2: In case it's not obvious, the purpose of the hash is to allow users to be pretty confident that the code they downloaded is indeed exactly the code used in the paper. By putting the hash in the paper, I make this promise. If I want to make an improved version available, I just put it up at the same site, but I must make the exact original available and identifiable as such. This simple method of ensuring identifiability is our contribution.)
My group does this with every paper. I have a paper describing this method coauthored with a student under review now at a good journal, and I'm looking forward to seeing the response. (Edit 1: see link in comment below)
"A few months ago, a graduate student in another country called me (Vaughan) to ask for the source code of one
of my multi-robot simulation experiments. The student had an idea for a modification that she thought would improve the system’s performance ... we were able to offer the requesting student some code that may or may not be that used in the paper. This was better than nothing, but not good enough, and we suspect this is quite typical in our community."
Fwiw I've had a similair experience.
A few years ago I was reading Daphne Koller's "Using learning for approximation in stochastic processes" and it was a very elegant and powerful idea, but it didn't seem to me at the time that there was enough detail in the paper to implement the algorithm and I wrote to Dr Koller and asked if she had source code available. She replied that her student had departed a decade ago and the code was effectively lost.
Dr Koller was very helpful and clarified some doubts I had and in the end I managed to re implement the algorithm and so a happ ending after all but if code was archived and made public for every paper, it would have really helped.
Awesome that you guys are adopting this methodology.
1. package code and data into tarball or VCS repo.; 2. place package on long-lived website; 3. compute SHA1 hash or similar from package (if git is used, this is the revision ID, conveniently); 4. publish the URI and hash in any paper that makes claims based on that code or data;
5. as a reviewer, prefer papers that follow this method, all else being equal; 6. as an editor, suggest that submissions use this method.
(Edit 2: In case it's not obvious, the purpose of the hash is to allow users to be pretty confident that the code they downloaded is indeed exactly the code used in the paper. By putting the hash in the paper, I make this promise. If I want to make an improved version available, I just put it up at the same site, but I must make the exact original available and identifiable as such. This simple method of ensuring identifiability is our contribution.)
My group does this with every paper. I have a paper describing this method coauthored with a student under review now at a good journal, and I'm looking forward to seeing the response. (Edit 1: see link in comment below)
I'd also appreciate feedback from HN.