I certainly agree with you that there is a lot of publication of results generated with computers that is non-reproducible. However, I'm not sure publishing the complete codes that were used is necessary, as long as there is a detailed description of the algorithm used. In essence, I think it's essential that multiple people write their own codes and reproduce the result. The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.
The PYTHIA example someone posted is a good one: It's my understanding that basically everyone in high-energy physics uses it, and while that means it gets a lot of testing I'm not sure that many actually bother to look through the code in detail, because "it's what everybody uses, so if it had a bug it would have been found already".
The situation is similar, but not quite as bad, in astrophysical hydrodynamics simulations. Basically the entire field uses 6 different codes, but they do work on different principles, and the teams do conduct controlled tests of the agreement between them on specific problems.
About the chaos problem, I suspect that the problem is worse. My impression is that often there is not even a basic test that the results are numerically converged, regardless of esoteric strange attractors.
I'm curious, can you mention any specific cases that you are worried about wrt the chaotic behavior.
The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.
Having a small number of codes seems like an excellent situation. What you should have is large of tests and much test data to test the small number of codes.
Testing code in thorough manner is an important addition to looking at it.
The PYTHIA example someone posted is a good one: It's my understanding that basically everyone in high-energy physics uses it, and while that means it gets a lot of testing I'm not sure that many actually bother to look through the code in detail, because "it's what everybody uses, so if it had a bug it would have been found already".
The situation is similar, but not quite as bad, in astrophysical hydrodynamics simulations. Basically the entire field uses 6 different codes, but they do work on different principles, and the teams do conduct controlled tests of the agreement between them on specific problems.
About the chaos problem, I suspect that the problem is worse. My impression is that often there is not even a basic test that the results are numerically converged, regardless of esoteric strange attractors.
I'm curious, can you mention any specific cases that you are worried about wrt the chaotic behavior.