> Ultimately, the question trying to be answered by the authors is not "does x code, when applied to y dataset, create z outcome?" but rather "how is z outcome created?"
Maybe, but the question asked by the people in the field, and even general public, is different: we're asking, "is this even real, or are they exaggerating or plain bullshitting us?". It's a reasonable question to ask when the results seem groundbreaking, and there's lots of money or reputation at stake.
Consider if they came out and said, we've invented machine converting electricity to mechanical motion with near-zero energy loss at room temperature. The design is complex, and uses exotic materials no one outside few megacorps could reasonably afford to get, and we just plan to use it to sell mechanical work as a service, so you're never gonna see how the thing is built (unless you're really important and sign an NDA) - but trust us, it works. Wouldn't some questions about reproducibility be warranted in such a case?
Maybe, but the question asked by the people in the field, and even general public, is different: we're asking, "is this even real, or are they exaggerating or plain bullshitting us?". It's a reasonable question to ask when the results seem groundbreaking, and there's lots of money or reputation at stake.
Consider if they came out and said, we've invented machine converting electricity to mechanical motion with near-zero energy loss at room temperature. The design is complex, and uses exotic materials no one outside few megacorps could reasonably afford to get, and we just plan to use it to sell mechanical work as a service, so you're never gonna see how the thing is built (unless you're really important and sign an NDA) - but trust us, it works. Wouldn't some questions about reproducibility be warranted in such a case?