Even if there's no money involved that can still damage the market for the work it derives from. It's one thing if the derivative work consists of original characters in the same setting, or existing characters but in a what-if scenario, but as soon as you start trying to write derivative works in the same vein as the original then you're damaging the original author's ability to market his existing work, and to continue that work himself. You are, in a very real sense, taking something that isn't yours.
That's great if you ignore the fact that almost all our ideas are borrowed. Copyrights work against the movement of ideas - the ground culture springs from. It is artificial scarcity, like DRM, unnatural in the domain of information.
> as soon as you start trying to write derivative works in the same vein as the original then you're damaging the original author's ability to market his existing work
That seems ridiculous. How would expanding literature hurt it? It would be like free PR for the original. Art (tales, legends, stories, music and potry) used to be orally transmitted, we forgot that too, with every retelling it was reinterpreted and adapted. Even Tolkien didn't invent the elves and Rowling didn't invent sorcery.