I think misappropriating someone else's work to make a buck is a cultural disaster, but then I work for a living and expect to get paid, so what do I know? Fair Use does exist to provide an avenue for the reuse of the work of others for criticism, parody, and the like., but straight up lifting someone else's work and using it to your own ends, even if noncommercial, is wrong.
Even if there's no money involved that can still damage the market for the work it derives from. It's one thing if the derivative work consists of original characters in the same setting, or existing characters but in a what-if scenario, but as soon as you start trying to write derivative works in the same vein as the original then you're damaging the original author's ability to market his existing work, and to continue that work himself. You are, in a very real sense, taking something that isn't yours.
That's great if you ignore the fact that almost all our ideas are borrowed. Copyrights work against the movement of ideas - the ground culture springs from. It is artificial scarcity, like DRM, unnatural in the domain of information.
> as soon as you start trying to write derivative works in the same vein as the original then you're damaging the original author's ability to market his existing work
That seems ridiculous. How would expanding literature hurt it? It would be like free PR for the original. Art (tales, legends, stories, music and potry) used to be orally transmitted, we forgot that too, with every retelling it was reinterpreted and adapted. Even Tolkien didn't invent the elves and Rowling didn't invent sorcery.
I think putting such constraints on creative work is a cultural disaster.