A small amount (and I mean small as in anyone can afford it, something like $5 or $10) should be required in escrow to make the report. If the report was valid, there is no contest and the content is taken down, your money is returned.
If the takedown is challenged, and turns out to be false, the money is split between the hosting service and the poster of the content you tried to have removed.
People will try to game the automatic detectors to get false positives (and some $ for themselves) but for some time, this will be a feature, not a bug. They'll get real accurate, real fast, even if it means human review of every takedown.
Except, it's not going to be $100, it's going to be $2000, because the hosting provider has an incentive to report as high a RevenueLossDuringTakedownProcess as they can get away with. A lot of small artists are not going to be able to lock up $2000 at several sites.
If the takedown is challenged, and turns out to be false, the money is split between the hosting service and the poster of the content you tried to have removed.
People will try to game the automatic detectors to get false positives (and some $ for themselves) but for some time, this will be a feature, not a bug. They'll get real accurate, real fast, even if it means human review of every takedown.