If this happens, there's a lot to be learned from Stack Exchange on how to do this well. The stats.stackexchange.com site is actually pretty high quality IMHO.
Arxiv could use some of that reputation & moderation structure to regulate content. Cranks and time-wasters could be sidelined. It would certainly work much better than a typical newspaper comments section.
It would work by establishing expectations of how much substance should be in a response to a paper. Then getting the community to pitch in on the quality of responses. You still get echo-chamber effects with this, but the quality filter is generally worth it.
I really like the idea of using something similar to the stack overflow model to help with peer review. It could open up the peer review process to more people for a longer period of time, giving a higher chance that more people will contribute if there is interest and a higher overall quality of discussion. All contributions give you a clear increase in reputation points.
Groups of academics could band together to form online-only journals on a specific topic and then allocate medals to papers that are considered accepted. So while there may be many loose papers out there with little or no comments (like on axiv currently) only a few will get the seal of approval.
I would fear fabricated likes and positive comments about "our" papers and fabricated un-likes and negative comments about "their" papers; academic cliques and individual professors can leverage their students to pollute any StackExchange-like mechanism, and if they have too few slaves there are plenty of PR firms to do the job.
FWIW I'd like to see this kind of model join the publishing landscape. It's galling that academics do peer review unpaid for the most part, only for publishers to get in the way of access. Open peer review definitely needs working on.
Arxiv could use some of that reputation & moderation structure to regulate content. Cranks and time-wasters could be sidelined. It would certainly work much better than a typical newspaper comments section.
It would work by establishing expectations of how much substance should be in a response to a paper. Then getting the community to pitch in on the quality of responses. You still get echo-chamber effects with this, but the quality filter is generally worth it.