One problem is poorly done reviews. Especially since many reviewers are not being paid anything for their time and have the incentive to ward off competing researchers while being protected by anonymity. It seems hopeless since who else would the publisher go to to assess the quality of supposedly new pursuits of expertise knowledge? I've had experienced many strawman rejections back in the day that I just stopped trusting the publishing system.
I'd rather just upload a PDF and let the readers judge its credibility and let (positive) citations be the benchmark instead on trusting a handful of elite critics with unclear incentives.
And, as you say, the expensive publishing houses don't do anything to ensure high quality reviews. In grad school I had a paper rejected by a prestigious outlet because one reviewer thought that the experimental setup I used was too difficult and therefore they didn't trust that I was being honest about my results. They stopped just a step short of calling me a fraud. I was baffled that this was accepted by the publisher.
It was only later when I was tasked with performing reviews that I realized that almost exactly zero work is done by the publishers. It is just "well there's one Strong Reject so see-ya."
I'd rather just upload a PDF and let the readers judge its credibility and let (positive) citations be the benchmark instead on trusting a handful of elite critics with unclear incentives.