I'm arguing against the specific point that merely leaking secret keys would give anybody great access.
As far as I understand it the governments are not asking for backdoored transport layer encryption. They want the data one stores or transmits to be ultimately decryptable by them one way or another. Most services that use secure transports merely use it to transmit cleartext and store it as such. With E2EE messaging even if you have the server tapped you'll only capture encrypted binary blobs and perhaps some metadata. That's what the governments don't like.
With backdoored E2EE you'd still need to hack the server AND have the key to access the data.
The problem is that you wouldn't have to 'hack the server', at least for Matrix, given anyone can run a server - so all it would take is for one server in the room to be vulnerable to a social or technical attack (e.g. a nosey sysadmin).
If an escrow public key has been mixed into the e2ee encryption, then all it takes is for the private key to be leaked (e.g. for a price on a dark market) and the nosey sysadmin can go and break the encryption of all its users.
The same goes for a centralised service too (c.f. the twitter hack) - while it might be less likely given the smaller attack envelope, it's a much bigger prize.
As far as I understand it the governments are not asking for backdoored transport layer encryption. They want the data one stores or transmits to be ultimately decryptable by them one way or another. Most services that use secure transports merely use it to transmit cleartext and store it as such. With E2EE messaging even if you have the server tapped you'll only capture encrypted binary blobs and perhaps some metadata. That's what the governments don't like.
With backdoored E2EE you'd still need to hack the server AND have the key to access the data.