I'll just pick one. Matrix is resource-heavy, even in the best of times. There is a confluence of smaller contributing factors, and several effects on both its usability and developer/platform friendliness. Consider this anecdote of the overhead of sending the message "hi" as a baseline: in Matrix, this takes about 1KiB on average after all is said and done in JSON (and varies, unfavorably). So to over-simplify, that's 1K to the disk, 1K to each server in the chatroom, for which there could be several hundred, and 1K read on query from clients and other servers -- all for a 2 byte payload.
Before I'm accused of being unfair because there's always going to be a ridiculous ratio for something like a "hi" message, consider some alternatives for both the format and payloads of the fundamental protocol primitives. Most of the messages contain cryptographic metadata which has a succinct binary representation, but JSON requires base64; when represented naturally with CBOR the overhead can be reduced ~40%, and without encoding/decoding either. Consider that virtually all of the overhead in "hi" is either cryptographic hashes or signatures or integers (depth/ts) which would benefit from compact formats.
Does representation actually matter though? Why can't I just store Matrix in my format and federate with your format? I guess this is an example of where theory and reality collide. One cannot write a server which handles messages as abstractly as possible (leveraging these JSON/CBOR extensible formats) while at the same time knowing which fields have a more efficient binary representation and transforming that. In reality that just looks like a CBOR message with a bunch of base64 plaintext strings. It doesn't achieve that 40%.
All of this is important because of Matrix's (superior) design over its event abstraction. When the whole protocol is hinged on fundamental primitives (a good thing) attention to detail and focus must be given to those primitives. The more optimized they are, the more everything built with them is also optimized. When the foundation is efficient, developers can do more with matrix and enrich the user experience. For another example, matrix has been reluctant to give developers the power to store shared-program (i.e bot) state efficiently in a chat room. This is because there's no mechanism to delete state_keys, or even discard overwritten state itself. That's an important cornerstone that's missing while the rest of the tower is being piled on.
It might be possible to confuse any of these issues as trivialities, and their solutions as bike-shedding. I contest their importance is evident in how they emerge to shape the character of the entire system. Consider: does matrix require the entire DAG to be acquired like a bitcoin block-chain? Then I better be careful and conservative about messages. Does matrix allow gaps in the DAG and have a smaller chain for auth? Then I can be liberal about messages and delete stuff later. When I go to build an application which communicates over Matrix those qualities emerge as fundamental limitations or liberations of what users can experience.
So i think the summary is that the SS API needs to support alternative encodings (and thus signings) than JSON. Totally agreed; as you know we've already been experimenting with compact representations including CBOR (https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barr... etc) which applied both to CS & SS API, which took the ratio down massively. However, we fudged it there by disabling the hashes entirely (given it was on a trusted private network). So from my pov Matrix is not remotely tied to the crappy HTTPS+JSON transport for S2S API in the long term, and it's only a matter of time before someone concretely proposes something better.
I selected some examples because I don't want to be lost in abstraction, but please, I am asking kindly that you not focus on my examples (unless it's pressing) because the issue here is systemic. I'm not giving these examples for you to just retreat back with your head down and address them. The locus is more fundamental: the etiology for why these errors even exist and why there continue to be new ones all the time; all at great cost, perhaps even existential cost. You should be concerned with that bigger picture, surely.
Scores of other issues could have been cited, but I included the representation example having confidence you'd probably tout those exact experiments and respond with that blog post. Your conclusion at the time was that CBOR and JSON had negligible differences. The company then proceeded to de-prioritize any serious work in that direction while simultaneously ramping up efforts for end-to-end encryption by default. This was a flawed conclusion and it has demonstrably added to your tower of debt. Even my "hi" example is already passé because now that too is another giant base64 string adding to this jumble. You can't bolt Ferrari body panels on a Honda and call it a Ferrari.
I can spend all day citing issues just as poignant, and you might even be kind enough to elevate them against the ~10,000 other [github] issues they're contending with. I can spend all day unpacking the string of missteps leading to all of your engineering disasters. It's not going to have any returns- not for either of us. It doesn't root the problems.
If I could impress upon you some serious advice, it would be to value people less transactionally. Don't estimate a person's worth by how they fill a few open gaps in your grand master plan and then dispense with them as if they won't be filling in any more. Avoid that by valuing people by how they think, and then don't try to think like them yourself. Delegate to them instead.
i think the tl;dr here is that we have a bunch of tech debt, and we’re not working on the things that you think we should be working on, and that we undervalue your input and should delegate to you rather than discussing specifics.
i’m afraid that nobody is going to delegate to you, but you are welcome to contribute via concrete spec proposals, which are assessed based on merits (and whose review is far from dominated by me; if they were my own proposals might move along faster than they do). if the spec process is too slow and doesn’t work for you, then feel free to spell out the proposal however you like.
Who am I to submit a proposal to fix fundamental problems in your company? That's called consulting. You just raised an 8 million dollar Series A you are soliciting me to work for you for free, in the leaf comments of a Hacker News thread? Are you for real?
I volunteered at least three solid examples of fundamental indecisions that have unambiguously cost your investors alone millions of dollars over several funding rounds; lest I mention also cost every poor soul that has volunteered their free time in the name of FOSS -- an ethically ambiguous embrace by you at best; every one of them is locked into the cost of your missteps.
I have clearly linked these indecisions to real-world outcomes. It's all history by now. The foundations of state resolution weren't in place when you built Communities/Groups so you didn't use the matrix chatroom; you built an entirely new inferior HTTP protocol without any state or replication whatsoever over several months. Now that you finally prioritized fixing that underlying problem with rooms there's a need to replace and spec an actual Groups system. In the meantime nobody can develop Matrix Groups into their software, not my server, not clients like Ditto, etc, because you tell everyone not to. You're going to overhaul the whole thing. You're doing them a favor by saving them that wasted effort. But Synapse and Riot use Groups and Matrix promotes it on matrix.org; still there's been no serious visible progress toward groups-as-rooms.
This is why Matrix is indeed a de facto centralized system. Let's not even get into issues in this system where every server trusts matrix.org for cryptographic keys. Matrix is really centralized around you.
In the end nobody is working with you. I have the only third-party implementation of this stack. Nobody is working with me either, but that's called the transitive property.
You have some time to think about this. I would say until the next recession hits. Then this is over. The way you do things won't survive that.
Before I'm accused of being unfair because there's always going to be a ridiculous ratio for something like a "hi" message, consider some alternatives for both the format and payloads of the fundamental protocol primitives. Most of the messages contain cryptographic metadata which has a succinct binary representation, but JSON requires base64; when represented naturally with CBOR the overhead can be reduced ~40%, and without encoding/decoding either. Consider that virtually all of the overhead in "hi" is either cryptographic hashes or signatures or integers (depth/ts) which would benefit from compact formats.
Does representation actually matter though? Why can't I just store Matrix in my format and federate with your format? I guess this is an example of where theory and reality collide. One cannot write a server which handles messages as abstractly as possible (leveraging these JSON/CBOR extensible formats) while at the same time knowing which fields have a more efficient binary representation and transforming that. In reality that just looks like a CBOR message with a bunch of base64 plaintext strings. It doesn't achieve that 40%.
All of this is important because of Matrix's (superior) design over its event abstraction. When the whole protocol is hinged on fundamental primitives (a good thing) attention to detail and focus must be given to those primitives. The more optimized they are, the more everything built with them is also optimized. When the foundation is efficient, developers can do more with matrix and enrich the user experience. For another example, matrix has been reluctant to give developers the power to store shared-program (i.e bot) state efficiently in a chat room. This is because there's no mechanism to delete state_keys, or even discard overwritten state itself. That's an important cornerstone that's missing while the rest of the tower is being piled on.
It might be possible to confuse any of these issues as trivialities, and their solutions as bike-shedding. I contest their importance is evident in how they emerge to shape the character of the entire system. Consider: does matrix require the entire DAG to be acquired like a bitcoin block-chain? Then I better be careful and conservative about messages. Does matrix allow gaps in the DAG and have a smaller chain for auth? Then I can be liberal about messages and delete stuff later. When I go to build an application which communicates over Matrix those qualities emerge as fundamental limitations or liberations of what users can experience.