(Breaking out this sentence into blocks because it's very large.)
The specification as-written is really, really bad (though I don't think this is a fault of the protocol),
Synapse is kind of...and work on Dendrite seems to have mostly stalled,
the billion people signed up on matrix.org has allowed for a sort of de facto centralization (not a huge problem, though in practice it sort of is, given how much the server buckles under the weight of all of the users; might be solved if it's ever moved onto a more performant implementation?),
the lack of encryption by default is iffy (though done for understandable reasons).
To lend credence to the specification being bad:
"I tried writing a matrix homeserver but I couldn't figure out how the state resolution process works" said by the author of Audacious/one of the authors of Pleroma/much more, so certainly a person who should be able to understand something that simple:
I know multiple people who've tried to implement the spec but who have gotten stuck on things that should be simple but are unclear if going by the spec.
> The specification as-written is really, really bad
We get a pretty wide range of feedback on the spec fwiw: some people seem to really like it. Others say that it’s “really, really bad” which doesn’t exactly give us much to go on...
> Synapse is kind of... and Dendrite is stalled
We have no choice but improve Synapse currently, and while it is still quite a resource hog it’s improved by at least 3-5x over the last year. Dendrite instead has become more of an R&D project for future homeserver shapes, but it’s not entirely stalled.
> The matrix.org server causes de-facto centralisation
The server has been less than 50% of the visible network for several years now - and ironically the datacenter perf issues (unrelated to Synapse) we had over the last few months have shifted that balance further - it’s about 35% and dropping. Ideally we will turn it off entirely once we have decentralised accounts.
> lack of e2ee by default is iffy
Our main project right now is to fix this. Cross-signing is mid flight; E2E search is done; Pantalaimon (E2E compat for dumb clients) is done; remaining key distribution bugs are in flight. We’re aiming to turn it in by default in Jan.
> “I couldn’t figure out how the state resolution works”
State resolution is the main technical novelty in Matrix, and yeah - it’s hard, much like git’s merge resolution is not exactly easy either. Unfortunately it comes with the territory; if you want to have consistent room state while replicating them over a byzantine network of servers to stop hijacks and other abuse, you have a relatively hard problem to solve. We got it wrong the first time; the current version gets it right (as far as we know).
It looks like you guys are on the right path for most things, and I wish you luck!
Why do you have no choice but improve Synapse?
One of the reasons you seem to only get negative opinions along the lines of "really, really bad" is because "The way it's written is stylistically bad/unapproachable and an entirely different approach should probably be taken" is something that takes a few hours to diffuse into something useful, and not something most people really receive in a way that's productive. It takes a lot more effort to give truly constructive, workable criticism than expressing any sort of opinion, negative or positive.
> It looks like you guys are on the right path for most things, and I wish you luck!
thanks :)
> Why do you have no choice but improve Synapse?
There are quite a lot of Synapses out there right now, and Synapse's featureset is pretty mature. Given we have pretty high profile people running Synapse (e.g. all of the French government) we have no choice but to invest the time to polish it and make it more efficient and stable. In an ideal world we'd have paused on Synapse ages ago and focused purely on a next-gen server, but we can't leave Synapse users in the lurch. Dendrite will be getting some attention in the coming months though, but not as a direct Synapse-replacement, but more of a playground for more experimental stuff.
> It takes a lot more effort to give truly constructive, workable criticism than expressing any sort of opinion, negative or positive.
True. fwiw, my guess is that kaniini was trying to implement state resolution before we published the first stable Server-Server API back in Feb - where it most certainly was the weakest part of the spec (and had design flaws too, which is why we hadn't bothered speccing it in detail).
Others have better responses to the rest of these points; many of them are still significant issues though a lot has progressed over the last couple years. I can preempt one though by saying alternatives to Synapse like https://github.com/matrix-construct/construct are probably something more aligned with your interests. (disclosure: I started the project with kaniini).
After more than two years in and around this space I'm still comfortable with stating that I'd take Matrix with all^wmost its flaws over ActivityPub. All of these protocols kind of suck. I see Matrix as having the lowest hamming distance between where it is and where it should be.
> I'd take Matrix with all^wmost its flaws over ActivityPub
protocol-wise, which flaws are currently the most egregious from your pov? (i.e. `all - most = ?`) (And if any are security issues, please chuck them to security@)
I'll just pick one. Matrix is resource-heavy, even in the best of times. There is a confluence of smaller contributing factors, and several effects on both its usability and developer/platform friendliness. Consider this anecdote of the overhead of sending the message "hi" as a baseline: in Matrix, this takes about 1KiB on average after all is said and done in JSON (and varies, unfavorably). So to over-simplify, that's 1K to the disk, 1K to each server in the chatroom, for which there could be several hundred, and 1K read on query from clients and other servers -- all for a 2 byte payload.
Before I'm accused of being unfair because there's always going to be a ridiculous ratio for something like a "hi" message, consider some alternatives for both the format and payloads of the fundamental protocol primitives. Most of the messages contain cryptographic metadata which has a succinct binary representation, but JSON requires base64; when represented naturally with CBOR the overhead can be reduced ~40%, and without encoding/decoding either. Consider that virtually all of the overhead in "hi" is either cryptographic hashes or signatures or integers (depth/ts) which would benefit from compact formats.
Does representation actually matter though? Why can't I just store Matrix in my format and federate with your format? I guess this is an example of where theory and reality collide. One cannot write a server which handles messages as abstractly as possible (leveraging these JSON/CBOR extensible formats) while at the same time knowing which fields have a more efficient binary representation and transforming that. In reality that just looks like a CBOR message with a bunch of base64 plaintext strings. It doesn't achieve that 40%.
All of this is important because of Matrix's (superior) design over its event abstraction. When the whole protocol is hinged on fundamental primitives (a good thing) attention to detail and focus must be given to those primitives. The more optimized they are, the more everything built with them is also optimized. When the foundation is efficient, developers can do more with matrix and enrich the user experience. For another example, matrix has been reluctant to give developers the power to store shared-program (i.e bot) state efficiently in a chat room. This is because there's no mechanism to delete state_keys, or even discard overwritten state itself. That's an important cornerstone that's missing while the rest of the tower is being piled on.
It might be possible to confuse any of these issues as trivialities, and their solutions as bike-shedding. I contest their importance is evident in how they emerge to shape the character of the entire system. Consider: does matrix require the entire DAG to be acquired like a bitcoin block-chain? Then I better be careful and conservative about messages. Does matrix allow gaps in the DAG and have a smaller chain for auth? Then I can be liberal about messages and delete stuff later. When I go to build an application which communicates over Matrix those qualities emerge as fundamental limitations or liberations of what users can experience.
So i think the summary is that the SS API needs to support alternative encodings (and thus signings) than JSON. Totally agreed; as you know we've already been experimenting with compact representations including CBOR (https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barr... etc) which applied both to CS & SS API, which took the ratio down massively. However, we fudged it there by disabling the hashes entirely (given it was on a trusted private network). So from my pov Matrix is not remotely tied to the crappy HTTPS+JSON transport for S2S API in the long term, and it's only a matter of time before someone concretely proposes something better.
I selected some examples because I don't want to be lost in abstraction, but please, I am asking kindly that you not focus on my examples (unless it's pressing) because the issue here is systemic. I'm not giving these examples for you to just retreat back with your head down and address them. The locus is more fundamental: the etiology for why these errors even exist and why there continue to be new ones all the time; all at great cost, perhaps even existential cost. You should be concerned with that bigger picture, surely.
Scores of other issues could have been cited, but I included the representation example having confidence you'd probably tout those exact experiments and respond with that blog post. Your conclusion at the time was that CBOR and JSON had negligible differences. The company then proceeded to de-prioritize any serious work in that direction while simultaneously ramping up efforts for end-to-end encryption by default. This was a flawed conclusion and it has demonstrably added to your tower of debt. Even my "hi" example is already passé because now that too is another giant base64 string adding to this jumble. You can't bolt Ferrari body panels on a Honda and call it a Ferrari.
I can spend all day citing issues just as poignant, and you might even be kind enough to elevate them against the ~10,000 other [github] issues they're contending with. I can spend all day unpacking the string of missteps leading to all of your engineering disasters. It's not going to have any returns- not for either of us. It doesn't root the problems.
If I could impress upon you some serious advice, it would be to value people less transactionally. Don't estimate a person's worth by how they fill a few open gaps in your grand master plan and then dispense with them as if they won't be filling in any more. Avoid that by valuing people by how they think, and then don't try to think like them yourself. Delegate to them instead.
i think the tl;dr here is that we have a bunch of tech debt, and we’re not working on the things that you think we should be working on, and that we undervalue your input and should delegate to you rather than discussing specifics.
i’m afraid that nobody is going to delegate to you, but you are welcome to contribute via concrete spec proposals, which are assessed based on merits (and whose review is far from dominated by me; if they were my own proposals might move along faster than they do). if the spec process is too slow and doesn’t work for you, then feel free to spell out the proposal however you like.
Who am I to submit a proposal to fix fundamental problems in your company? That's called consulting. You just raised an 8 million dollar Series A you are soliciting me to work for you for free, in the leaf comments of a Hacker News thread? Are you for real?
I volunteered at least three solid examples of fundamental indecisions that have unambiguously cost your investors alone millions of dollars over several funding rounds; lest I mention also cost every poor soul that has volunteered their free time in the name of FOSS -- an ethically ambiguous embrace by you at best; every one of them is locked into the cost of your missteps.
I have clearly linked these indecisions to real-world outcomes. It's all history by now. The foundations of state resolution weren't in place when you built Communities/Groups so you didn't use the matrix chatroom; you built an entirely new inferior HTTP protocol without any state or replication whatsoever over several months. Now that you finally prioritized fixing that underlying problem with rooms there's a need to replace and spec an actual Groups system. In the meantime nobody can develop Matrix Groups into their software, not my server, not clients like Ditto, etc, because you tell everyone not to. You're going to overhaul the whole thing. You're doing them a favor by saving them that wasted effort. But Synapse and Riot use Groups and Matrix promotes it on matrix.org; still there's been no serious visible progress toward groups-as-rooms.
This is why Matrix is indeed a de facto centralized system. Let's not even get into issues in this system where every server trusts matrix.org for cryptographic keys. Matrix is really centralized around you.
In the end nobody is working with you. I have the only third-party implementation of this stack. Nobody is working with me either, but that's called the transitive property.
You have some time to think about this. I would say until the next recession hits. Then this is over. The way you do things won't survive that.
