A lot of the newer local first systems, like Triplit (biased because I work on it), support partial replication so only the requested/queried data is sent and subscribed to on the client.
The other issue of relying on a just the server to build these highly collaborative apps is you can't wait for a roundtrip to the server for each interaction if you want it to feel fast. Sure you can for those rare cases where your user is on a stable Wifi network, on a fast connection, and near their data; however, a lot computing is now on mobile where pings are much much higher than 10ms and on top of that when you have two people collaborating from different regions, someone will be too far away to rely on round trips.
Ultimately, you're going to need a client caching component (at least optimistic mutations) so you can either lean into that or try to create a complicated mess where all of your logic needs to be duplicated in the frontend and backend (potentially in different programming languages!).
The best approach IMO (again biased to what Triplit does) is to have the same query engine on both client and server so the exact query you use to get data from your central server can also be used to resolve from the clients cache.
I've stated before that I think the main thing holding back collaborative text / sequence CRDTs is integration with a production database.
Eg-walker looks interesting because it might lend itself to be integrated into a database because the operations are immutable and only appended. However, to demonstrate the effectiveness of these algorithms library authors (see Yjs, DiamondTypes, etc) build stand-alone data structures (usually specialized search trees) that most databases already provide.
Personally, I've been trying to adapt a Piece Table[1] to be collaborative and stored in Triplit[2] which runs on both client and server and already implements logical clocks but I might see how well I can adapt this algorithm instead!
This seems to be a holy grail, to be honest! Super-simple database representations with barely any processing required on the "write path," instant startup, minimal memory requirements on both server and client without a need for CRDT data structures to be in memory, none of the O(n^2) complexity of OT. In fact, if I'm interpreting it correctly, it should be straightforward to get this working in a serverless environment without any notion of session fixation, nor active documents needing to be kept in memory.
I can see this completely reshaping the landscape of what's possible with collaborative documents!
Egwalker has one other advantage here: the data format will be stable and consistent. With CRDTs, every different crdt algorithm (Yjs, automerge/rga, fugue, etc) actually stores different fields on disk. So if someone figure out a new way to make text editing work better, we need to rip up our file formats and network protocols.
Egwalker just stores the editing events in their original form. (Eg insert “a” at position 100). It uses a crdt implementation in memory to merge concurrent changes (and everyone needs to use the same crdt algorithm for convergence). But the network protocol and file format is stable no matter what algorithm you use.
I’ve loved learning all your detailed info on CRDT work. Thank you for progressing the field!
Since it stores all the editing events, does this mean that the complexity of opening a document is at least O(N) in terms of number of edits? Or are there interim snapshots / merging / and/or intelligent range computations to reduce the number of edits that need to be processed?
You can just store a snapshot on disk (ie, the raw text) and load that directly. You only ever need to look at historical edits when merging concurrent changes into the local document state. (And thats pretty rare in practice).
Even when that happens, the algorithm only needs to look at operations as far back as the most recent "fork point" between the two branches in order to merge. (And we can compute that fork point in O(n) time - where n is the number of events that have happened since then). Its usually very very fast.
In an application like google docs or a wiki, the browser will usually never need to look at any historical changes at all in order to edit a document.
Awesome, I'm been following Seph's work for many years! Always thoughtful and well-executed. Probably the most prolific and insightful engineer in the "collaborative text editing" universe.
I use ShareDB every day, which originated from Seph's excellent work on OT algorithms. Good stuff!
Do you see Ladybird beating the incumbent browsers in any dimension .e.g. performance, usability?, security, etc?
Personally, I much prefer developing for the web than native so if there were APIs exclusive to Ladybird it might create a nice virtuous cycle of developers targeting Ladybird to do new things and users using Ladybird to try those new experiences.
on self-hosting: We're cleaning up the docs on self-hosting to make the configuration clearer, thanks for point that out.
on querying: Yeah we don't have aggregations yet but it's on our roadmap. Don't want to over promise but I think we can make something awesome here by leverage our incremental querying engine. Like consider a data dashboard that needs to be updated every hour; in a traditional system (postgres, mongo, etc) you would need to rerun the query from scratch each time. Our plan is to create something closer to what Materialize does and just process the new data so it's much more efficient and can just update continuously rather than every hour.
re Evolu: I haven't actually gotten a change to try it out but there might be in someone in our Discord[1] that has that could compare/contrast
This is a great question! The short answer is by maintaining backwards compatibility in your schema--it's basically the easiest way to guarantee compatibility. We have warnings in place to let you know when you've made a change that isn't backwards compatible, you can read about it in our docs: https://www.triplit.dev/docs/schemas/updating#pushing-the-sc...
However, overtime this can naturally lead to a mess of a schema definition that has a lot of confusing names. We haven't released a solution to fix this yet but we're working on a few things that should make this less painful. For background on the various approaches, the Cambria doc is an amazing resource: https://www.inkandswitch.com/cambria/
Lenses make total sense. You don't have to maintain lenses for all past versions, just a rolling set with old ones scheduled for deprecation announcements and a sunset schedule.
It's not currently possible to use MongoDB with Triplit, you would use Triplit's server instead but it's very easy to setup with React: https://www.triplit.dev/docs/quick-start
Yeah it's totally usable as is but we're nearly finished with a new system that has more granular controls like delete, preUpdate, postUpdate, insert, and read. We'll still support the existing rules for backwards compatibility.
Yes but the 'backend' (the client, but the non-presentational bit) is Rust; it'd be there you'd want to do this sort of synchronisation or anything to do with state/database. You could absolutely shoe-horn it in, it just wouldn't be the way you'd expect it to work or feel particularly right/supported.
Docs for it: https://www.triplit.dev/docs/frameworks/tanstack-router#exam...
It by default uses IndexedDB but can also use SQLite but does real time, relational querying and (optionally) syncs with you server.
Re: developer ergonomics, this is our primary focus so I don't love to get your feedback on!
reply