A number of years back, I wrote an OT implementation in Clojure/ClojureScript as a hobby project [1] and learned a lot along the way. I even extracting out the core client/server implementations to their own package [2].
It's an incredibly fun topic, and one that's also really challenging. It also gave me first-hand experience with fuzz-testing where I could simulate all kinds of concurrent-edit conflicts and ensure both clients came out with the same end-result. While the end client/server implementations are working, it was a lot more effort than I anticipated building a full-fledged editor on top. I was too stubborn at the time to attempt to integrate with existing RTEs though, so the most notable part of the project was the core lib, not the end product.
For those interested in implementing it for themselves, I can't recommend this piece [3] by @djspiewak enough.
I found this recent paper highly enlightening: “Real Differences between OT and CRDT for Co-Editors”–
Chengzheng Sun, David Sun, Agustina, Weiwei Cai, October 2018.
It’s well written imo. The conclusions chapter is very good.
“In this work, we have critically reviewed and evaluated representative OT and CRDT solutions, with respect to correctness, time and space complexity, simplicity, applicability in real world co- editors, and suitability in peer-to-peer co-editing. The evidences and evaluation results from this work disprove superiority claims made by CRDT over OT on all accounts.”
And it is also argued that OT is a more direct approach to the fundamental issues of realtime multiuser editing, and that that directness brings effectiveness.
Thanks for the mention. We've re-organized the material and updated the submissions by splitting into three parts. We wanted to extend the content but felt that in its original form the article was too dense. The new structure consists of three related but more self-contained articles to help articulate the key differences:
I still need to read that one, though it looks interesting (yes, I only got through the introduction and summary :)). Let's keep in mind that the authors already wrote several papers on OT (also ones that we based CKE5 on) -- so they have vast knowledge in the subject but may also favor one technology over the other. As much as I'd like to read that OT was the good choice I also want to be cautious in this regard :).
Do you have any idea which would be better for a vector based drawing app? The data is mostly strokes (lists of points), and individual strokes generally don't change.
"After over a decade, however, CRDT solutions are rarely found in working co- editors or industry co-editing products, but OT solutions remain the choice for building the vast majority of co-editors. In addition, the scope of nearly all CRDT solutions for co-editing are confined to resolving issues in plain-text editing, while the scope of OT has been extended from plain-text editing to rich text word processors, and 3D digital media designs, etc."
I'm not sure what to make of that, though, as there seem to be quite a few different JSON CRDT libraries, for example.
Just after that it says "The contradictions between these realities and CRDT’s purported advantages have been the
source of much debate and confusion in co-editing research and developer communities." That's kind of my question/confusion: people writing CRDTs really talk them up, and I can't tell what's true in practice. I have no way to judge the accuracy of this paper myself (I don't know enough to really understand it), so I'm hoping someone who knows this stuff in a practical way can shed some light on it.
I found that I could eventually power through the paper. I thought it was beyond me at first. Then I got angry and just shoved piece by piece into my head :) Not in linear fashion, and it didn’t happen quickly. Actully not that slow either once I got going. I found I was maybe rather more of a paper-reader than I assumed I was. Probably you too!!
Surprised the author didn't mention Quill.js, which works really well with ShareDB, and is fully open source. We've been doing a lot of fun stuff with it - here's a talk I gave recently as a job talk: https://www.youtube.com/watch?v=gN37rJRmISQ. (It worked, I got hired :)). As a side project, I'm working on a ShareDB backed wiki, where we can use rich text for editing pages, but also other components, like spreadsheets or mindmaps, and everything supports live editing - I think it would be amazing for classrooms, hackathons, any kind of meeting etc. Would love to talk with anyone interested in collaborating. I have been waiting for a few features to complete to make a proper demo video, but this early one shows off some of the ideas: https://www.youtube.com/watch?v=9-lU-in3ydc (https://github.com/chili-epfl/FROG)
Thanks for the shoutout! We are actively developing on ShareDB (https://github.com/share/sharedb) and if anyone is really interested in this, please reach out to me (https://github.com/nateps). Also, Lever is looking to hire someone to work full time on our internal + open source frameworks including ShareDB.
Thanks for ShareDB! Funny thing, I was looking for a self-hosted real-time collaborative editor myself just a week ago, and landed exactly on Quill.js backed by ShareDB. (I also considered SharedDB-backed Monaco, but it doesn’t seem to support OT out of the box, whereas Quill is literally plug and play. Of course one can write a translation layer between Monaco’s change events and OT.)
I'm just curious as it doesn't mention anything about it being a wiki in the readme (it's also a bit unclear how/what to use it for from the description).
Yeah, we need to do a lot around messaging, to be clear - the wiki is in an early prototype and there are known bugs, etc. However if you want to try it out, install the repo (run initial_setup, then npm start server), then go to localhost:3000/wiki/ANYTHING. It will create a new wiki named ANYTHING?login=YOURNAME, and there you are.
What many people not fully grasp, at first sight, is that offline support for a collaborative editor is at least an order of magnitude more complex than just building a collaborative editor. In online collaborative environments, solving conflicts is relatively simple, as the changes are supposed to be small. But when two (or more) people are working on the same document without synching it for hours or days, the merge strategy has to be extremely good to come up with anything remotely intended (assuming the authors had a common intention).
So just because your favorite editor can be used offline and it has an (online) collaboration feature, does not mean it is capable of being an offline-capable collaborative editor.
This is where differential synchronization shines (basically git). While OT/CRDT are really good for syncing small edits, they suck at being eventually consistent (with intentions preserved) on very large edits.
We make a powerful collaborative word processor for the browser (Zoho Writer https://writer.zoho.com) with complete offline support. This is one problem we'd like to solve in the long term. One way to solve this is to fallback to differential sync when syncing huge offline edits - and present the changes as tracked-changes (underlined and striked-out) to the user. This way the user has the power to resolve the conflicts, instead of the app messing it up by itself.
I'd like to hear if there are better ways to sync large offline edits, with the intentions preserved.
> One way to solve this is to fallback to differential sync when syncing huge offline edits
This is another question -- does automatic syncing make sense for huge differences in users' contents? The answer might be no.
OTOH, isn't this a rare scenario? How perfect it has to be?How much of a compromise can the users accept? These are interesting questions which shows that creating this kind of software isn't easy -- both from technical and UX point of view.
First of all when discussing about offline edits we should make effort to draw attention that "offline" could mean different thing to different people.
Offline editor from the point of view of the app's software developer might mean that editor can sometimes survive few minutes of lost connectivity, after which it will reconnect and possibly sync the team work.
Offline from the perspective of a team of scientists writing a joint paper means that some of them will take their work truly offline, to a secluded mountain hut, for an extended period (week or month) and will come back with a complete rewrite of the text to the point of being unrecognizable to the other authors. And then the coauthors will disagree with most of it, and will want to revert some of the pieces and keep some other pieces.
And scientists are not the only people needing joint collaborative environment. And not just for short papers. Few years ago, I was working in a larger team with a group of people doing revisions of a book, and there were so many revisions and revisions of revisions of revisions, and zillions of comments, that we had to split the book not in chapters, since Word got stuck even with 20-page chapters. We had to split the chapters in sub-chapters, of 5-10 pages in separate files, so that the processor would not get stuck with the myriad of comments and revisions.
These are all separate scenarios in collaborative writing, and there could be many other scenarios, so any solution should first explain for what type of scenario is it really targeting. Automated sync of collaborative work is not always the best thing to do.
Well it's quite common for people using “normal” email-exchanged MS office revisions. I have no clue if there's specific tooling for these cases. Of course you could brush them off and say they wouldn't be a problem if they just used online editors.
I work with a tool that provides diffing and merging for rich text (effectively a limited subset of HTML). It doesn’t have or need real-time collaborative functionality at all. I have read all the papers that I can find and understand, but I’m still not happy with our merging algorithm for rich text.
The issue is less about which fancy algorithm/data structure to use and more about even defining in human terms what would be expected in certain merge conflicts. Currently, we err on the side of raising a merge conflict rather than deciding to use one version for a certain change, but in practice most conflicts have a pretty obvious resolution when a human looks at the two conflicting versions.
> The issue is less about which fancy algorithm/data structure to use and more about even defining in human terms what would be expected in certain merge conflicts
Exactly. When it comes to rich text editing which can have semantic trees (e.g tables and tables within tables), merge conflicts are so tough to handle.
Consider this: one version deleting a column in a table, and the other version splitting the column and adding a new row - by this time it's sort of impossible to find a meaningful representation of the table without manual intervention.
> sync large offline edits, with the intentions preserved
How would that be possible if none of the users can see what the others are doing? If you have three users that start with version 1 and they all make their own version 2, even the users don't know what the "correct" version 2 should look like. Or to put it differently, I don't even know what is meant by "intentions" in this scenario.
Of course, it depends on the collaborative editing solution. When it comes to OT -- if the implementation is correct, it doesn't really matter how many operations are queued for synchronising. So, it doesn't really matter how long users were de-synced when creating their content. But this is only theory and it applies more to technical side as there might be some semantical problems (intention preservation).
I believe the same is true for CRDT but I am not sure.
So, on one hand - if the editor is working fine for small batch of changes, it should also work fine for big batch of changes. The reality is often more harsh, though, and full of edge case scenarios.
OT is very simple to implement, but needs a server, although a simple one, that have a counter, and attach the count to each message when sending it to the clients. That way clients can know how to transform the message, and which message came first. CRDT however is more complex, but does not require a server. It can also tolerate some "offline" time. I do not however think CRDT can be used in order to get conflict free merging, I think there always need to be a manual merge resolution. But I would love to be proven wrong! Maybe machine learning and live training could be used to make better merge suggestions. If you solve the merge problem that would also make software version control merging easier! Which is a huge market. And there are probably a ton of other use cases too, besides just collaborate editing.
> OT is very simple to implement, but needs a server (...)
Well, OT in general does not need the server but server-less implementations are more complex (more transforming functions to write, except of "inclusion transformation" you also have to write "exclusion transformation" algorithms).
I also wouldn't say that "OT is very simple to implement" - it is in it's base form, for linear data, with the server in the network. But every enhancement brings a lot of complexity.
On a work "test" I was asked to write a diff function (least amount of transformation to get from one state to another state) from scratch, even though I've read many diff-algorithm-papers in my life I couldn't actually implement one (even though I got 99/100 tests right, they failed me with the feedback that I need to work on my algorithms ... ). But after getting the concept of OT it was straight forward to implement it from scratch. That's my relativity, maybe someone else think it's easier to implement a diff algorithm.
If two clients, your client (a) and another client (b), writes the letters a and b respectively at the same time, from either client's perspective they where first, so on client A the state will be "ab" and on client B the state will be "ba". How do you solve that without a server/master ? With a server/master the server just have to increment a counter for each operation, and the clients can use that counter to know which order. So if a has counter 77, and b has counter 76, the state will be "ba".
Indeed, the space is growing quickly. I run a company that provides a backend for real-time collaboration, so we necessarily stay on top of what's out there. There's a reason the server-side tech tends be paid: it is extraordinarily difficult to provide guaranteed eventual consistency of data at low latency. The CKEditor guys (and us, for that matter) have put YEARS of development effort into their solutions.
We're working on both offline mode and a generic rich-text data model with support for the major editors.
If anyone's interested, I'd happily do some Q and A here.
I appreciate that you mentioned a competitor without shamelessly plugging your own product, but out of curiosity, and if you don't mind sharing, what editor do you work on?
You can currently already do good-enough-for-most DOM-level synchronization using any editor, of which we have a few examples. A lot of the examples of RTC out there (apps using TogetherJS for instance) are doing this already. For rich text, we'd like to be able to provide 100% support for a particular editor's capabilities, which necessitates a deeper, custom data model.
One of our value propositions is a unified backend for dealing with any JSON data (and eventually beyond), avoiding binding server-side code to a particular choice of UI component.
I am curious about this argument - I think CKEditor 5 made a similar one in this epic blog post about how they implemented real-time collaboration (https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri...). We're using Quill.js with ShareDB, which supports JSON structures (which is great, because for us we often have documents with several rich text fields, and other complex structures). So far we've been able to do anything we wanted with Quill, and I've never felt limited by the data structures we have available... (We also do all kinds of other stuff with ShareDB JSON).
I guess one reason you could need custom types would be to ensure consistency - if two keys depend on each other, and one user sets one key, and the other user sets another key, and the document is now invalid, you'd need the engine to be able to reconcile at a higher level?
> (...) if two keys depend on each other, and one user sets one key, and the other user sets another key, and the document is now invalid, you'd need the engine to be able to reconcile at a higher level?
I am not sure if I understand you correctly here, but it's not really that. Could you give me a more concrete example?
The kind of problems for extra types are, for example: user A changes a paragraph to a list item and user B splits it. As a result you'd like to have two list items instead of a list item followed by a paragraph. This is impossible if you don't give more semantic meaning to the operations.
There are other problems though, as you mentioned - with invalid document. For example, you have this kind of a list:
* Foo
__* Bar
__* Baz
User A outdents "Bar" and user B indents "Baz" creating a list like this:
* Foo
* Bar
____* Baz
In CKE5 this is an incorrect list (we don't allow indent differences bigger than one). This cannot be fixed through OT so we fix it in post-fixers which are fired after all the changes are applied.
These content-preservation edge cases weren't possible to solve with what was available (at least at the time we started the project).
Even apart of that, ottypes/json0 was lacking some basic things, like moving objects. I see they came up with a new implementation recently (https://github.com/ottypes/json1) and it allows moving objects. Maybe the new implementation would solve some problems. However, it is in "preview" state, and the last update was 2 months ago, so I am not sure how well it will be maintained.
Also, there are some edge cases when transforming ranges (which CKE5 use to represent, for example, comments on text or content created in track changes mode). I don't want to bury you in difficult to understand examples but if you are interested you might want to check the examples listed in inline codes for this function: https://github.com/ckeditor/ckeditor5-engine/blob/master/src....
As far as Quill.js is concerned, it is based on the linear data model, which brings limitations when it comes to complex features. Transformation algorithms for linear data models are much simpler and there are more implementations and articles in this area. Everything depends on your needs. If Quill.js features set and functionality fit your needs then the solution you chose is correct.
With CKE5 however, we didn't want to go on any compromises. We needed complex structures for our features, and for having a powerful framework - to enable other developers to write whatever feature they want and have those features working in real-time collaboration. We wanted transformation algorithms which will handle all the edge cases. It is true, some of those cases are quite rare. And the old "10/90" mantra applies here, in this case "10% of use cases brings 90% of complexity". But those edge cases happen and we didn't want to disappoint our users.
I think the argument is more about the historical data structures that were used in rich text. A lot of editors either used the DOM, or a very flattened data structure like Google Wave, Quill.js, DraftJS etc. With these flattened data structures it becomes harder to represent complex rich text with things like tables, nested blocks, etc. If you have a nice JSON data structure that is collaborative you can do a lot, and in many / most use cases it is sufficient. However, you can run into use cases where the collaborative data structure will ensure consistency across clients, but violate some semantic constraint on the data.
For example, imagine you have an application that has a list that must contain at least one element. Assume there are two elements in the list. A Shared JSON data structure on its own (that allows for immediate local edits) would to allow two clients to simultaneously delete one element each. The end result is that the client app on both sides will become aware that the constraint was violated only when the remote operation comes in. Resolving this becomes difficult. What is the resolution strategy? Which of the two clients should initiate it? This is a contrived example for sure. But you run into things like this in various use cases, and occasionally you need either new data structures that encode these semantics, or you need an extendable system that allows you to customize constraints, and resolutions.
Could you point us to some of those togetherJS apps? I just recently learned about it in the context of collaborative browsing. How is this related to collaborative text stuff?
Are you working only with editors running inside web browsers or also native ones? What I would like to have is something that bridges two native editors, let's say emacs and VS Code, and let's a team share the editor (whatever each developer's preferred editor is) and not only the screen on Slack or similar. That would be great for remote pair programming. I guess there would be a protocol for selecting the buffer/tab and opening files (with local approval to prevent remote file reads/writes).
> What I would like to have is something that bridges two native editors
The big failing of co-editors is that currently they all silo into a single editor. THat's OK for e.g co-editing stuff on a web platform (e.g Google Docs) but it totally sucks for collaborating on code/documents you edit locally, forcing everyone to use the same editor/IDE, which ultimately fails because no one will use the same editor/IDE across a whole company/team/project, so although useful I've never seen one take hold for anything but a very short amount of time in any company.
We desperately need (a couple of) well-defined protocols that get implemented across editors, and one to emerge as a widely supported winner, whatever its shortcomings.
For plain text, this is not so hard. The data model for plain text (e.g. a string) and the set of mutations on that model are pretty small. Also describing where a cursor is and what is selected is likewise fairly straight forward. Co-editing between plain text editors is completely doable, IMHO.
It's much harder for rich text editors (RTEs) because the various RTE's vary widely in the exact subset of rich text features they support. One will support tables, and another will not. One will support video and another will not. One will use a linear position to describe where the cursor is, and another will use a DOM Range. This makes it very hard to support co-editing between different rich text editors. It's not impossible, just hard enough where most of us are not tackling it.
I think 'Google Wave Operational Transformation' was an attempt at this. I believe there is some merit in this; though my doubts come from the failure of Google Wave.
There has been a proliferation of realtime collaboration within code editors because the underlying data model (a string!) is so simple. As you mentioned, you would need to define a sufficient protocol encapsulating the common functionality between code editors along with defining which aspects would be shared and broadcasted vs kept private (e.g. an independent vs "follow" mode).
The data model could easily be defined in Convergence.
Implementation-wise, the primary limitation would be that at the moment we only offer a javascript/typescript client. I haven't explored the VSCode base but I know it was written in TS and provided as a "native app". We do have an example of Convergence running in node.js (NWJS).
Do you guys rely on a central authority approach like Prosemirror does? I'm desperately looking for something that's somewhat stateless or at least replicable (even better if shardable) on a document basis.
We do have a server in the architecture, and there is a small amount of state maintained by the server, per document as it is being edited. However the system is clusterable and documents shard across the cluster.
Thanks for the component you provided for shared cursors in text areas (https://github.com/convergencelabs/html-text-collab-ext). I've been using it with ShareDB, and have been working on adapting it to work with single text inputs, which has been surprisingly difficult (mainly dealing with scroll/overflow behaviour).
Our pleasure! If you want to share what you are doing we might be able to incorporate it into the utility for you! Happy to support a pull request, or to work collaboratively on it.
I love your description of "the dive". I am happy to see I am not the only one experiencing these recurring devouring interests that bring me much much deeper than I would like to go until I am starting to fork a github repo at 2am wondering "what the hell am I doing?"
And that article is tangentially related to a recurring interest of mine (shared editable 3d universe in which you can code collaboratively). If you happened to have an article comparing the merit of several server architectures/algorithms for collaborative text edition, I think many things can be related. I am now reading about CRDT (What the hell am I doing?) and that's something I may use to edit a scene graph collaboratively!
Hey thanks for that. :) Diving is super fun but also exceptionally exhausting. But often it's impossible to 'load' a complex problem in your head if you don't dive in for hours. Would be nice if there was a less stressful way.
Right I understood that. I was replying to the other guy who suggested putting the scene in a JSON Three.js thing. So the idea was that the thing I suggested was just a possibly easy way to get a collaborative JSON editor into your program.
Diving usually happens to me when I resume coffee consumption after going dry for a few days :-)
Right now I am staying in the python world, that I know well, and appreciate its introspection capabilities. I play with OpenGL and PyQt to display in 3d the code of objects loaded in memory. It was surprisingly easy to do. When I first tried, 15 years ago, that was more of a mess.
Now I mentioned CRDT but I was not thinking on raw text but using nodes of the scene graph as characters. I like the idea of insertions and deletions that keep invisible tombstones available.
You've probably heard of it already, but if not you should check out the game Cube/Sauerbraten. It's an open source quake-style fps game, with its level editor integrated right into the game itself, and you can edit levels collaboratively with other players at the same time.
I would be very appreciative if Google made a stripped down version of Google Docs for plaintext editing.
There are a bunch of basic integration features that would also be fantastic, but if the product was literally just Google Docs, minus the WYSIWYG bits, and all plaintext w/ maybe some syntax highlighting thrown in, I'd be super happy.
That's basically what I'm trying to build - we're starting with Quill.js and ShareDB, adding extensions, configuring, and then adding a plugin system based on React, which can take arbitrary components all based on ShareDB sync (think spreadsheets, forms, but I'd also love to integrate the Pyiodide stuff to have runnable Python code inside a rich text document, etc). We built this as part of a synchronous collaborative learning platform, but I'm currently trying to extract the key components so that it can be a stand-alone open source library that others can help build on. Very early prototype: https://www.npmjs.com/package/@chilifrog/reactive-rich-text
> Google Docs, minus the WYSIWYG bits, and all plaintext w/ maybe some syntax highlighting thrown in
That is the subset of features you want. Everyone else wants a slightly different subset of features. Add all of those subsets together and you end up with the total feature set of Google Docs.
Another product that should be mentioned here is OX Documents [1] by Open-Xchange. They have a web-based Office suite with MS file types (docx, xlsx, etc.) that has a collaboration feature. However, the product is not 100% Open-Source (CCSA:NC) [2]. The source code is accessible, but you are not allowed to use it in a commercial product.
IANAL, but if you want to use the product yourself (even as a company) that should be just fine, but you are neither allowed to host the product for someone else (commercially) nor to created a product based on their frontend code.
If you're looking for a plaintext collaborative editor, then Quill and a bit of glue-code is all you need. I believe there's even example code that shows how to maintain the document on a server.
We went with CKEditor5 with a commonmark backend and I cannot recommend it more highly. commonmark data is converted upon initially opening and when closing the editor, allowing all native functionality to happen efficiently.
However I do note that we do not yet have a production ready backend for collaborative editing. I would suggest to use their service if possible, since we're developing an open source product it would not be an option for us.
There is a quite leaning curve when implementing new plugins due to the strict separation of model and data (commonmark in our case) and user-facing views.
CKEditor 5 looks amazing (from tech to docs), but that lack of an available backend is a bummer. Do you know if there's anything out there that would work as an initial/hacky/example backend? How do you plan to tackle this?
I was able to get real-time collaborative text editing to work on one of my side projects by combining quill.js[1], ot.js[2], and firebase[3]. This was a few years ago when there weren't any fully open-source options. I'm in the process of switching out firebase for an elixir/phoenix backend[4]. I'm really glad for this article because it captures the exact feelings that I have.
That looks like a really cool project but.. there weren't any fully open source options? The original Etherpad came out 10 years ago. Etherpad Lite has been out for several years also and has had formatting plugins for quite awhile.
You're right, I completely forgot about Etherpad. But I don't consider Etherpad the same as Quilljs or ProseMirror. Etherpad is something you host (an app itself) while the other text editors are something you can build upon/integrate into your app. I didn't want to host an instance of Etherpad, I needed a library.
I wanted an editing experience similar to Medium.com text editor: consistent, well-formed html output and formatting. At the time, Quilljs was one of the best open-source editor that produced consistent output regardless of the browser.
Also I didn't want to store html, I wanted to store operations in the database.
I have an app similar to Figma[1] (basically real-time Sketch) and I needed just the real-time text editing feature.
tmbb has been doing work on writing a ShareDB compatible backend in Elixir (https://elixirforum.com/t/realtime-collaboration/9736/5), this has long been a dream of mine. If not, having a separate Node server that just deals with ShareDB documents would also be feasible - all the app logic could still be in Elixir. Happy to share stuff we've been doing with Quill, for example my company recently sponsored work on shared cursors/presence for Quill.
The server does have a certain power, it can send you evil javascript which does the wrong thing (leaks the key or the data back to the server or to someone else). This is however an active attack which makes it detectable. The NSA really hates doing these because they might get caught and laughed at and humiliated in front of the whole world (again). If you're making the NSA mad enough for them to use an active attack against you, Great Success Highfive, now take the battery out of your computer before it spawns Agent Smith.
This was one of the project idea I made myself work on during this April. The motivation being, "how hard is it to make a real-time code editor". Long story short, it is far more complicated than I had imagine and to complete in less than a week. Implementing the CRDT itself, the logic itself is challenging and than putting it down to code is another story.
From my understanding, an implementation of logoot[1] is built to create the skeleton of a collaborative environment. From there on, you have the building block to make other things work with it, such as collaborative canvas.
I built one using Socket.io on Angular and used Ace Editor own implementation of CRDT
I wonder why would one make a headline "Open source collaborative text editors", and then write about online editors which are neither fully open-sourced, nor fully collaborative. And then stop with them, without even mentioning solutions that are both truly real-time collaborative and truly open source.
That certainly wasn’t the intention, the title is a bit of a mix of: the initial intent to list all open source collaborative text editors, and then realizing, as I state in the post, that it wouldn’t add much value to have a long shallow list, then deciding to limit the scope significantly, but not wanting to redesign the headline illustration, as so much effort already went into the post (both content and the code behind it) that I just needed to stop working on it and ship it :) Hope that makes sense!
You should try https://www.fiduswriter.org. It does have realtime collaboration, export to ODT/DOCX/Epub/LaTeX, tracked changes, WYSIWYG interface, etc. and in the latest version even a document template editor. I think relatively speaking it's quite developed in comparison to some of the other word processors presented here. It does include both frontend and backend parts to make collaboration work. (I did a lot of the programming over the past 7 years.)
Yes. It's not the sort of thing you'd use with untrusted parties over the internet since it requires access to the remote x-server, but it totally works for collaborative editing (esp with people sitting in the same room):
So, lots of open-source UIs that use closed-source servers for collaborative editing?
This figures. This is one of the ways 'open source' companies can make money.
So, the author sets about making an open-source collaboration server that inter-operates with the existing open-source editor?
Kudos!
But now the company can retaliate by adding features or inconsistencies to the closed source server and its integration into the UI, making the open-source server forever out-of-date and playing catch-up.
SubEthaEdit is a collaborative real-time editor designed for Mac OS X. The name comes from the Sub-Etha communication network in The Hitchhiker's Guide to the Galaxy series.
Not sure why you got downvoted. SubEthaEdit was magical back when - here is the link to the github repo for the recently released source: https://github.com/subethaedit/SubEthaEdit
One neat thing about live collaboration is that every single edit is stored. This enables some nice things, like seamless replay of the entire document editing history. We've been working for a while on trying to analyze edit histories, to see if we could predict which stage of editing a document is in, which writing strategy users use, how a small team is collaborating etc. (Main purpose is to support teachers using this with student groups). We have some code here that works with Etherpad and ShareDB (https://github.com/chili-epfl/FROG-analytics), and happy to share preprints with anyone interested.
Hi Houshuang - very interested in this area. My company produces a web app for document-based collaboration in Higher Education (working with Universities in Australia and NZ over the last 10 years). We are also research-active and involved in collaborative research projects with a number of Unis. Please contact me via my profile. ~ Rodney
Hey houshuang! Thanks for mentioning this. I do think that replay and the potential to invert operations (including some operations and not others) is a very interesting feature of OT and we use it at Lever quite often. It is incredibly useful when doing enterprise customer support in addition to something that you can build user-facing features around.
I’m new to the topic from a technical point of view. I was wondering if the conflict resolving issues are fundamentally different from online gaming scenarios. I.e., could the ways problems are solved in multiplayer games also be applied to collaborative editors and vice versa?
From my limited exploration and talk with Justin over at http://www.kinematicsoup.com, the requirement for low latency combined with a lot of large binary data requires different approaches.
Since it didn't make the list, nor has been mentioned in this thread, I want to recommend https://firepad.io/ - we use it quite heavily at our company. It's uses Firebase behind the scenes so is essentially 'serverless' and we use it for editing our e-mail newsletters live as a team. It literally takes minutes to throw into an app.
Until last year and for two years I kept an Etherpad Lite to do shared text editing and it was amazing. It was in use for three years. The only reason I stopped using it was Google Docs, who bought the company or IP for Etherpad, can do the same thing and therefore I don't have to do the Linux maintenance. I even had it use SQlite as the database and it could handle as many as 30 simultaneous users without a hitch.
Etherpad has been open source since 2009, when it was bought by Google and immediately re-licensed. It stayed open source ever since. It was written in a mix of Scala & JS.
Etherpad Lite (https://github.com/ether/etherpad-lite) is a port to javascript, is backed by the Free Software Conservancy and is not owned by any corporation.
Yes, if you do not mind using Google's infrastructure, Google Docs is easier & more featureul. There are some cases in which running on premises may make a difference, and Etherpad strives to fill that palce.
We (at brainioapp.com) are right now working on collaborative mind map app that works like notes that you can convert (based on mark-down) to mind map. It's like a combination of Evernote and XMind. Video here: https://youtu.be/nJjFs7GmIBQ.
VSCode is collaborative and Open Source. You install a plugin and send someone a link - you are then hacking on their document, in your VScode on your keyboard with your plugins installed.
Sorry I can't remember what the plugin is called, I used it every day at my last contract though. Made pairing super easy.
I'm curious about the choice of PubSweet for the collaboration components. Can you summarise what functions PubSweet is performing in your example app, and are there alternatives to PubSweet that you looked at for this functionality?
It mainly has to do with familiarity with the framework. The focus of the work was on the collaborative server bits, not learning a new framework/ecosystem and I feel very comfortable in PubSweet, being its lead dev. The project also served as an app-building exercise, as we’re just now meeting in Cambridge to focus on that aspect and reusable components within PubSweet. And when all said and done, there weren’t any friction points with this approach, it all just sort of worked, from data model migrations to React frontend, so yay :) Hope that clears it up!
A CKEditor 5 team member here. Thanks a lot for such an in-depth analysis, the article is a great read and we truly appreciate being a contender there.
As for the existing implementations of CKEditor 5 with real-time collaboration, I'm afraid since the collaborative editor is usually a component of a larger platform (publishing, e-learning, CMS, intranet, documentation management system etc.) it'd be difficult to find something like a publicly available real-life use case demo of it. Without much effort, you can check out the real-time collaborative editing demos (https://ckeditor.com/collaboration/real-time-collaborative-e...) that we have on our website.
A while ago we started building our own internal implementation of a collaborative document management system using CKEditor 5 and features such as real-time collaboration with track changes (something like Google Docs) for our own needs - hopefully, once we polish it a bit, we will be able to release it. We are working on publishing a few case studies with existing customers, too.
As for offline collaboration, our current solution switches the editor to read-only when you e.g. lose connectivity, and then the editing mode is back on when you reconnect, along with the changes that were done in the meantime by other users when you were offline. The platform is ready to fully support "offline" writing. We tested some more complex scenarios with one of the users being offline - and it works (however, no periods as long as days of being offline). We focused on delivering other collaborative editing features such as track changes or comments that were more required by our customers. Full support for writing while being offline is still on our roadmap, though.
As for the answer to why the collaboration component is not Open Source: we have currently 40 people in total working on CKEditor (CKEditor 4 & CKEditor 5). Developing such a complex application takes a lot of time and resources and we simply need money to cover the expenses. Without that, we’d not be able to spend so much time on fixing bugs or bringing new features. Also, unfortunately, we learned the hard way that some of the biggest IT companies don’t want to help Open Source projects by spending even a single dollar on them, even if they use it in an application that brings them millions of dollars.
Regarding mobile support: CKEditor 5 works well on mobiles (with and without real-time collaboration). The comment you referred to was about the lack of an online demo of real-time collaboration on ckeditor.com for mobile devices. The reason behind this decision was that we are using the sidebar for displaying comments, which results in rather poor UX on mobiles. Fortunately, we are almost finishing the inline annotations plugin which will display comments inline, without using the sidebar. Feel free to assign a full Kiwi for mobile support :)
I don't like "merge" word because it implies how the problem is solved. "Synchronise" would be better, as there is no merging, actually.
We use Operational Transformation, so all the changes are stored as operations (with their most important property being path - the position where the operation happened).
When a user comes online, they try to re-send all the operations that are buffered and not yet sent to the server. Then it is a matter of transforming these operations by the new operations that happened in the meantime (when the user was offline). Of course, the user also has to accept the new operations.
This is a bigger / more complex version of the basic problem of the real-time collaboration. During real-time collaboration with all users online you might need to transform your operation by maybe several other operations at most. When you go offline, it might be tens or hundreds of operations but the problem is basically the same, just bigger.
The quality of the transformation algorithms will make a huge role in how well the user intentions will be preserved.
"Platform is ready" means that the theory behind our solution is correct and that we checked some moderately complex scenarios.
Edit: still, the base scenario is that the users write "online" and "offline" kicks in when you lose a connection (for hours, even). If we are talking about "everyone writes offline and then they magically merge" then I think this might be a totally different feature (and maybe even outside of the editor and inside the CMS).
I know the folks at Overleaf and they're awesome! However, their editor is not open source. But yes, ShareLaTex probably needs a spot on there, though I haven't looked at it in a long while and it seems to be pretty dormant in terms of development: https://github.com/sharelatex/sharelatex/graphs/contributors
Had no idea it had real-time collaboration... Wonder how it's done. Digging in now... :)
If you're using it, it would be great to get your thoughts on it and what particular things you like about it.
Update: ShareLaTeX uses a string-based collaboration algorithm (i.e. LaTeX is plain text, not a structured document format), and seems to be limited to max 2 collaborators (though that could be old info), so that explains why it didn't appear on my radar. I'll take it for a spin anyway, thanks!
Also, you might be unaware of CoCalc, which is open source https://github.com/sagemathinc/cocalc. CoCalc supports collaborative editing of many structured documents, including Jupyter notebooks, but not rich text at present.
Just to be clear, open source development of ShareLaTeX is ongoing and indeed has accelerated since Overleaf and ShareLaTeX joined forces. ShareLaTeX is split into many repos, and the one linked above is a 'coordinator' repo that doesn't get updated much. See https://github.com/sharelatex/web-sharelatex for one that is more active.
The free version of Overleaf/ShareLaTeX does have a restriction on the number of collaborators: you plus one other. Paid accounts support more collaborators [1], and you can also get more collaborators on the free account by referring others [2].
Not really. I use tmux + vim on a daily basis, but it starts with having just one cursor. Goes on with not being web-based and in the end, it has no offline collaboration support (that requires some kind of merge strategy).
So yes, there are some use-cases when it is sufficient for collaboration, but most certainly not for the use-cases described in the blog post.
My use-case isn't similar to the ones described in the blog post.
You could easily make tmux + vim web-based if you desire (I don't see why, I'd just use Mosh or SSH instead). For merging you could abstract the content to a Git repository. For multiple cursors you could try Wemux [1].
I've just found that Cryptpad offers a whiteboard feature.[1] I know nothing more about them and was just checking them out after seeing the comment in this thread.
It's an incredibly fun topic, and one that's also really challenging. It also gave me first-hand experience with fuzz-testing where I could simulate all kinds of concurrent-edit conflicts and ensure both clients came out with the same end-result. While the end client/server implementations are working, it was a lot more effort than I anticipated building a full-fledged editor on top. I was too stubborn at the time to attempt to integrate with existing RTEs though, so the most notable part of the project was the core lib, not the end product.
For those interested in implementing it for themselves, I can't recommend this piece [3] by @djspiewak enough.
[1] https://github.com/jahfer/ot [2] https://github.com/jahfer/othello [3] http://www.codecommit.com/blog/java/understanding-and-applyi...