You'd already be using source maps in any real-world scenario so I am not sure what's the value proposition here outside of "just for fun, I guess".
The tsc transpilation to lower ES versions is actually really useful when using not-so-recent Node versions. Not to mention this severely restricts TypeScript syntax to "just types" which isn't too bad but it means you now have to worry about yet another thing.
Then there's the ESM & CJS mess. You almost always want to export in both formats which will change a lot of the syntax, so this quickly falls apart for anything serious.
Our goal with Notesnook has always been to make privacy simpler to adopt for normal users who don't have time to think about privacy. We do that by making privacy that "default" everywhere in the app. We recently crossed 100K registered users with over 4 million notes created on our platform.
If anyone has any questions I'd be happy to answer.
Multiple minor UI glitches, search and note organization is a mess, the app is not local-first, it lacks offline mode which would be guaranteed to work on a plane, the editor is not bad but might be better, no encrypted backups, extremely limited set of syntax highlighters. And many, many other issues.
Across everything I've tried, Standard Notes seems to have the best set of tradeoffs, but they refuse to implement features (like add syntax highliters for modern lagnuages, if I remember correctly, they said that they would only consider it if I prove that thousands of people need it).
Anytype provides good privacy guarantees but they have crappy UX and, ergh, their app needs 5 seconds to start on a modern mac.
The whole notetaking apps market is a shitshow and a total mess. Don't waste your time and money on these half-backed subscription-based electron apps.
It seems to me you are looking for a syntax highlighter in a note taking app. Which modern language are you looking for?
There are encrypted backups. Actually, backups are encrypted by default.
Notesnook also works offline by default so there's no need for a special mode. It's also, obviously, local-first since everything gets encrypted on your device. We don't have a lot of options there.
Obsidian is a cool app but it's targeted for a very specific kind of user. Of course, for a technical user Obsidian would feel like home but for an average user with not a lot of technical know how, Obsidian would go way over their heads. Not everyone appreciates having to handle files, worry about syncing, security etc.
> Not everyone appreciates having to handle files, worry about syncing, security etc.
I don't think that's a requirement to use Obsidian? I've had, let's say "less computer savvy" friends who've gotten use out of Obsidian.
AFAIK, you need to create a vault in the beginning, so you need to select somewhere on your computer where stuff gets stored. All the other things are automatic and within Obsidian, although some things are paid addons (like Obsidian Sync).
Markdown itself is more technical than, say, a WYSIWYG editor. People are more used to using MS Word and Google Docs. Obsidian prioritizes a different kind of note taking and, hence, you'll find a lot of tech-savvy users among its community. There's a reason for that.
Oh wow. Just heard the Notesnook is on the front page of HN. I am the co-founder so if you have any questions, feel free to ask.
Oh and to clear a few confusions:
1. Notesnook is 100% open source. That includes the server, client apps, and everything else. It's not partially open source.
2. Zero knowledge does not mean Zero Knowledge Proof but Zero Knowledge as in we, the company and people behind Notesnook, have no knowledge regarding what you have in your notes. I see that this might be more accurately called "no knowledge".
There's no point in offering users self hosting if you are going to throw them into a fire by doing so. Notesnook is rapidly evolving and changing, and hoping for self hosted users to keep up is impractical. Instead, we want to first stabilize the backend, and then think about self hosting.
After v3, our primary focus will be on self hosting and getting an audit done.
> Notesnook is rapidly evolving and changing, and hoping for self hosted users to keep up is impractical
I don't know...does self-host equal slow changes? For me that's part of being a self-hoster, I have to keep the softwares up-to-date, and I actually appreciate if updates are frequent.
> After v3, our primary focus will be on self hosting
Sounds good, good luck on further development, if I ever see a self-hosting guide I'll check y'all out ;)
Not slow changes. Offering self hosted officially means that we have to be aware of users who are self hosting before we make any drastic changes, writing migrating guides, and giving some sort of support. All that has an impact on productivity.
build an export and import and offer it with docker and I'm on board. I dont care much If I have to take a few extra steps... but I'm done hosting my private stuff in the cloud. Self host is the future.
yes, it's a pwa so offline first. it uses yjs with webrtc for p2p generically and the "last mile" will be a "light" electron app that you can use via webrtc to store files .
no technical server, so really the only infrastructure is the signalling, and it's straight forward to scale that.
might still be a niche but it serves my day job needs. field work and reporting asynchronous is a pain.
As someone who has been doing local-first for the last 3 years with Notesnook, let me tell you: it's not all gardens and roses. Local first has its own very unique set of problems:
1. What to do with stale user data? What happens if a user doesn't open the app for a year? How do you handle migrations?
2. What about data corruption? What happens if the user has a network interruption during a sync? How do you handle partial states?
3. What happens when you have merge conflicts during a sync? CRDT structures are not even close to enough for this.
4. What happens when the user has millions of items? How do you handle sync and storage for that? How do you handle backups? How do you handle exports?
One would imagine that having all your data locally would make things fast and easy, but oh boy! Not everyone has a high end machine. Mobiles are really bad with memory. iOS and Android have insane level of restrictions on how much memory an app can consume, and for good reason because most consumer mobile phones have 4-6 gbs of RAM.
All these problems do not exist in a non-local-first situation (but other problems do). Things are actually simpler in a server-first environment because all the heavy lifting is done by you instead of the user.
> 1. What to do with stale user data? What happens if a user doesn't open the app for a year? How do you handle migrations?
version = db.query("select value from config where key='version'").fetch_one()
switch (version) {
case 1:
db.migrate_to_version_2()
fallthrough
case 2:
db.migrate_to_version_3()
// ... and so on
}
assert(version == 3)
start_sync()
Just don't delete the old cases. Refuse to run sync if device is not on the latest schema version.
One of my Django projects started in 2018 and has over 150 migration files, some involving major schema refactors (including introducing multi-tenancy). I can take a DB dump from 2018, migrate it, and have the app run against master, without any manual fixes. I don't think it's an unsolved problem.
> 2. What about data corruption? What happens if the user has a network interruption during a sync? How do you handle partial states?
Run the sync in a transaction.
> 3. What happens when you have merge conflicts during a sync? CRDT structures are not even close to enough for this.
CRDTs are probably the best we have so far, but what you should do depends on the application. You may have to ask the user to pick one of the possible resolutions. "Keep version A, version B, or both?"
> 4. What happens when the user has millions of items? How do you handle sync and storage for that?
Every system has practical limits. Set a soft limit to warn people about pushing the app too far. Find out who your user with a million items is. Talk to them about their use cases. Figure out if you can improve the product, maybe offer a pro/higher-priced tier.
> Mobiles are really bad with memory. iOS and Android have insane level of restrictions on how much memory an app can consume, and for good reason because most consumer mobile phones have 4-6 gbs of RAM.
You don't load up your entire DB into memory on the backend either. (Well your database server is probably powerful enough to keep the entire working set in memory, but you don't start your request handler with "select * from users".)
You're asking very broad questions, and I know these are very simplistic answers - every product will be slightly different and face unique trade-offs. But I don't think the solutions are outside of reach for an average semi-competent engineer.
> You may have to ask the user to pick one of the possible resolutions. "Keep version A, version B, or both?"
For structured data, with compound entities, linked entities, both, or even both in the same entity, that can be a lot more complicated.
If a user has updated an object and some of its children, is that an atomic change or might they want the child/descendent/parent/ancestor/linked updates to go through even if the others don't? All of them or some? If you can't automatically decide this (which you possibly can't in a way that will satisfy a large enough majority of use cases) how do you present the question to the user (baring in mind this might be a very non-technical user)?
Also what if another user wants to override an update that invalidates part/all of their own? Or try to merge them? Depending on your app this might not matter (the user might always be me on different devices, likely using one at once, that is easier to understand than the user interacting with others potentially making many overlapping updates).
I think you misunderstand. My intention was not to say local-first is bad or impossible; it's not. We have been local-first at Notesnook since the beginning and it has been going alright so far.
But anyone looking to go local-first or build a local-first solution should have a clear idea of what problems can arise. As I said in the original comment: it's not all gardens and roses.
Just a few weeks back a user came to us after failing to migrate GBs of their data off of Evernote. This, of course, included attachments. They had successfully imported & synced 80K items, but when they tried to login on their iPhone, the sync was really, really slow. They had to wait 5 hours just to get the count up to 20K items. And that's when the app crashed resetting the whole sync progress to 0.
In short, we had not considered someone syncing 80K items. To be clear, 80K is not a lot of items even for a local-first sync system, but you do have to optimize for it. The solution consisted of extensively utilizing batching & parallelization on both the backend & the users' device.
The result? Now their 80K items sync within 2 minutes.
The problem wouldn't exist. This was about the phone fetching 80k new items from the server. If the phone just shows the item you're looking at, one at a time, and doesn't try to sync everything, there's no such problem.
I've been working on mobile apps developed for education in Afghanistan, rural Rwanda, etc for the last 9 years. I used to think that sync was the way, but I have learned from painful experience.
4 (extended): What happens when the user has access to millions of items, but they probably only want a few (e.g. an ebook library catalog)? Do you waste huge amounts of bandwidth and storage to transfer data, of which 99.9% will be useless? We had a situation where the admin user logging in on a project that had been running for years resulted in a sync of 500MB+, 99.99% of that data the admin would never directly use.
Also: do you make users wait for all that old data to sync before they can do anything when they login?
Relying on sync is basically relying on push-only data access. I think in reality most offline/local first applications will work best when they push what a user probably wants (e.g. because they selected it for offline sync, starred it, etc) and can pull any other data on demand (in a way that it can be accessed later offline).
Query-based replication works when you know what the user probably wants to have in advance (e.g. a device in a warehouse needs records for that stock in that warehouse, not others). But that's still push.
You still need pull on demand access when a user opens any random item where we don't know in advance what they probably want (e.g. a discussion board scenario).
I'd say you're spot on except for point (3). There's a number of crdt and event log approaches that, when combined properly in order to preserve user intent, can solve almost all merge issues of applications that do not require strong consistency.
> 4. What happens when the user has millions of items?
Partial replication is a problem I haven't seen many people solving but it is definitely the next frontier in this space.
I am the developer of RxDB, a local-first javascript database, and I made multiple apps with it and worked with many people creating local first apps. The problems you describe are basically solved.
> What to do with stale user data?
Version/Schema migration in RxDB (same for IndexedDB and watermelonDB) can be done in simple javascript functions. This just works. https://rxdb.info/data-migration.html
> What about data corruption?
Offline first apps are built on the premise that internet connections drop or do not exist at all. The replication protocols are build with exact that purpose so they do not have any problems with that. https://rxdb.info/replication.html
> What happens when you have merge conflicts during a sync?
> What happens when the user has millions of items?
There is a limit on how much you can store on a client device. If you need to store gigabytes of data, it will just not work. You are right at that point.
> How do you handle backups? How do you handle exports?
Some domains just don't have these issues, like note taking where data is small and there are many acceptable ways to handle conflicts, like backup files and three way merges.
Maybe the question is less "How to make this work local first" and more "How to squash down the problem domain into something that just naturally fits with local first"?
I wish we had something like an embeddable version of SyncThing, that had incoming file change notifications, conflict handler plugins, and partial sync capabilities, and an easy cloud storage option.
I think most everything I've ever wanted to p2pify could probably be built on top of that, except social media which is a separate really hard thing.
I don't think even you understand what you just said.
Consumer devices are notorious for their reliability problems. Compared to a full blown server that you have 100% control over with almost insane amounts of RAM & CPU power & a lot of guarantees.
Running a migrations on a server is far, far different to running it on every users' device. The sheer scale of it is different.
> Using 4 GB per user on your backend works?
That was a comment on the average RAM on a consumer device - not the total RAM required per user.
> Running a migrations on a server is far, far different to running it on every users' device. The sheer scale of it is different.
Well, it’s not only just that. Among the other things, some of the application instances would be outdated but still need to work, so you would need to support _all_ the DB schemes you have ever had for your app.
I know I understand what I said, and I am not convinced by anything you said.
What reliability problem would prevent you from running local-first software but doesn't interfere with running a thin client?
Why would the business logic part of your app require more RAM on the end-user device that it requires per-user (or per-document, etc) on a server?
Why do you claim that running migrations is so fundamentally different here and there?
If you want to argue I would appreciate you doing it with real arguments and experiences rather than even more unsubstantiated claims and statements like "you don't understand what you just said".
[0]: https://notesnook.com/