Excited to finally post it on HN. It's something I've been working on for the last year, though in a sense, for much longer!
Etebase is an open-source SDK and backend-as-a-service for building end-to-end encrypted applications. Think Firebase but end-to-end encrypted.
More users care about the privacy and security of their data every day, and encrypted applications are becoming mainstream. However, talking with developers, many get encryption wrong, and they don't even realise it. As good cryptography and bad cryptography look identical to the untrained eye.
The idea behind Etebase is to make an easy-to-use API to enable developers to build encrypted apps, and it's based on our learnings from building EteSync[2] over the years. There's still a lot more to get done, but Etebase is already used in GNOME, KDE, Tasks.org, EteSync, and more. With libraries available for Rust, JavaScript/TypeScript, Java, Python and C/C++.
I hope Etebase will enable a new age of privacy-first and end-to-end encrypted applications, and that our data will no longer be used to track, analyze and manipulate us. Your data, yours only. Let's end-to-end encrypt everything!
I'd love to hear about your experience building encrypted applications. Got any questions? Suggestions? Please let me know!
Nice work! I spent about six months pursuing the exact same idea about five years ago with the hope of building a company around it. I got reasonably far with a prototype, but gave up on the business side of things after doing more extensive market research and interviews.
One of the people I talked to had been involved with a startup called Adrenaline Mobility which did something very similar, but ultimately ran out of runway and was acquired by Twitter. He told an interesting story about how they (over-)built a highly scalable service but had a really hard time selling it. Those who lack the training to know the difference between good security and hand waving didn't see the value, and those who did have the training and skills didn't trust a third party to do it right anyway.
Nonetheless, a lot has changed in the past five years, and end-to-end encryption is more widely appreciated now. I hope you have better luck with the commercial side of things; this looks like a very promising start.
Thanks a lot! Things have indeed changed, and I feel like the public is really starting to understand the dangers with having their data exposed. I've been running EteSync for years now, and it's still growing.
I'd love to have a chat with you to learn from your experience if you are willing! My email is tom at etebase, please drop me a line (or let me know how to best reach you).
This might be very good timing for your product - the European Data Protection Board has just out their first guidance after the Schrems II case. It seems to firmly point towards cloud services where the provider can access the data in plaintext as being "unable to have the risks mitigated" from the perspective of international transfers.
Perhaps a straightforward E2EE toolbox could help companies implement their products rapidly so they remain complaint. Clearly this won't help fix issues like Microsoft 365 or GSuite (which need access to the plaintext for email and similar), but it might help some types of SaaS to thicken up their client application and prevent the backend having access to unencrypted data, thus making the transfer permitted.
I just want to echo the above comment around how most customers don't have the knowledge to understand the benefit of this, but hopefully this is changing as we see more strict enforcement of penalties for data breaches. Data is fast becoming a liability you don't want to have the ability to see, and systems like yours offer usable solutions for those who don't understand all the technology, but need a solution.
I wrote something similar, a couple of years ago[0], but it doesn't include encryption; it merely gives a place to add encryption. I didn't want to deal with the legalities of included encryption, and I think others can do far better than I (but it is quite secure, nonetheless[1]). It was really done, just to "retool" my architectural and engineering skills, as I was pivoting from being a manager, back to being a developer.
I am now using it as a backend for an app under development (currently closed-source). It has been working a treat.
But after the app has been released, I may encourage the folks to look at alternatives for the backend. I am not really into maintaining a server system. I like to stay on the app side of things.
The deal with mine, is the encryption is left to the client. The server just stores a binary "thing." That "thing" can be an encrypted envelope, managed by the client[s]. Since it's a closed-loop system, then a key could be shared programmatically, outside the purview of the server.
I don't think Firefox sync makes it very easy for devs to build other encrypted applications. Even if they did however, there are a lot of challenges with encrypted applications that they just don't take care of. Not to mention that they are very we centric and don't have developer APIs in many languages.
Is there any technical document describing how it works ? For example, order-preserving indices (for range queries) are often an issue in end-to-end databases ; did your design allow to you to provide them, or did you just consider solving this problem out-of-scope ?
There are docs here: https://docs.etebase.com/
Though they mostly focus on how to use rather than how it works (though we are making progress on that section).
As for your specific question: we currently don't solve that, but we plan on doing it in the near future.
Our approach will be to maintain a client-side index that will be synced across devices. Etebase already utilities to maintain consistency, so this can be done safely.
This question, and ones like it, are exactly why we created Etebase. Building encrypted applications is fundamentally different and comes with a lot of challenges. We plan on continue building all of the tools needed to develop using this different paradigm.
Hard to know exactly as people get it from multiple sources, but there are a few hundred thousands downloads on Dockerhub, people run it from source, and it's packaged for OpenWrt, Arch (AUR), Fedora (COPR) and a few other Linux distributions. So yeah, quite a few users.
I have been looking for something exactly like this. Your origin story is very apt: there really isn't an approachable cross-platform solution for noobs like me. Will definitely be checking this out. Thank you for sharing.
This is neat, but it seems like it's missing a few things. For one, how do you do migrations? Do I need to include every migration in my application bundle in perpetuity, running them in the client on startup? There doesn't seem to be a mechanism to help with downloading, migrating, and reuploading data. That doesn't feel so nice.
More pertinent, though, how do I ensure my users aren't abusing my service? If I allow folks to upload avatars for folks in their address book, how can I stop someone from running up my bill by uploading their MP3 collection? As far as I can tell, if everything is encrypted, I just have some very active users. Obviously there's some amount of tradeoff here, but it would seem like a necessary counterbalance is strong quota controls, which I don't see in the docs.
Yeah, as I said in my main comment, there are still things that need to be done. One of them is strong quotas. We plan on having it in the near future (we are also running a service, EteSync, using Etebase) though I think it's not an issue that should prevent you from getting started. :)
As for migrations: that's a tough one, and we indeed don't have an easy-to-use mechanism yet. I mentioned it in another comment: the API at the moment is powerful and used in production, but it's still missing some goodies to make the developer experience even better.
As for including every migration in your application bundle: yeah, well until you consider an old version EOL. It's the same challenge Microsoft has with Office, and every app has with their local storage.
Since the pitch keeps referring to Firebase, I thought I might as well mention this here: a while ago I built an encryption layer for the Firebase RTDB at https://github.com/Reviewable/firecrypt. It's not nearly as nice as Etsebase appears to be, but if you're already committed (cough locked in cough) to Firebase then it will get you a good chunk of the way there. (And yes, I still actively use and maintain it.)
VERY interesting! Reading the README first paragraph, I see you mention this doesn't make sense for apps where anon users can see data. But what about when anon users can sign up for an account, THEN have access to sensitive data?
It might work, if you truly trust whoever signs up. The fundamental issue is that there's just one encryption key for the whole database, and you need to provide it to everyone who you want to have access to encrypted data. If you stop trusting someone for any reason your only recourse is re-encrypting the entire database, which is a painful offline operation. (You might also be able to support multiple encryption keys -- one per user, say -- but the library isn't really set up to make that convenient right now.)
As for userbase: I think our and their scopes are very different. They are more of an easy way to store data that happens to use some encryption, not a comprehensive end-to-end encryption toolkit. They don't do sharing, integrity management, data de-duplication, and lot of other things that are needed for more serious applications. In addition, they don't have libraries for anything other than JS.
As said in the other comment, our mission is to encrypt everything, and this is the next step towards that! It's actually something that we've been planning since forever, but it's so nice to finally have it out!
Firebaser here! Just want to say this is (in my personal opinion) pretty cool and even though it's the "competition" you're doing something unique and we'll take the comparison as a compliment.
I don't think we are really direct competitors, we are targeting a different niche. It's just that you are the best example of a backend-as-a-service and thus the best way to explain what we do. :)
Very interesting, I've been on the lookout for something like this for a long time and it still surprises me how few out-of-the-box, open source, E2E encryption solutions there are available (especially for us Europeans...).
One thing that's not clear to me from first glance at the docs: does this support offline-first apps? So what about sync with local storage, search, CRUD of collections/items offline, conflict resolution, ...? Am I missing something, or is this something that the developer has to implement themselves?
Thanks for your questions! What we have now is the first step of our vision. It supports offline-first apps in the sense that you can work offline and sync later. If you are talking about not using a server at all: that's coming too.
All the rest of the things you mention: they are all in the works. What we have now is already enough to build great apps (as done with EteSync for example), though we want to build the entire ecosystem around it.
This looks exactly like the application I've been wanting to build for quite some time! Thank god I didn't have to. :)
Question: The Collection/Item data structure seems simple but also quite powerful at the same time. However, for lots of applications I imagine it would be nice to have some common abstractions available in the API instead of having to roll my own. I'm thinking of, e.g., data structures like queues, stacks, dictionaries, trees, and also conflict resolution strategies. Are there any plans to add those to the API?
This is a great question, and the answer is a resounding yes!
The API is already powerful, and powers real applications, but it's quite raw. We plan on gradually building all of the above and more. We just first needed a solid base to work from.
Tangential, but is Etesync and Notes a cheap, unbloated and foolproof app combo compared to paid competitors? And would the existing protocol support a (team) password manager? I would buy that trio if its a healthy business at $2 per month and avoided lock-in. And in future enabled "SaaS" products which is a static site + a safe backend.
I'm wary of SaaS that earns a ton of cash and then implodes under their expenses. I'll be happy to get an Etesync account and use apps built for it. Like how draw.io is a mostly static site using Dropbox, GDrive etc as portable backends.
We are not going to implode under our expenses. We keep our expenses in check (and have been for the last four years), and we put a big emphasis on the sustainability of the service.
After reading the docs I have a couple more questions:
- there's the notion of a local cache but is it synced / de-duped automatically or is it something I still have to do?
Let's say I have a ToDo mobile app. While online I enter the task "buy milk". Sometimes later I'm in airplane mode and I modify the task as "buy condensed milk". Do your library updates the changes automagically once online?
- how realistic is it to run your own server? Is it a couple hours of config and forget about it or it's a multi-days endeavor without certainty of success?
- in your examples you use the name Cyberdyne. If I use Etebase do I unknowingly contribute to Skynet? ;)
Local cache: no, you need to do it yourself once the app goes online. We plan on having things like that soon as part of support for automatic conflict resolution and etc.
Running your own server: trivial, not even a couple of hours. Check out the repo for instructions, or just use the docker images (created by the community).
This looks really good and I can tell a lot of work went into this. Bravo.
Although this does not yet have the same sort of real-time push syncing (or subscriptions) the way Firestore (or Firebase) both have, it looks like it has the right foundation to support that.
Also impressed that this has such wide client SDK support already. Again, I can tell a lot of work went into it. I'll be trying this out and following the project! Excited to see where it goes next.
A secure encryption key is derived from the user's password using a random salt an Argon2id, though the data is encrypted with a randomly generated key. Keys are generated for each "layer" of the account (so one for the account, one for collections, and etc), so each part can be re-encrypted (or not) separately if needed.
Password change: you can either re-encrypt the data if you want, or more likely, if the password hasn't be compromised, just re-encrypt the main encryption key that's used to encrypt the data.
Password lost: tough luck. We can't help you recover that, because we don't have access to your data. Though we have some ideas on how to maybe enable recovery (using key custodians, shamir secret sharing, or another method, haven't decided).
The salt is kept on the server, so you can just fetch it again when switching devices.
However, on the long term, we plan on considering passwords more of a backup feature, and instead move to a model where you just authorise your devices from running devices (though can always fallback to passwords).
That's a question that's looms over every new company and open-source project.
I don't think Firebase will go down this route, it's a very different offering to what they already have, and any encrypted stuff will be incompatible with the rest of it. There's also Google's (rightfully earned) bad public image when it comes to privacy, which will limit adoption.
In addition, being open-source is very important for a solution revolving around data ownership and privacy, something that Google will just not do. Our plan is therefore to continue building the best open-source project we can and let the developers decide.
I think it's very different to supabase.io. They are focusing on being a Firebase alternative, we are focusing on making it easy to build encrypted applications. Not the same target audience and not the same offering.
With that being said, Supabase is very cool, and I'm a big fan! If you don't care about end-to-end encryption, you should definitely use Supabase!
This is a very good question, and one thing we had issues with over the years while developing EteSync. At EteSync, we ended up building client side debug utils that we can get users to launch in order to figure out what's going on and selectively report issues. You can, depending on your users' risk profile, also have it more automatic (we kept it semi-automatic).
Developer tooling is definitely lacking at the moment for building encrypted applications and it's something we plan on addressing.
One of the shortcomings of Firebase is that you can't self host. Self-hosting is still relevant for developers and opens up the device market where cloud connectivity isn't appropriate. Great timing on Etebase.
You don't have to self-host Etebase, you can also use the hosted version on etebase.com if you wish. However, what we are offering here is an alternative to the Firebase APIs, rather than the fact that it's cloud hosted.
As said in a sibling comment, we currently use public-key encryption and verifying keys. However, we plan on adding support for SPAKE2+ (like e.g. Magic wormhole) among other methods, to make it even easier!
I see in another comment that users have a key pair generated for them. What you can do is fetch other users public keys and encrypt data to them, or better yet, generate a symmetric key for a project and encrypt that to trusted users.
It's how we did it for Portabella (https://portabella.io) anyway but that's not based on Etebase/Etesync, just another e2ee tool
A quick bit of feedback regarding your website: I was interested in reading the front page but ended up leaving because the “perfect for...” type and delete effect was causing the remainder of the page to jump up and down by one line every few seconds. It just happens to wrap in the wrong place on my phone.
More users care about the privacy and security of their data every day, and encrypted applications are becoming mainstream. However, talking with developers, many get encryption wrong, and they don't even realise it. As good cryptography and bad cryptography look identical to the untrained eye.
The idea behind Etebase is to make an easy-to-use API to enable developers to build encrypted apps, and it's based on our learnings from building EteSync[2] over the years. There's still a lot more to get done, but Etebase is already used in GNOME, KDE, Tasks.org, EteSync, and more. With libraries available for Rust, JavaScript/TypeScript, Java, Python and C/C++.
I hope Etebase will enable a new age of privacy-first and end-to-end encrypted applications, and that our data will no longer be used to track, analyze and manipulate us. Your data, yours only. Let's end-to-end encrypt everything!
I'd love to hear about your experience building encrypted applications. Got any questions? Suggestions? Please let me know!
Docs: https://docs.etebase.com/
[1] https://libsodium.org
[2] https://www.etesync.com