Real-time queries should be the default. Some kind of smart query invalidation (similar to incremental view maintenance) as a result of DML statements.
2. Can run in a web browser / mobile phone
Your client-side cache is usually a messy denormalization of your SQL database. For offline/local-first apps you essentially need to run your entire backend API in your browser. So you should be running an SQL database in the browser. For this to be seamless you need your database targetting JS during design stage.
3. Syncing
To support offline/local/mobile use cases, your database needs to be able to sync changes efficiently.
4. Logical-first data schema
You should be able to give your db a logical schema instead of a physical one, and the database can take care of the necessary denormalizations for performance reasons automatically.
5. Lower-level api for query planner
Instead of having SQL as your only interface, there should be a way to interact with the query tree and the query plan directly. People are always creating new query languages hacked over SQL (like Google's Logica).
6. Graph-based queries
Support a query language like Cypher. Provide better solutions for tree-based data structures - the queries get too unwieldy today.
A lot of the points you mention are really interesting to me, as I've been coming to similar conclusions recently. What are good choices that solve these particularly in the context of js/clientside apps?
absurd-sql[1] is the coolest thing I've seen in this space so far. sqlite.js brings sqlite to the browser via wasm, and then absurd-sql implements the storage layer in IndexedDb. So now we have an sqlite compatible db in the browser to play with.
There is also alasql[3] which is implemented in js, and lovefield[2] from Google which seems like an experiment that is now abandoned.
First, you could implement a REST/GraphQL cache in SQL. This would require maintaining mappings of your API response fields to SQL tables.
Going further, you could implement your backend API on the client, and have requests to it go directly to your browser SQL db. The benefit of this is you write your API once, and you get full offline support. If you don't need "local-first" then it's just a matter of figuring out when a query becomes invalid on the server. I could show instant results from local db, and then send a query to the server to ask if the data I already have in my cache can fulfill this request. Could optimize this on the server by listening to DML queries via WAL/LISTEN. WebSockets would be used to maintain real-time updates to only the data you are viewing.
You could also just use SQL directly as your UI API and then trigger these queries on the backend (respecting security of course).
What's doesn't feel optimal though is subscribing to sqlite.js updates in the browser. This makes me feel like we need an SQL db written in JS.
Also, if our DB is running in the same execution environment as our frontend and has one consumer, we could store each row as a JS object, and then reference this directly in our views, and subscribe to updates on a row-by-row basis. So if you are rendering a large table, if a column in a single row changes, when this row is updated in the database, instead of re-rendering the entire table manually, (or smartly detecting if the query was invalidated), you just bind to updates on the row instance, which is what the db is actually storing. I think this would reduce huge amounts of code.
Local-first is a little more difficult. There is some good exploration here[4].
I think writing a db in JS on the backend is not such a bad idea either. The hot paths would be written as Rust addons, but the majority of stuff people want today is just implementing the right logic to automatically handle denormalizations and subscriptions which we already hack together in our application layer using JS et al.
Thank you for the links, I've heard about some of these projects but will check out others. I've actually been working on a project that is running fully on front-end side and I'm leaning towards indexeddb+observability, but in my case there would be a lot of different data in one view and I wanted to see if I can materialize a view in worker and then push the deltas using messages. Also, while exploring it, I found out you could use service workers to create a virtual server [0] that would cache or produce views as needed, but that solution doesn't seem to be very popular.
Real-time queries should be the default. Some kind of smart query invalidation (similar to incremental view maintenance) as a result of DML statements.
2. Can run in a web browser / mobile phone
Your client-side cache is usually a messy denormalization of your SQL database. For offline/local-first apps you essentially need to run your entire backend API in your browser. So you should be running an SQL database in the browser. For this to be seamless you need your database targetting JS during design stage.
3. Syncing
To support offline/local/mobile use cases, your database needs to be able to sync changes efficiently.
4. Logical-first data schema
You should be able to give your db a logical schema instead of a physical one, and the database can take care of the necessary denormalizations for performance reasons automatically.
5. Lower-level api for query planner
Instead of having SQL as your only interface, there should be a way to interact with the query tree and the query plan directly. People are always creating new query languages hacked over SQL (like Google's Logica).
6. Graph-based queries
Support a query language like Cypher. Provide better solutions for tree-based data structures - the queries get too unwieldy today.