by frant1c on 6/7/21, 4:31 PM with 73 comments
by dgreensp on 6/8/21, 12:26 AM
I think it's correct that, ideally, there would be a framework that handles real-time collaboration, undo/redo, and offline support for you, and then you build your app with these problems already solved. I will probably create such a framework eventually. I don't see it as a database engineering problem, it's more like a framework or application architecture, which every app like Google Docs or Figma has its own version of. Writing such a framework is not too much harder than writing such an app, it just requires a little more abstraction and some documentation.
If you've never written an undo manager, sync engine, etc., and you aren't writing a complex app, it's hard to arrive at the right design by thinking about pure data sync. Also, storing and querying data are solved problems; it's more a question of coming up with a generic data model for an application and defining its semantics.
by rektide on 6/7/21, 6:36 PM
Google Docs is themselves moving to canvas based rendering, which might as well be turning the screen into a giant VNC session into their codebases[1]. Web, dead, pushing pixels in people's faces, in. All the extensions that extend & enhance Google Docs are about to die, being replaced with a very small, much narrower API provided explicitly by google.
I see statements like,
> This gets me excited about whats to come, because what's at the edge of difficulty today tends to become the new normal tomorrow.
And think, please, let us not be path dependent on a web-recreation of classic desktop apps. The web is more interesting, it has online content, liveliness & connectivity to it that far surpasses the other platform's norms. Let us think of how we might advance the web for good web things. There's worth & value to examining hard problems, but I am worried that this attitude has us set out to build faster horses.
by vaughan on 6/8/21, 6:22 PM
There is a moment when you step back and realize that all the data-wrangling code we write in apps is essentially what an SQL query planner does. And our frontend is just one big materialized view that we need to update.
I often think that if we had a database that runs in the browser, and we could subscribe to queries, and missing local data would be fetched on-demand, frontend development would be a lot easier. Its of course much more complicated than that.
I think the reality though is that backend scaling requirements always end up dwarfing frontend productivity concerns. And there is also a huge amount of glue between backend data sources, preventing the creation of a clean database on the backend to sync with, meaning we are creating API gateways and GraphQL federation layers, and all we can hope for is a good client-side caching layer.
If you look deeper though you will find many projects doing all these kinds of things already. Mobile developers are very familiar with offline techniques and SQLite on the client, and it doesn't feel like anything special to them. For the web/desktop, maybe no one has packaged it in the right way yet, or maybe we are still digesting GraphQL, SSR, and serverless, and then there will be another shift with offline-first, reactive SQL in the frontend.
It's funny though that if you think long enough about these concerns, you always seem to end up wanting some Datalog thing.
by ampdepolymerase on 6/7/21, 10:55 PM
by code-is-code on 6/7/21, 11:37 PM
by eyelovewe on 6/7/21, 9:14 PM
by Cthulhu_ on 6/8/21, 2:13 PM
I really liked this post, because it touches on so many things that I have to build as well at the moment. I'm building a configuration management interface; the front-end is basically authentication and heaps of forms, the back-end transforms it into XML and uses some shell scripts to rsync them to servers and SNMP to trigger a deployment. But the users have worries about overwriting each other's work, they ask for undo / revert support, and there's things like audit logging, user and permissions management, etc involved.
At the moment I'm slowly building all of that with a React/TS front end and a Go backend using a REST API, just trying to be as tidy and complete as possible. But it's a lot of work, and I'm afraid that once I get to things like versioning, locking, undo / revert, auditing and permissions, my fairly straightforward codebase will just explode in complexity, with each endpoint having loads of documentation and code to represent the different concerns. Client- and server-side validation is another concern, importing existing data, migrating user data from a staging to production environment, etc.
It's a lot. It's a project that should be done by a full team of developers, maybe even multiple but ATM I'm doing it on my own.
I'll never finish it. I'm currently trying to plough through and make sure everything is set up, but I'm hoping we'll get a big financial injection and I'd be able to hire a bunch of consultants (that I'm familiar with as being decent if expensive developers).
I don't know how the previous guy managed to get as far as he did on the older version, other than being a mediocre but productive and persistent developer for all that time, seeing the whole thing grow over time instead of trying to reach feature parity with a product nine years in the making.
by whoisnnamdi on 6/7/21, 6:41 PM
Thanks so much for writing it!
by ralusek on 6/8/21, 2:39 PM
by abss on 6/8/21, 7:45 PM
by tlarkworthy on 6/8/21, 4:36 PM
It's got: offline first, latency compensation, pubsub, partial sync, authorization rules and serverside timestamps, and global transactions, and 5 9s availability backed by spanner for umpteen languages. https://tomlarkworthy.endpointservices.net/blogs/firestores-...
by j-pb on 6/8/21, 4:00 PM
Funnily enough we are targeting the browser AND FPGAs, and I think the latter is the much more interesting use case for a distributed reactive CRDT-like database.
Datalog is actually trivial to incrementally materialize in open world semantics. Which is why datomic (while a great foundation in theory) turns out to be non ideal. TxnIDs and retractions are essentially nonmononic negation in disguise, and CALM (consistency as logical monotonicity) a.k.a. distributedness, doesn't go well with that.
Seeing that we're not the only ones dreaming of this gives me hope though, that we might get out of the tar pit someday.
by zubairq on 6/8/21, 3:32 PM
by twobitshifter on 6/8/21, 4:36 PM
No sure on the scalability of SQLite but here’s a simple way to get database undo/redo.
by enos_feedler on 6/8/21, 11:09 PM
by farnulfo on 6/8/21, 8:37 PM
by thewakalix on 6/9/21, 12:01 AM
by nickthemagicman on 6/9/21, 12:24 AM
What is this hosted on?
by psvj on 6/8/21, 11:33 PM