by nezaj on 8/22/24, 5:08 PM with 297 comments
Building modern apps these days involves a lot of schleps. For a basic CRUD app you need to spin up servers, wire up endpoints, integrate auth, add permissions, and then marshal data from the backend to the frontend and back again. If you want to deliver a buttery smooth user experience, you’ll need to add optimistic updates and rollbacks. We do these steps over and over for every feature we build, which can make it difficult to build delightful software. Could it be better?
We were senior and staff engineers at Facebook and Airbnb and had been thinking about this problem for years. In 2021, Stopa wrote an essay talking about how these schleps are actually database problems in disguise [1]. In 2022, Stopa wrote another essay sketching out a solution with a Firebase-like database with support for relations [2]. In the last two years we got the backing of James Tamplin (CEO of Firebase), became a team of 5 engineers, pushed almost ~2k commits, and today became open source.
Making a chat app in Instant is as simple as
function Chat() {
// 1. Read
const { isLoading, error, data } = useQuery({
messages: {},
});
// 2. Write
const addMessage = (message) => {
transact(tx.messages[id()].update(message));
}
// 3. Render!
return <UI data={data} onAdd={addMessage} />
}
Instant gives you a database you can subscribe to directly in the browser. You write relational queries in the shape of the data you want and we handle all the data fetching, permission checking, and offline caching. When you write transactions, optimistic updates and rollbacks are handled for you as well.Under the hood we save data to postgres as triples and wrote a datalog engine for fetching data [3]. We don’t expect you to write datalog queries so we wrote a graphql-like query language that doesn’t require any build step.
Taking inspiration from Asana’s WorldStore and Figma’s LiveGraph, we tail postgres’ WAL to detect novelty and use last-write-win semantics to handle conflicts [4][5]. We also handle websocket connections and persist data to IndexDB on web and AsyncStorage for React Native, giving you multiplayer and offline mode for free.
This is the kind of infrastructure Linear uses to power their sync and build better features faster [6]. Instant gives you this infrastructure so you can focus on what’s important: building a great UX for your users, and doing it quickly. We have auth, permissions, and a dashboard with a suite tools for you to explore and manage your data. We also support ephemeral capabilities like presence (e.g. sharing cursors) and broadcast (e.g. live reactions) [7][8].
We have a free hosted solution where we don’t pause projects, we don’t limit the number of active applications, and we have no restrictions for commercial use. We can do this because our architecture doesn’t require spinning up a separate servers for each app. When you’re ready to grow, we have paid plans that scale with you. And of course you can self host both the backend and the dashboard tools on your own.
Give us a spin today at https://instantdb.com/tutorial and see our code at https://github.com/instantdb/instant
We love feedback :)
[1] https://www.instantdb.com/essays/db_browser
[2] https://www.instantdb.com/essays/next_firebase
[3] https://www.instantdb.com/essays/datalogjs
[4] https://asana.com/inside-asana/worldstore-distributed-cachin...
[5] https://www.figma.com/blog/how-figmas-multiplayer-technology...
[6] https://www.youtube.com/live/WxK11RsLqp4?t=2175s
by jamest on 8/22/24, 6:00 PM
Good luck, Joe, Stopa and team!
by sibeliuss on 8/22/24, 10:58 PM
by codersfocus on 8/22/24, 8:15 PM
Unfortunately this area is still immature, and there aren't really great options but PowerSync was the least bad. I'll probably pair it with Supabase for the backend.
by swalsh on 8/23/24, 12:30 PM
by blixt on 8/23/24, 5:15 AM
Does Instant have a way to merge many frequent updates into fewer Postgres transactions while maintaining high frequency for multiplayer?
Regardless this is super cool for so many other things where you’re modifying more regular app data. Apps often have bugs when attempting to synchronize data across multiple endpoints and tend to drift over time when data mutation logic is spread across the code base. Just being able to treat the data as one big object usually helps even if it seems to go against some principles (like microservices but don’t get me started on why that fails more often than not due to the discipline it requires).
by lewisl9029 on 8/23/24, 12:45 AM
Apparently I signed up for Instant previously but completely forgot about it. Only realized I had an account when I went to the dashboard to find myself still logged in. I dug up the sign up email and apparently I signed up back in 2022, so some kind of default invalidation period on your auth tokens would definitely make me a bit more comfortable.
Regardless, I'm still as excited about the idea of a client-side, offline-first, realtime syncing db as ever, especially now that the space has really been picking up steam with new entrants showing up every few weeks.
One thing I was curious about is how well the system currently supports users with multiple emails? GitHub popularized this pattern, and these days it's pretty much table stakes in the dev tools space to be able to sign in once and use the same account across personal accounts and orgs associated with different emails.
Looking at the docs I'm getting the sense that there might be an assumption of 1 email per user in the user model currently. Is that correct? If so, any plans to evolve the model to become more flexible?
by coffeemug on 8/22/24, 8:32 PM
by antidnan on 8/22/24, 6:59 PM
Congrats team!
by breatheoften on 8/22/24, 6:47 PM
One of the things I find quite nice about firebase is the quite powerful separation between the logic of data retrieval / update and the enforcement of access policy -- if you understand it you can build the prototype on a happy path with barely any authorization enforcement and then add it later and have quite complete confidence that you aren't leaking data between users or allowing them to change something they shouldn't be able to. Although you do need to keep the way this system works in mind as you build and I have found that developers often don't really grasp the shape of these mechanisms at first
From what I can tell -- the instant system is different in that the permission logic is evaluated on the results of queries -- vs firebase which enforces whether the query is safe to run prior to it even being executed ...
by the_duke on 8/22/24, 8:53 PM
Postgres also isn't terrible, but also not brilliant for that use case.
How has your experience been in that regard?
by remolacha on 8/22/24, 8:00 PM
In ActiveRecord, I can do this:
```rb
post = Post.find_by(author: "John Smith")
post.author.email = "john@example.com"
post.save
```
In React/Vue/Solid, I want to express things like this:
```jsx
function BlogPostDetailComponent(...) {
// `subscribe` or `useSnapshot` or whatever would be the hook that gives me a reactive post object
const post = subscribe(Posts.find(props.id));
function updateAuthorName(newName) {
// This should handle the join between posts and authors, optimistically update the UI
post.author.name = newName;
// This should attempt to persist any pending changes to browser storage, then
// sync to remote db, rolling back changes if there's a failure, and
// giving me an easy way to show an error toast if the update failed.
post.save();
}
return (
<>
...
</>
)
}```
I don't want to think about joining up-front, and I want the ORM to give me an object-graph-like API, not a SQL-like API.
In ActiveRecord, I can fall back to SQL or build my ORM query with the join specified to avoid N+1s, but in most cases I can just act as if my whole object graph is in memory, which is the ideal DX.
by w10-1 on 8/22/24, 8:22 PM
Other datalog engines support recursive queries, which makes my life so much easier. Can I do that now with this? Or is it on the roadmap?
I have fairly large and overlapping rules/queries. Is there any way to store parsed queries and combine them?
Also, why the same name as the (Lutris) Enhydra java database? Your domain is currently listed as a "failed company" from 1997-2000 (actual usage of the Java InstantDB was much longer)
https://dbdb.io/db/instantdb
Given that it's implemented clojure and some other datalog engines are in clojure, can you say anything about antecedents?Some other Clojure datalog implementations, most in open source
- Datomic is the long-standing market leader
- XTDB (MPL): https://github.com/xtdb/xtdb
- Datascript (EPL): https://github.com/tonsky/datascript
- Datalevin ((forking datascript, EPL): https://github.com/juji-io/datalevin
- datahike (forking datascript, EPL): https://github.com/replikativ/datahike
- Naga (EPL): https://github.com/quoll/naga
by webdevladder on 8/22/24, 6:00 PM
In the docs[1]:
> Instant uses a declarative syntax for querying. It's like GraphQL without the configuration.
Would you be interested in elaborating more about this decision/design?
by remolacha on 8/22/24, 6:02 PM
I appreciate that you're thinking about relational data and about permissions. I've seen a bunch of sync engine projects that don't have a good story for those things.
imo, the more that you can make the ORM feel like ActiveRecord, the better.
by TeeWEE on 8/23/24, 4:45 AM
However for our use case we want total control over the server database. And wanted to store it in normalized tables.
The solution we went for us is streaming the mutation stream (basically the WAL) from/to client and server. And use table stream duality to store them in a table.
Permissions are handled on a table level.
When a client writes it sends a mutation to the servers. Or queues it locally if offline. Writes never conflict: we employ a CRDT “last write wins” policy.
Queries are represented by objects and need to be implemented both in Postgres as wel as SQLLite (if you want offline querying, often we don’t). A query we implement for small tables is: “SELECT *”.
Note that the result set being queried is updated realtime for any mutation coming in.
It’s by default not enforcing relational constraints on the clientside so no rollbacks needed.
However you can set a table in different modes: - online synchronous writes only: allows us to have relational constraints. And to validate the creation against other server only business rules.
The tech stack is Kotlin on client (KMM) and server, websocket for streaming. Kafka for all mutations messaging. And vanilla Postgres for storing.
The nice thing is that we now have a Kafka topic that contains all mutations that we can listen to. For example to send emails or handle other use cases.
For every table you: - create a serializable Kotlin data class - create a Postgres table on the server - implement reading and writing that data, and custom queries
Done: the apps have offline support for reading a single entity and upserts. Querying require to be online if not implemented on the client.
by RoboTeddy on 8/24/24, 6:18 PM
(2) When a schema is provided, is it fully enforced? Is there a way to do migrations?
Migrations are the only remaining challenge I can think of that could screw up this tool long-term unless a good approach gets baked in early. (They're critically important + very often done poorly or not supported.) When you're dealing with a lot of data in a production app, definitely want some means of making schema changes in a safe way. Also important for devex when working on a project with multiple people — need a way to sync migrations across developers.
Stuff like scalability — not worried about that — this tool seems fundamentally possible to scale and your team is smart :) Migrations though... hope you focus on it early if you haven't yet!
by taw1285 on 8/22/24, 6:46 PM
by projektfu on 8/22/24, 6:52 PM
by monomers on 8/23/24, 8:30 AM
Say I have an InstantDB app, can I stream events from the instant backend to somewhere else?
by IanCal on 8/22/24, 5:58 PM
I'm not sure about how things grow from here in terms of larger aggregates and more complex queries though so am slightly worried I'm painting myself into a corner. Do you have any guides or pointers here? Or key areas people shouldn't use your db?
by the_king on 8/23/24, 3:47 AM
For example, creating a user table and ensuring that emails are unique - I've done it 50 times with Postgres. Is that part built out yet?
Very cool. Appreciate the "Don't Make Me Think" API.
(written with aqua)
by cheema33 on 8/23/24, 4:07 PM
In other words it primarily targets brand new projects or projects that can completely migrate away from their current database?
by kabes on 8/23/24, 11:53 AM
by ochiba on 8/22/24, 8:32 PM
I haven't looked in detail yet — what are the main differences relative to Zero?
by jeromechoo on 8/22/24, 6:46 PM
It was so fast I was able to build basic collision physics of letter tiles and have their positions sync to multiple clients at once. What a shame to be killed by Google.
I haven't had a need for real time databasing since, but this is inspiring me to build another collaborative app.
by mattfrommars on 8/23/24, 12:15 AM
How do I need to start thinking conceptually for this, InstantDB or Firebase concept to kick in?
Say for a collaborative text editor, I'd use off the shelf CRDT Javascript implementation.
by jgeurts on 8/25/24, 12:34 AM
by p2hari on 8/22/24, 8:00 PM
by DandyDev on 8/22/24, 5:56 PM
by piyushtechsavy on 8/23/24, 7:40 AM
by koito17 on 8/22/24, 8:17 PM
I wouldn't mind using the Datalog syntax as-is since I have some experience using Clojure with Datomic, but it did surprise that someone would decide to use this syntax over a syntax used in other Datalog engines (and predating Datomic itself).
by h4ny on 8/23/24, 3:02 PM
Some super minor feedback appended. All the best with InstantDB!
There is a missing word in the message that appears after clicking on the "Create an app" button:
> With that one-click you’ve claimed an id that you can use for storing your data. Now we'll show you [how] to wire up your db to an app and start adding data. Check out the walkthrough below on your left with the full code and preview on the right.
Also on smaller screen sizes there is no left and right. :)
by cheema33 on 8/22/24, 10:17 PM
Side note: Electric SQL is currently going through a rewrite, so things are a bit up in the air.
by deepsun on 8/23/24, 1:39 AM
Data always needs a schema (unless it's random bits aka max entropy). The question is just where it's managed and enforced.
by Kiro on 8/22/24, 7:41 PM
by robinpdx on 8/22/24, 6:49 PM
I’ve built a prototype of a full stack using Flask + SocketIO + SQLite as well as an iOS client to prove the concept to the VPs.
How well do you think this can scale? Any plans to make native SDKs for iOS and Android?
by agambrahma on 8/22/24, 8:37 PM
Clojure + TS seems to be a good way to go, without being hung up on CLJS.
by kachapopopow on 8/23/24, 4:56 PM
Why not the standard node.js with shared module? Assuming performance is not the primary goal.
Why not generated rust structures from model file and a rust server? Assuming performance is the primary goal.
Why not a jvm with a lightweight runtime? (Assuming instancing is used for scale here, a lot of wasted ram usage here)
by techn00 on 8/22/24, 8:29 PM
by mizzao on 8/23/24, 2:32 PM
However, it didn't scale well in terms of performance to large numbers of users.
Would anyone have thoughts on comparisons to Meteor?
by j_san on 8/23/24, 9:55 AM
by TripleChecker on 8/22/24, 10:16 PM
by terpimost on 8/23/24, 12:31 AM
by aidos on 8/22/24, 8:10 PM
Noticed in your docs you say that Hasura uses RLS for permissions but that’s not true. They have their own language for effectively specifying the filters to apply on a query. It’s a design decisions that allows them to execute the same query for all connected clients at the same time using different parameters for each one.
by punnerud on 8/23/24, 6:30 AM
by k__ on 8/22/24, 7:54 PM
Can the backend be replaced?
by legohorizons on 8/22/24, 6:29 PM
by samstave on 8/22/24, 11:02 PM
https://blog.sergeantbiggs.net/posts/aerc-a-well-crafted-tui...
It seems that building a version of this Aerc Email TUi with Instant is a completely doable?
Might be an interesting tutorial to build out an Instant FroBenDB (Instant is an instant Front-BackendDB :-) --- btu the txtual nature of aerc and its configs seem ripe for just bolting it to Instant.
by gr4vityWall on 8/23/24, 2:49 AM
I wish you the best of luck, having a real-time database on the client does make things much easier.
by ripped_britches on 8/23/24, 12:10 AM
by chrysoprace on 8/23/24, 12:25 AM
by wheelerwj on 8/22/24, 6:02 PM
I think this looks like, a backend-end-in-box type of product? So that you just have to focus on front end mostly?
Could be cool for early stage projects.
by thruflo on 8/22/24, 8:31 PM
by notsahil on 8/22/24, 6:14 PM
by avinassh on 8/22/24, 6:10 PM
can you elaborate more on how you achieve this
by xnx on 8/22/24, 5:46 PM
by sgonz on 8/22/24, 6:32 PM
by truetraveller on 8/23/24, 4:38 AM
by dsmurrell on 8/23/24, 8:15 AM
Hasura offer a self hosted solution so that I know if they decide to close shop for whatever reason, I'm not stuck in the lurch and have to reengineer my entire solution. Do you offer this now or are you planning to?
by superfunguy_94 on 8/23/24, 7:33 AM
by manx on 8/23/24, 7:39 AM
by warden_2003 on 8/22/24, 5:10 PM
by h_tbob on 8/22/24, 6:14 PM
by kentor on 8/29/24, 7:02 AM
by eashish93 on 8/23/24, 1:18 PM
by reichertjalex on 8/23/24, 3:28 PM
tl;dr -- I'm a big fan :)
by aipro778 on 8/26/24, 5:11 PM
by unsupp0rted on 8/25/24, 9:41 AM
Would you be able to add examples for Vue JS?
by novoreorx on 8/23/24, 4:44 AM
by yblu on 8/25/24, 5:39 AM
by hardwaresofton on 8/23/24, 5:15 AM
It seems like that’s what would do best in the marketplace… people seem to be fine with the API of firebase and just want it to be cheaper
by bigblind on 8/22/24, 7:37 PM
I'm slightly worried about permissions evaluating to "true" if they're not specified. I think this will lead to a lot of actions being accidentally allowed.
by Chipshuffle on 8/24/24, 5:55 AM
by buf on 8/23/24, 2:07 PM
by serial_dev on 8/22/24, 7:13 PM
by _kidlike on 8/22/24, 8:38 PM
by jakubmazanec on 8/23/24, 8:21 PM
by Alifatisk on 8/22/24, 10:04 PM
by Hadriel on 8/23/24, 6:15 AM
by intellix on 8/23/24, 8:48 AM
by ibash on 8/22/24, 7:26 PM
by charlie0 on 8/22/24, 6:51 PM
by android521 on 8/22/24, 6:53 PM
by maninblackv2 on 8/23/24, 5:13 AM
by softlylove on 8/23/24, 1:39 AM
by thomasfl on 8/23/24, 9:10 AM
by mr-pink on 8/22/24, 11:06 PM
by bpiroman on 8/23/24, 4:31 AM
by Lionga on 8/23/24, 11:06 AM
by patrickaljord on 8/23/24, 2:06 PM
by otteromkram on 8/23/24, 3:22 AM
I know it's "just a paycheck," but I despise both of these companies, so I'll probably pass on this.
The usage of "schlep" so freely is also a bit unsettling.
Good luck!
by pyryt on 8/22/24, 6:56 PM
by bnormative on 8/22/24, 7:26 PM