from Hacker News

Litestream: Revamped

by usrme on 5/20/25, 7:58 PM with 99 comments

• by maxmcd on 5/20/25, 8:12 PM

  Looks like the code is here: https://github.com/benbjohnson/litestream/tree/v0.5

  Really nice to see this, I wrote this comment almost 2 years ago when I was a little miffed about trying to use litestream and litefs: https://news.ycombinator.com/item?id=37614193

  I think this solves most of the issues? You can now freely run litestream on your DB and not worry about issues with multiple writers? I wonder how the handoff is handled.

  The read replica FUSE layer sounds like a real nice thing to have.

  edit: Ah, it works like this: https://github.com/benbjohnson/litestream/pull/617

  > When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.

  Sounds workable!

• by simonw on 5/20/25, 9:01 PM

  This post is like they read my mind and implemented everything I wanted from a new Litestream. So exciting.

• by thewisenerd on 5/20/25, 10:37 PM

  ben, thanks for litestream!

  we're using it on production for a write-heavy interal use-case (~12GB compressed) for more than a year now; and it's costing us a couple hundred pennies per month (azure).

  excited to try the new changes when they land.

• by bradgessler on 5/20/25, 10:34 PM

  I wish Fly would polish the developer experience on top of SQLite. They're close, but it's missing:

  1. A built-in UI and CLI that manages SQLite from a volume. Getting the initial database on a Fly Machine requires more work than it should.

  2. `fly console` doesn't work with SQLite because it spins up a separate machine, which isn't connected to the same volume where the SQLite data resides. Instead you have to know to run `fly ssh console —pty`, which effectively SSH's into the machine with the database.

  The problem in general with SQLite web apps is they tend to be small apps, so you need a lot of them to make a decent amount of money hosting them.

• by jasonthorsness on 5/20/25, 8:52 PM

  What a coincidence, I was just researching Litestream today! I use Sqlite on my VPS and was thinking about adding this.

  Am I understanding correctly that I will be able to restore a database to any point-in-time that is while the litestream process is running? Because auto-checkpointing could consume the WAL while it isn't running?

  So for an extreme example if the process crashed for an hour between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the information required to restore between 2:00 and 3:00 is lost?

• by psanford on 5/20/25, 9:07 PM

  This looks great! A few years ago I wrote a sqlite vfs for using dynamodb as a backing store[0] called DonutDB. With the recent addition of CAS to S3, I was thinking about making a new version of DonutDB backed by S3. I'm really glad lightstream supports this so I don't have to!

  I can't wait to try this out.

  [0]: https://github.com/psanford/donutdb

• by ignoramous on 5/20/25, 9:37 PM

    We have a sneaking suspicion that the robots that write LLM code are going to like SQLite too. We think what coding agents like Phoenix.new want is a way to try out code on live data, screw it up, and then rollback both the code and the state.
  

  Prescient.

  Agents would of course work well if they can go back in time to checkpoints and branch from there, exploring solutions parallely as needed.

  Anyone who has experience with building workflows (Amazon SWF, Temporal, and the like) knows how difficult it is to maintain determinism in face of retries & re-drives in multi-tier setups (especially, those involving databases).

  Replit recently announced their Agent's integration with Neon's time travel feature [0] for exactly the purpose outlined in TFA. Unlike Fly.io though, Replit is built on GCP and other 3p providers like Neon and it is unclear if both GCP & Databricks won't go all Oracle on them.

  [0] https://blog.replit.com/safe-vibe-coding

• by hobo_mark on 5/20/25, 10:37 PM

  If you wanted to use litestream to replicate many databases (ideally, one or more per user), which is one of the use cases described here (and elsewhere), how do you tell litestream to add new databases dynamically? The configuration file is static and I haven't found an API to tell it to track a new db at runtime.

• by danielblignaut on 5/30/25, 12:14 PM

  Have these changes landed yet? I don't see any updates in the documentation website around some of these features. Most notably, I'm interested in read replication and the ability for a read replica to take over becoming the leader (even if it requires a config update and litestream restart) in cases where we aren't using Consul (which is why LiteFS is not a workable solution in our environment).

• by srameshc on 5/20/25, 8:21 PM

  I have been following Ben for a long time but I never knew LiteFS was based on his work. I somehow settled eventually for rqlite for self managed distributed.

  https://github.com/rqlite/rqlite

• by rads on 5/21/25, 2:47 AM

  What will be required from users of the existing Litestream version to upgrade to the new one? Is it a matter of bumping the version when it comes out or is there more to it?

• by wg0 on 5/21/25, 4:40 PM

  > Now that we’ve switched to LTX, this isn’t a problem any more. It should thus be possible to replicate /data/*.db, even if there’s hundreds or thousands of databases in that directory.

  That was the show stopper. Now multi tenant with per tenant database whee (in theory) each user can roll back to certain point in time or at least completely download their database and take away for whatever they want to do with it is going to be possible.

• by wim on 5/21/25, 5:57 PM

  Very cool! This is so clever and makes deploying it so simple. I just wasn't able to use it yet because we'd have (many) thousands of SQLite DBs to backup. I quickly hacked something together using fanotify + SQLite's Backup API to have some copies at least, but I'm going to try to switch to Litestream if this amount of files would be supported by the wildcard replication.

• by JSR_FDED on 5/21/25, 4:30 AM

  Will the new litestream work with object stores that don’t provided conditional writes?

• by Skinney on 5/21/25, 10:22 AM

  If I’m deploying a new version of my app, the typical managed solution will spawn a new server instance with that new version, and once a health check has succeeded a couple of times it will reroute trafic to this new instance and kill the old one.

  Previously this would be problematic, as the new instance might miss changes made by the old server. Is this fixed by these new changes?

• by malkia on 5/21/25, 12:18 AM

  So fossil (which is built on top of sqlite) + this = SCM?

• by neom on 5/20/25, 10:56 PM

  Is Litestream on a path to subsume LiteFS's capabilities? Re: PITR, would this be used to facilitate automated A/B testing of AI-generated code changes against live data subsets? I can imagine a lot of cool stuff in that direction. This is really cool Ben!

• by j0e1 on 5/20/25, 9:48 PM

  This is exciting! Especially glad that Litestream is still maintained. Is there a use-case for Litestream for more than backup? I am a fan of offline-first but it would be cool to have a way to synchronize on-device SQLite instances to a single central instance.

• by mythz on 5/21/25, 3:31 AM

  Awesome stuff, this resolves my #1 feature request of being able to replicate an entire directory of SQLite *.db's from a single Litestream process - happy it's finally here.

  Should make replicating Multi tenant per-user SQLite databases a lot more appealing.

• by bambax on 5/21/25, 6:53 AM

  Very cool!

  There may be a typo here:

  > The most straightforward way around this problem is to make sure only one instance of Litestream can replication to a given destination.

  Can replicate? Or can do replications?

• by wiradikusuma on 5/21/25, 6:23 AM

  For Fly.io employees here: Can I finally replace my Postgre with this a'la Cloudflare D1 (which is also Sqlite based)?

• by rawkode on 5/20/25, 9:09 PM

  Amazing to see and hear about the progress. Always a pleasure when Ben works on something and shares it. Keep it up!

• by Nelkins on 5/21/25, 5:41 PM

  Does anybody have a list of which S3-compatible object storage providers support Compare-And-Swap?

• by nodesocket on 5/20/25, 10:11 PM

  Is there a migration guide from stable to the branch 0.5? I’m running Litestream as a Docker sidecar alongside my Python app container and it’s been great and a nice comfort knowing my SQLite db is backed up to S3.

• by oliwary on 5/20/25, 10:37 PM

  Fantastic to see it's getting updated! I am a big fan of litestream, have been using it for a while together with pocketbase. It's like a cheat code for a cheap, reliable and safe backend.

• by fra on 5/20/25, 8:56 PM

  Litestream has seen very little development lately and I was worried it was dead. Very glad to see Ben Johnson is continuing to push the project forward with some exciting new plans.

• by ChocolateGod on 5/21/25, 7:46 AM

  > It will be able to fetch and cache pages directly from S3-compatible object storage.

  Does this mean your SQLite database size is no longer restricted by your local disk capacity?

• by dastbe on 5/21/25, 2:49 AM

  asking since ben does take a look here...

  will revamped litestream have a solution for ACKing only when transactions have durably committed to storage?

• by caleblloyd on 5/20/25, 9:57 PM

  Is the backend pluggable? Could it be configured to write to any key value store with support for optimistic concurrency control?

• by yowmamasita on 5/21/25, 12:05 AM

  tangent: in modern SQLite, are writes still serialized? That's my main concern when choosing a tech stack for an app that might have thousands of writes happening on peak periods

• by m3sta on 5/21/25, 1:23 AM

  Is there anything like Livestream that can be just pip installed?

• by nico on 5/20/25, 9:17 PM

  Very cool idea, I wonder if that works better than their Postgres instances

  Recently, I deployed a little side project using a small postgres vm on fly.io After a couple of days, and only having about 500kb of data stored in that db, the postgres vm went into an unrecoverable fail loop, saying it ran out of memory, restarting, then immediately running out of memory again, so on and so forth

  It took about 3-4hrs to recover the data jumping through a lot of hoops to be able to access the data, copy it to another volume and finally download it

  I would've reached for support, but it seems like the only option available is just posting on their forum. I saw a couple of related posts, all with unsatisfactory answers unfortunately

  To be fair, it was incredibly easy to get up and running with them. On the other hand, almost all the time I saved by that quick start, was wasted recovering the failing db, all the while my site was down

  Ironically, I originally developed the project using sqlite, but then switched to postgres to deploy

• by gavinray on 5/20/25, 8:19 PM

  Just a heads-up, the link in the "Litestream is fully open source" callout is malformed and leads to:

  https://http//litestream.io/

• by mixmastamyk on 5/20/25, 9:37 PM

  For something rather new there seems to be too many choices already. Please pick a strategy under one name, good defaults, and a couple of config options.