from Hacker News

Supabase Wrappers: A framework for building Postgres foreign data wrappers

by yurisagalov on 12/15/22, 3:59 PM with 46 comments

• by kiwicopple on 12/15/22, 4:02 PM

  hey hn, supabase ceo here

  this one might be more in of a “Show HN” because it’s a pre-release - something that you might want to try yourself or contribute to. The GitHub repo is here: https://github.com/supabase/wrappers

  For context, Postgres has Foreign Data Wrappers (FDW) [0]. These allow you to connect one Postgres database to another, and then query the secondary database directly in the first.

      CREATE FOREIGN TABLE other_database_table ( 
        id integer, 
        title text 
      )
      SERVER other_database;
  

  The data does not need to be “moved” to the primary database - it stays in the secondary and is fetched when you run a select query:

      select * from other_database_table;
  

  Our release today is a framework which extends this functionality to other databases/systems. If you’re familiar with Multicorn[1] or Steampipe[2], then it’s very similar. The framework is written in Rust, using the excellent pgx[3].

  We have developed FDWs for Stripe, Firebase, BigQuery, Clickhouse, and Airtable - all in various pre-release states. We'll focus on the systems we’re using internally while we stabalize the framework.

  There’s a lot in the blog post into our goals for this release. It’s early, but one of the things I’m most excited about.

  [0] Postgres FDW: https://www.postgresql.org/docs/current/sql-createforeigndat...

  [1] Multicorn: https://multicorn.org/

  [2] Steampipe: https://steampipe.io/

  [2] pgx: https://github.com/tcdi/pgx

• by atonse on 12/15/22, 4:16 PM

  This is wonderful!

  I love the pragmatism behind tools like osquery and steampipe that expose a lot of APIs as database tables. It makes these datasets available to non-programmers that are more comfortable with a database/tabular format.

  Is it fair to say though, that FDWs have to run as compiled shared libs? I've always wondered if there can be (like with VS code and language servers) a protocol where we can run a generic FDW that speaks a particular API over the network, and then you can build out-of-process connectors to it and just have to know the usual tools (HTTP, JSON, etc).

  Thoughts? Maybe this already exists.

  Then anyone could potentially build a bespoke API/dataset and just point an FDW to it.

• by sisve on 12/15/22, 9:20 PM

  So much great postgres stuff comming out from supabase. I'm really impressed with how much focus and direction the team have. So many companies would try to broaden their product/feature suite, but fail at understanding what made the core sucsessful and stretch the company in many directions. It really seems like they are just making them self better and better experts at postgres. And how to use it well. Really looking forward to see where this is going.

  On a sidenote, is not the wrappers for Airtable, BigQuery and ClickHouse opensourced? Or why did they skip that column in the second table?

  Is all of supabase opensourced now? I'm meaning I heard something about function not being opensourced but I can be remembering wrong. Most stuff is on GitHub I see

• by nixalaister on 12/15/22, 5:02 PM

  I'm excited about Wrappers because of an idea born in the GraphQL community: using GraphQL as the point of integration for many different services in your app. A use case GraphQL seemed made for. It's not without problems, though, such as when you need data from one of those services inside your database. So this takes that idea and moves it one level down into the database. You still get the benefits of a unified API (through REST or GraphQL) without the same limitations. I think there couldn't be a more ideal point of integration!

  disclaimer: I am a Supabase employee, but these views are my own.

• by claytongulick on 12/15/22, 6:24 PM

  I love Supabase and what they're doing! I evaluated them heavily when designing architecture for a healthcare product.

  I'm not sure about this one though - rust is a great systems language, but it wouldn't be my first choice for bridging the db <-> api gap.

  I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project. Python seems like a better choice of language for dealing with io bound problems like talking to remote APIs.

  Multicorn is mature, stable, well tested and has a ton of FDW implementations already.

  The dynamic nature of python simplifies the development/debug cycle, and serialization to/from JSON is easier than in any mainstream language except for javascript.

  I'd love to understand more about the technical rationale that drove this.

  [1] https://multicorn.org/

• by abuehrle on 12/15/22, 5:52 PM

  Sorry I couldn't figure this out from the docs, but Stripe data is queried "live" from Stripe, right? The abstraction is great, but won't this lead to unexpected N API calls when joining across my domain + Stripe?

• by thedangler on 12/15/22, 4:19 PM

  So you're telling me I can use this to create my own wrapper to get data directly from another API?

  I can now use this instead of creating my own workflow to get the data via an api which will be stored in the DB anyways?

• by jzelinskie on 12/15/22, 4:41 PM

  I really want to love foreign data wrappers for Postgres and this seems like a big improvement over existing Python library, but the lack of support for them in managed databases services makes them a non-starter for so many use-cases.

  Because RDS, for example, will only support the foreign data wrapper for reading from another "Postgres", what we really need is a server that supports the Postgres wire protocol (easier said than done) and you implement your drivers as a handler to that server.

• by jansommer on 12/16/22, 8:28 PM

  I'm eager to start using foreign data wrappers in PG, but there are close to zero benchmarks. Is my database going to break when transactions start piling up because an API I'm using is slow? Will my TPS drop significantly due to longer transactions? In other words, what am I paying for all this good stuff?

• by spankalee on 12/15/22, 10:54 PM

  How well do foreign data wrappers work with Postgres's query planner? Does't the planner need to estimate the cardinality of results from each table? Are the Supabase wrappers providing this info from the wrappers even though it might be hard to get from the data source?

  Also, is the Firebase connector for Realtime Database or Firestore?

• by dennisy on 12/15/22, 8:03 PM

  This seems cool to nerd out to! So thanks for that!

  But what is the point? Why is this better than making the API call?

  SQL is for queries, why would I want to do networking with it?