from Hacker News

Show HN: Snowflake warehouse implementation using DuckDB and Apache Iceberg

by buremba on 9/5/24, 12:19 AM with 0 comments

Hi all! I built this tool for data exploration on Snowflake tables without any need to run a warehouse. Snowflake is very powerful at scale but for small data (< 100GB) I rather work locally and then deploy my models to Snowflake later on. Universql lets you keep using Snowflake clients with its proxy layer and executes the queries locally on DuckDB.

This tech only became possible with Iceberg as it keeps the data references in manifest files where you can intelligently cache the data locally, similar to Snowflake's warehouse cache.