from Hacker News

SQLFlow: DuckDB for Stream Processing – 45k messages/second

by dm03514 on 3/29/24, 12:16 PM with 1 comments

• by dm03514 on 3/29/24, 12:16 PM

  Hello Everyone, having lots of fun figuring out ways to use DuckDB in common data operations. Already seeing huge savings in traditional data engineering applications replacing data warehouse queries with duckdb over blob storage (parquet).

  SQLFlow uses duckdb to process kafka streaming data.

  The goal is to create an environment where stream logic can be expressed in pure SQL with a simple and performant deployment, leveraging kafka consumer groups for state management.

  The new in-memory processor easily handles 45,000 messages / second for simple aggregation.

  https://github.com/turbolytics/sql-flow?tab=readme-ov-file#b...

  ------

  I'd really appreciate feedback on features that would make this more useful. It's basically just a POC at this point.

  Thank you