by mritchie712 on 1/26/25, 11:23 AM
by dm03514 on 1/26/25, 1:10 PM
I think iceberg solves a lot of big data problems, for handling huge amounts of data on blob storage, including partitioning, compaction and ACID semantics.
I really like the way the catalog standard can decouple underlying storage as well.
My biggest concern is how inaccessible the implementations are, Java / spark has the only mature implementation right now,
Even DuckDB doesn’t support writing yet.
I built out a tool to stream data to iceberg which uses the python iceberg client:
https://www.linkedin.com/pulse/streaming-iceberg-using-sqlfl...
by gopalv on 1/26/25, 2:32 AM
Hidden partitioning is the most interesting Iceberg feature, because most of the very large datasets are timeseries fact tables.
I don't remember seeing that in Delta Lake [1], which is probably because the industry standard benchmarks use date as a column (tpc-h) or join date as a dimension table (tpc-ds) and do not use timestamp ranges instead of dates.
[1] - https://github.com/delta-io/delta/issues/490
by teleforce on 1/26/25, 5:13 AM
by pradeepchhetri on 1/26/25, 7:17 AM
by volderette on 1/26/25, 6:37 AM
How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option.
[1] https://www.starrocks.io/
by crorella on 1/26/25, 10:55 AM
What I like about iceberg is that the partitions of the tables are not tightly coupled to the subfolder structure of the storage layer (at least logically, at the end of the day the partitions are still subfolders with files), but at least the metadata is not tied to that, so you can change the partition of the tables going forward and still query a mix of old and new partitions time ranges.
In the other hand, since one of the use cases they created it at Netflix was to consume directly from real time systems, the management of the file creation when updates to the data is less trivial (the CoW vs MoR problem and how to compact small files) which becomes important on multi-petabytes tables with lots of users and frequent updates. This is something I assume not a lot companies put a lot of attention to (heck, not even at Netflix) and have big performance and cost implications.
by varsketiz on 1/26/25, 1:53 AM
I'm somewhat surprised to see it here - Iceberg is around for some time already.
by nikolatt on 1/26/25, 8:22 AM
I've been looking at Iceberg for a while, but in the end went with Delta Lake because it doesn't have a dependency on a catalog. It also has good support for reading and writing from it without needing Spark.
Does anyone know if Iceberg has plans to support similar use cases?
by apwell23 on 1/26/25, 2:37 PM
I am stockholder in snowflake and iceberg's ascendance seems to coincide with snow's downfall.
Is the query engine value add justify snowflake's valuation. Their data marketplace thing didn't seem to have actually worked.
by mkl95 on 1/26/25, 8:19 AM
Iceberg on S3 tables is going to be a hot topic in the next few years.
by npalli on 1/26/25, 7:20 PM
Are there robust non-JVM based implementations for Iceberg currently? Sorry to say, but recommending JVM ecosystems around large data just feels like professional malpractice at this point. Whether deployment complexity, resource overhead, tool sprawl or operational complexity the ecosystem seems to attract people who solve only 50% of the problem and have another tool to solve the rest, which in turn only solves 50% etc.. ad infinitum. The popularity of solutions like Snowflake, Clickhouse, or DuckDB is not an accident and is the direction everything should go. I hear Snowflake will adopt this in the future, that is good news.
by rdegges on 1/26/25, 3:12 PM
by chehai on 1/26/25, 10:15 PM
In order to get good query performance from Iceberg, we have to run compaction frequently. Compaction turns out to be very expensive. Any tip to minimize compaction while keeping queries fast?
by vonnik on 1/26/25, 1:30 AM
Curious to what extent Iceberg enables data composability and what the best complements and alternatives are.
by jmakov on 1/26/25, 4:33 PM
Why would one choose this instead of DeltaLake?
by jeffhuys on 1/26/25, 4:05 PM
Looks good, but come on… at least try to open your website on a mobile device.
by dangoodmanUT on 1/26/25, 2:34 PM
iceberg is plauged with the problems it tries to solve, like being too tied to spark just to write data
by honestSysAdmin on 1/26/25, 4:34 AM
Iceberg is a pretty cool guy, he consolidates the Parquet and doesn't afraid of anything.
by rubenvanwyk on 1/26/25, 4:58 AM
And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.