from Hacker News

K4 – High performance open-source transactional, durable embedded storage engine

by alexpadula on 10/30/24, 1:58 AM with 28 comments

  • by alexpadula on 10/30/24, 1:58 AM

    Hello everyone! I hope you are all well.

    I've been working on a few storage engine designs recently implementing an LSM tree and would like to share K4.

    K4 is a storage engine written completely in GO with no dependencies. The write speed surpasses RocksDB (7.8.3) with similar configuration.

    I will write way more comprehensive benchmarks down the line.

    The current features of K4

    ------------------------------

    - High speed writes and reads

    - Durability

    - Variable length binary keys and values. Keys and their values can be any length

    - Write-Ahead Logging (WAL). System writes PUT and DELETE operations to a log file before applying them to the LSM tree.

    - Atomic transactions. Multiple PUT and DELETE operations can be grouped together and applied atomically to the LSM tree.

    - Paired compaction. SSTables are paired up during compaction and merged into a single SSTable(s). This reduces the number of SSTables and minimizes disk I/O for read operations.

    - Memtable implemented as a skip list.

    - In-memory and disk-based storage

    - Configurable memtable flush threshold

    - Configurable compaction interval (in seconds)

    - Configurable logging

    - Configurable skip list

    - Bloom filter for faster lookups. SSTable initial pages contain a bloom filter. The system uses the bloom filter to determine if a key is in the SSTable before scanning the SSTable.

    - Recovery from WAL

    - Granular page locking

    - Thread-safe

    - TTL (time to live) support

    - Optional compression support (Simple lightweight and optimized Lempel-Ziv 1977 inspired compression algorithm

    - Range and equi functionality (Get, NGet, Range, NRange, GreaterThan, GreaterThanEq, LessThan, LessThanEq)

    - No dependencies

    I am in the process of writing a C binding for K4 which will enable me to write FFI's for multiple other languages like Python, Node.JS, Ruby, etc.

    I hope you get a chance to check it out and do let me know your thoughts!

    Thank you kindly.

  • by jitl on 10/30/24, 3:22 PM

    It’s cool to see new storage engines, thanks for sharing.

    For transaction, how are conflicts on concurrent transaction handled? What’s the ordering? Are you considering any conditional check operation like compare-and-swap? For example I think Deno KV’s API where each value has a versionstamp and for transactions there’s an operation CHECK(key, versionstamp) that cancels the transaction if key doesn’t have the given versionstamp at the time of transaction.

  • by fithisux on 10/30/24, 3:51 PM

    Keep it coming. I love the work people put on storage engines.

    Do you believe a graph storage can be layered on top of K4?

  • by alexpadula on 10/30/24, 2:57 PM

    Just putting this out there. I will be writing a paper on my experiences writing storage engines and I will be releasing a full website for K4 at k4db.com

    Cheers all

  • by knowitnone on 10/30/24, 3:56 PM

    Nice work! Any reason why Go was used and not C or Rust? I was under the impression Go would be slower and you are aiming for high performance.

  • by Gys on 10/30/24, 8:23 AM

    This should be a ‘Show HN’?

  • by 0x3331 on 10/31/24, 9:53 PM

    Insane.