from Hacker News

Making my debug build run 100x faster so that it is finally usable

by broken_broken_ on 2/18/25, 8:48 AM with 20 comments

  • by molenzwiebel on 2/18/25, 7:46 PM

    For this use-case, you can squeeze out even more performance by using the SHA-1 implementation in Intel ISA-L Crypto [1]. The SHA-1 implementation there allows for multi-buffer hashes, giving you the ability to calculate the hashes for multiple chunks in parallel on a single core. Given that that is basically your usecase, it might be worth considering. I doubt it'll provide much speedup if you're already I/O bound here though.

    [1]: https://github.com/intel/isa-l_crypto

  • by ack_complete on 2/18/25, 8:07 PM

    SHA1 is difficult to vectorize due to a tight loop-carried dependency in the main operation. In an optimized build, I've only seen about a 15% speedup over the scalar version with x64 SSSE3 without hardware SHA1 support. Debug builds of course can benefit more from the reduction in operations since the inefficient code generation is a bigger issue there than the dependency chains. I think the performance delta is bigger for ARM64 CPUs, but it's pretty rare to not have the Crypto extension (except notably some Raspberry Pi models).

    The comments in the SSE2 version are a bit odd as it references MMX, and the Pentium M and Efficeon CPUs. Those CPUs are ancient -- 2003/2004 era. The vectorized code you have also uses SSE2 and not MMX, which is important since SSE2 is double the width and has different performance characteristics from MMX. IIRC, Intel CPUs didn't start supporting SHA until ~2019 with Ice Lake, so the target for non-hardware-accelerated vectorized SHA1 for Intel CPUs would be mostly Skylake-based.

  • by bean-weevil on 2/18/25, 2:26 PM

    Why not just compile that particular object with optimizations on and the rest of the file with optimizations off?

  • by tvbusy on 2/18/25, 9:35 PM

    I understand the post is about learning to speed up SHA1 calculation, that I have no comment. However, the state file is a solved problem for me. It's a rare case where state files are corrupted and it's simple to just re-check the file. I cannot imagine a torrent client checking the hash of TBs of files for every single start. It's not a coincidence that many torrent clients have a feature to skip hash checking and just immediately assume the file is correct and start seeding immediately.

  • by Neywiny on 2/19/25, 1:56 AM

    If I were a betting person I'd bet that the sha1 instructions and the openssl instruction map to similar enough uops. Unsure if there's a way to check, but that's my understanding of the thousands of instructions in modern processors - mostly just assigning names to common patterns.