from Hacker News

120ms to 30ms: Python to Rust

by 0xedb on 6/27/24, 8:46 PM with 23 comments

  • by galkk on 6/27/24, 11:39 PM

    Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else.

    Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs.

    It did work in python, in the end, but I feel that writing that in Rust/C# (even if I don't know Rust besides tutorials) in the end would be much more performant.

  • by bin_bash on 6/28/24, 2:30 AM

    I enjoy both languages quite a bit, but a 4x improvement is more of a testament to python than it is to rust imo!

  • by gumby on 6/27/24, 10:05 PM

    What I get from this is that the python interpreter is better than I would have guessed.

  • by Talinx on 6/28/24, 2:54 PM

    That is a lot of text for not determining why the new solution is faster. The only relevant part:

    > Before our migration, the old pipeline utilized a C library accessed through a Python service, which buffered and bundled data. This was really the critical aspect that was causing our latency.

    How much speed up would there have been if they moved to a Rust wrapper around the same C library?

    Using something other than Python is almost always going to be faster. This Reddit post does not give any insights into which aspects of Python lead to small/large performance hits. They show that it was the right solution for them with ample documentation which is great, but they don't provide any generalizable information.

  • by Havoc on 6/28/24, 9:43 AM

    Noticed similar scale of improvement when testing the two on cloud functions.

    Just comes down to whether you need speed of building it or speed of program

  • by pjmlp on 6/28/24, 9:01 AM

    Could be renamed as 120ms to 30ms: Python to "AOT compiled language".

  • by gregors on 6/28/24, 4:44 PM

    Am I the only one thinking "why wasn't it better?"

    Guess I'm never satisfied

  • by jti107 on 6/28/24, 12:34 AM

    use the right tool for the right job. your software requirements and design document should drive the language and the tools you use.