from Hacker News

How to write Python extensions in Rust with PyO3

by jeremychone on 2/28/23, 11:31 AM with 9 comments

  • by MrPowers on 2/28/23, 1:45 PM

    PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs

    It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust.

    This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.

  • by Galanwe on 2/28/23, 1:21 PM

    I found PyO3 to be a bit weird to use. There's plenty of documentation, but it's kind of vague on the general basic concepts, which makes the library hard to understand (even though I have a background in writing CPython extensions in C).

    Also, once you play with it more than 30m, you quickly realize that struct attributes are copied all over the place... which is not only slow, but more sadly any mutable function on an attribute will just silently modify a temporary copy...

  • by UncleEntity on 2/28/23, 3:43 PM

    Isn’t python “memory safe” too?

    From my experience the trickiest bit of getting python extensions to work (i.e. not segfault) is dealing with the lifetimes of the objects in the wrapped library. If python owns the object you’re golden, if it borrows a reference then things get real complicated real fast. It usually takes a lot longer to get the ownership model correct than doing the grunt work of wrapping the library — unless you own the lib and make it python friendly it’s usually a case of “do what you can with what you were given”.

    Blender’s python API is full of cases where the python object can’t be trusted to be valid if you do anything with the underlying container and it is super easy to crash the whole program doing something that should Just Work™.

    Wondering how rust would deal with the cross-domain model where nothing can be guaranteed once python gets its dirty little paws on the pointer?

  • by rob74 on 2/28/23, 1:23 PM

    I assume PyO3 is a reference to the chemical formula of iron oxide (a.k.a. rust), which is Fe2O3?

  • by russnes on 2/28/23, 12:29 PM

    PyO3 works great. I've written an example repository for a hello world app for this if anyone's interested: https://github.com/russnes/rust_python_package_example

  • by stefanka on 2/28/23, 3:03 PM

    Together with rust-numpy, it is a great way of writing numeric code (using the ndarray library) https://github.com/PyO3/rust-numpy