from Hacker News

Cython 3.0 Released

by ngoldbaum on 7/18/23, 8:26 PM with 82 comments

• by earthnail on 7/19/23, 11:15 AM

  Amazing to read that most comments here are about broken build dependencies. I completely get the frustration, but somewhat sad to see that there's not that much talk about the actual improvements.

  There seem to be a lot of cool things in this. For example, semantics for division, power operator, print, classes, types and subscripting are now identical with Python 3. https://cython.readthedocs.io/en/latest/src/changes.html#imp...

  Improved interaction with numpy by moving that part of the code into numpy itself: https://cython.readthedocs.io/en/latest/src/changes.html#int...

  Improved support for const and volatile in C, and improved support for C++ expressions like std::move https://cython.readthedocs.io/en/latest/src/changes.html#id6

  I love how the changelog also shows the bugfixes and improvements that each of these changes enabled. Hats off to the team behind this release.

• by mturk on 7/18/23, 9:10 PM

  A project I work on is currently working through a bunch of performance issues that arose because we had inlined functions in `pxd` files that we did not declare `noexcept nogil`. So, heads up if you see similar regressions! We saw a ridiculous slowdown (7s to 100m, for one suite) because they were inside some awfully tight loops. But the fix was pretty straightforward once we worked it out.

• by navels on 7/18/23, 9:06 PM

  And chaos ensued. Unable to install aws-cli via pip: https://github.com/aws/aws-cli/issues/8036

• by Waterluvian on 7/18/23, 11:08 PM

  The release of this pretty much hosed my morning at work. Suddenly CI wouldn’t build because this broke things in pyyaml and awscli.

  And then I got to re-learn how very bad Poetry and python dependency management is. I’m trying to update pyyaml to 6.0.1 and Poetry systematically downloaded every single major, minor, patch version of another dependency in trying to find which would fit the version constraints. Took half an hour.

• by ggm on 7/19/23, 2:46 AM

  I get that if I adorn my python3 with types the right way, cython can uplift C betterer, but if I don't adorn my python3 and only use core imports (no pip) do I get any benefit?

  I have some humongous (for me) dict lookups I am doing on strings, counting in ints, and if I could 2x or 3x faster without having to recode I'd love it. The Cython web is directed at people willing to recode, I am ultimately willing but I wish there was some low hanging fruit evidence: is this even plausibly going to get faster or is dict() walk cost just "it is what it is" ?

  Oddly, it looks like across pickle dump/load I get some improvement. Does pickle restoring consume less memory than raw-made?

• by westurner on 7/18/23, 8:59 PM

  From https://cython.readthedocs.io/en/latest/src/changes.html#com... :

  > Since Cython 3.0.0 started development, CPython 3.8-3.11 were released. All these are supported in Cython, including experimental support for the in-development CPython 3.12

• by bippingchip on 7/19/23, 7:09 AM

  This is a somewhat tangential question to the new release, but there might be folks here that can answer this question.

  Having used swig to create Python bindings for C++ code over 10 years ago, what’s the recommended way to do this in 2023? There’s still swig, there’s Cython, pybind11 and a few others. I do know swig and how complicated it can get with footguns abound when things grow more complex.

  Is Cython the way to go? How does it hold up to the alternatives? Google search gives many articles on the topic, but many typical SEO optimized low-value, and those that do show a bit of depth, focus on the basic mechanics, not really on things hold up for larger projects…

• by itamarst on 7/19/23, 11:13 AM

  Cython is really great at a small scale, but it has issues scaling up to large codebases (of Cython), where I think it's an anti-pattern. None of these are the fault of the Cython creators, it's really an extremely useful tool, it's just inherent in the design space, the things that make it so cool (transparently mix C and Python!) come with trade-offs.

  1. Memory unsafety. It's still C or C++ in the end, the more you shift your code in that direction the easier it is to screw up.

  2. Two compiler passes: first Cython->C, then C->machine code. This means some errors only get caught in second pass, when it's much harder to match back to the original code. Extra bad when using C++. Perhaps Cython 3 made this better, but it's a very hard problem to solve.

  3. Lack of tooling. IDE support, linting, autoformatting... it's all much less extensive than alternatives.

  4. Python only. Polars is written in Rust, so you can use it in Rust and Python, and there's work on JavaScript and R bindings. Large Cython code bases are Python only, which makes them less useful.

  Long version: https://pythonspeed.com/articles/cython-limitations/

• by zaps on 7/19/23, 12:56 AM

  Comments confirmed—Python packaging is broken

• by ilayn on 7/19/23, 11:22 AM

  This is probably one of the most anticipated release in the one recent times for all python ecosystem. A sincere thanks to handful (literally) of people who made this possible and despite all kinds of funding challenges.

  Nevermind the foo broke bar comments here.

• by gigatexal on 7/18/23, 11:25 PM

  This broke a lot of stuff for us at work today. Building the wheels or something I can’t remember. The fix was forcing pip to use not the 3.x and use something older I think.