from Hacker News

Problems with Python dependency management

by sanbor on 12/14/24, 10:11 PM with 135 comments

• by Alir3z4 on 12/15/24, 1:05 AM

  Almost 2 decades of working with python.

  I create a venv. Pip install and keep my direct deps in requirements.txt

  That's it. Never understood all these python dependency management problems dramas.

  Recently, I started using pyproject.toml as well which makes the whole thing more compact.

  I make lots of python packages too. Either I go setup.py or sometimes I like to use flit for no specific reason.

  I haven't ever felt the need for something like uv. I'm good with pip.

• by stavros on 12/15/24, 12:48 AM

  These days, I am extremely happy to be using uv with single scripts. Put this at the start of your script (which has to end in .py):

  (EDIT: Sorry, HN doesn't like code, see the start of https://github.com/skorokithakis/calumny/blob/master/calumny... for an example)

  The script will run with uv and automatically create a venv and install all dependencies in it. It's fantastic.

  The other alternative, if you want to be extra sure, is to create a pex. It'll even bundle the Python interpreter in the executable (or download it if it's not available on the target machine), and will run anywhere with no other dependency (maybe libc? I forget).

• by ethbr1 on 12/15/24, 12:44 AM

  In before 'Python dependency management isn't hard. You just use {newest Python flavor-of-the-year way} of doing {same thing that is standardized in other languages}.'

  Which, fair. Python is and will always be a bazaar.

  >> PDM (Edit 14/12/2024) When I shared this article online, I was asked why I did not mention PDM. The honest reason was because I had not heard of it.

  Ha! On brand.

• by buggy6257 on 12/15/24, 12:40 AM

  This really just reads like someone who ignored literally every good practice for using python and then pikachu_shocked.jpg when he set his world on fire

  Just having a virtual environment and requirements.txt alone would solve 90% of this article.

  Also with python 3.12 you literally CANT install python packages at the system level. Giant full page warning saying “use a venv you idiot”

  I expected something along these lines and was still disappointed by TFA

• by stcg on 12/15/24, 2:13 AM

  One of the biggest usability problems with Python dependencies is that the name you import might be different from the name that you use to install the package.

  So if you find some script on the web that has an `import foo` at the top, you cannot just `pip install foo`. Instead, you'll have to do some research into which package was originally used. Maybe it's named `pyfoo` or `foolib`.

  Compare that to for example Java, which does not have that problem, thanks to Reverse Domain Name Notation. That is a much better system.

• by dmart on 12/15/24, 1:31 AM

  Venvs are so clunky and probably the biggest stumbling block for beginners. There was a proposal for a directory-based node_modules analogue which was unfortunately rejected.

  I think that would have been the single biggest improvement to the Python onboarding experience ever.

• by daemonologist on 12/15/24, 1:26 AM

  For me the vast majority of the pain occurs when _updating_ dependencies. It can be an all-day chore to find and force versions of everything which will work together, and occasionally you'll have an irreconcilable conflict in subdependencies. I have in the past had to hack in a second version under a different name and update all imports in the parent dependencies to match.

  I know other languages have various solutions for this (to basically have package namespaces local to a branch of the dependency tree), but I don't know how much better that experience is.

• by roenxi on 12/15/24, 12:54 AM

  Bit of a Chesterton's fence situation. One of the reasons that Python is popular at all is because the dependency management is terrible and easy to get wrong. The point is it enables relative novices to get started without engaging with important questions. Things like what environmental assumptions are being made, what dependency versions are supported, what a sustainable build pipeline looks like. They can just be vague and their project will work in the here and now - enough to do whatever they want to do.

  There is a trade off here and the equilibrium is probably deliberate. The sort of person who tries to get dependencies right up is a professional programmer and although a lot of them use Python the language is designed for a much broader audience. Java is an example of much better dependency management by default and in the main it is only professionals with a view to the long term using Java. Setting up a Java project needs a tutorial and ideally specialist software.

• by rednafi on 12/15/24, 12:46 AM

  Don’t we see this kind of rant pop up all the time? So much misery could be avoided by simply using a virtual environment. Plus, uv handles it all in a more elegant manner.

• by abeppu on 12/15/24, 1:29 AM

  > Good dependency management means ... > it is possible to safely evolve and update your environment when new versions of dependencies become available.

  The post, despite its length, doesn't really spend time on this aspect, and I think it's one of the weaker areas.

  Suppose my library depends on both A and B, and both of A and B depend on C, and in the current version of my code, there's some set of versions that matches all declared dependencies. But A releases a new version which has new features I want to use, and also depends on a newer version of C than B supports. I'm blocked from upgrading A with the normal tooling, unless I either ditch or modify B.

  This can be a problem in other ecosystems too, but better tools support working around it. In the java world (and I may be out of date here), the maven-shade tool would allow you to effectively use multiple versions of C, e.g. by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need.

  I think this is also most of the time in principle possible in python (though perhaps some relevant static analysis of seeing what is actually used in more difficult), but the ecosystem doesn't really support or encourage it.

• by shlomo_z on 12/15/24, 12:44 AM

  Some sent this in a group I am in 2 weeks ago. I will copy my response here:

  "Articles like that over-dramatize things a lot. As a self-taught Python developer, I mostly learned from looking at code. When I started, I didn't know anything about packaging. I just installed stuff globally with pip and moved on. I didn't really build projects, just scripts. Once in a while I used a venv when I was following a guide using them. To give a bit of perspective, for a while I didn't really know how to make a class, only a function. Fast forward a few months, I learned a bit more about projects and also started contributing and/or forking open source projects (again, never took classes, I just explore). I used poetry a little, it works nicely. Now, I use uv for anything new, and it works beautifully. It's really simple. uv init, uv add any deps, and uv run to run the project or just activate the venv. And I never run into dependency issues. It's like really simple. And being able to manage python versions really simply is a really nice bonus. My cicd doesn't even use a special docker container or anything. Literally installs uv (using curl | sh), uv sync, uv run. Finished. And very fast too. So yeah, Python dependencies aren't automatically vendored. And yes, Python tooling has historically been bad. But now we have amazing tooling and it's really really easy to learn. uv is wonderful, poetry is also great, and anyone complaining it's too hard is either over dramatizing it or doesn't have 5 minutes to read a readme. So yeah, people should stop over dramatizing something that really isn't dramatic."

• by sunshowers on 12/15/24, 1:04 AM

  I'm quite happy with uv these days! It's taken most of the pain out of Python management, and I can just think in terms of `uv run`, etc.

  (Astral sponsors me on GitHub, but I'd hold the same belief even if they didn't.)

• by jredwards on 12/15/24, 3:38 AM

  This post feels like it was written 10 years ago. There are plenty of excellent dependency management systems for Python these days. Honestly, the biggest problem is that there are too many ways to set it up.

• by m4thfr34k on 12/15/24, 1:00 AM

  Requirements file and everything is installed in a container where the code runs from. Lots of problems solved.

• by sgarland on 12/15/24, 2:38 PM

  One easy trick they don’t want you to know about: stop importing dependencies. You probably don’t actually need numpy. You probably don’t need requests. The stdlib is incredibly broad.

  This doesn’t always work, of course (especially for large projects), but for smaller ones, it absolutely does.

• by hk1337 on 12/15/24, 1:03 AM

  I think the biggest issue is that nobody can settle on what to use and how to use it. pip, pdm, poetry, and maybe something else I don’t know about. Some new flashy dependency manager comes along and that’s the new hotness. I'm probably just as guilty of that.

• by sebazzz on 12/15/24, 8:08 AM

  > You decide to re-install your operating system and vow to go back to doing everything in Excel.

  Which itself can run Python now. Only in the Microsoft Cloud, not only to rake in that sweet subscription money, but probably also to avoid these headaches.

• by cr125rider on 12/15/24, 1:32 PM

  Docker and Devcontainers. You can make a mess in your own little box and not mess with my system.

  We use conda/momba to install some more complex dependencies that are piles of shared libraries and a messy web of sub dependencies.

• by aszen on 12/15/24, 10:03 AM

  Everyone seems to praising uv in the comments, seems like no one read this article, it has its drawbacks, it can't install system level libraries and dependencies which many packages depend on.

• by TZubiri on 12/15/24, 1:27 AM

  'Python is mostly a "glue" language'

  100% disagree, yes it may be used for that, but it's a fully fledged language, people use it for data science or for backend business logic.

• by ltbarcly3 on 12/15/24, 1:23 AM

  Nah, it's fine. Use poetry or uv and get on with it.

• by egberts1 on 12/15/24, 3:32 PM

  Biggest turn off of all is the lack of centralized (if any) search engine for Python packages at CLI-level `pip` command.

• by VeejayRampay on 12/15/24, 1:47 PM

  really sucks that we're stuck with python when languages of the same realm (JS and ruby for example) are so much better (as languages and with regard to the associated tooling)

• by lofaszvanitt on 12/15/24, 1:52 AM

  And the drama keeps happening because the article is 70K long...

• by ac130kz on 12/15/24, 5:15 AM

  uv (and requirements.txt as a lock) + Debian Docker container (distroless with added packages for runtime). There are no problems with this setup.

• by llIIllIIllIIl on 12/15/24, 1:42 AM

  Python deps are alright unless you’re updating celery.

• by booleandilemma on 12/15/24, 1:26 AM

  Having recently used python and poetry - I agree 100%!

• by WillAdams on 12/15/24, 1:00 AM

  Well, there is an XKCD on this:

  https://xkcd.com/1987/

  FWIW, I've rolled, crashed, and burned on at least one attempt to use Python tooling due to forgotten about installs/environments which required uninstalling/deleting everything and start anew.

  It would be nice if there was a consensus and an agreed-upon solution and process which would work as a new user might expect.

• by martynassubo on 12/15/24, 12:55 AM

  While uv wasn't very mature, I wrote a similar piece, where in the end I tried suggesting pyenv/pipx/poetry:

  https://open.substack.com/pub/martynassubonis/p/python-proje...

  However, these days, I would just probably go with uv and hope that astral doesn't pull any VC shenanigans:

  https://open.substack.com/pub/martynassubonis/p/python-proje...

  It's not Rust/Go toolchain, but we work with what we got here...

• by yosito on 12/15/24, 12:50 AM

  Wait until you hear about JavaScript dependency management!