If there is ever a Python 4, the number one goal of the project should be to make Python completely hermetic, repeatable, and redistributable. And there should only be one way to do it.
pip, requirements.txt, and venv are madness. Look at how much suffering they cause! I was dealing with (different) pip issues today myself, and it's not an infrequent occurrence.
I've always found it odd that Python has no built-in version awareness. All it knows is a single installation and a PYTHONPATH, and you fake per-project dependencies by copying everything into a virtualenv. It seems very much bolted-on. (Granted, most other languages don't do it much better.)
I wish you could just install all packages (in multiple versions!) into one place like /usr/lib/pythoncache, and the module loader would then decide which package version to load at import time. You could define your dependencies as usual in reqirements.txt or pyproject.yaml. And you could use either pip or the system package manager to manage the global package cache.
I am 100% with you. Love python the language, but the ecosystem around it was never thought out properly. It grew organically over the years and it is a complete and utter mess.
My last attempt at using pip this very morning to install a bunch of deps crashed and burn with an obscure python stack dump.
Oh, and, in your taxonomy of partial and inadequate solutions, you forgot to mention anaconda, possibly the most invasive package management solution I have ever seen, which exhibits the same problems pip does, i.e. where each install is a total crapshoot.
I started to use python about 20 years ago, which was very pleasant (there was a nearly seamless transition from the old Numeric to Numpy at the time). I was using it less regularly for a number of years and came back to it to find what appears like an absolute mess.
Is there any good overview which packaging solution has which properties, and will work in specific situations?
In fairness, many of these are open source and at least in the case of pip, for many years there was literally only two maintainers although now there a couple more. This is what I learned at one of the python conferences.
It’s odd how open source packages that big companies use can be reliant on so few people.
Edit: although checking the pip github, there more people committing so maybe I got something wrong, but there is far more activity starting around 2019.
> pip, requirements.txt, and venv are madness. Look at how much suffering they cause!
Perhaps I'm just lucky but I never experienced any major issue with pip+venv. It always worked as advertised: create venv, activate venv, install/update pip locally, install packages listed in requirements.txt, and you're all set.
If you write a package and you use pip to install and test it, pip will install the requirements according to what is requested, by default the latest acceptable version.
If somebody else installs your package at a later time, pip will again install the latest fitting versions.
This install will be different from your install, or from other installs other people made in the meantime.
If a newer version if a dependency breaks your package because it has a backwards-incompatible change, your package won't work - and it will not show up in your testing because pip sees the existing, already installed packages and will think they are new enough. So, you'll find "it works for me".
A good examples are packages like python-opencv which break on python2 because the versioning implies they are backwards-compatible but they (or their given dependencies) use syntax which is not supported by python2.
And these things tend to snowball quickly because the number of packages which other packages depend on tends to grow exponentially, without a real upper bound.
And while there is lots of annoyment and cursing about package managers, I think a huge part of the problem is a cultural issue in the python community because people accept and create libraries with backwards-incompatible changes without marking that in the version numbers.
You can go along way with that setup solo, and even in big companies, but it’s not hermetic, it’s not repeatable (byte-for-byte), and it’s not easily distributable.
Assuming we want something that works now and prefer something that works on POSIX systems over something that is not portable at all, would using GNU Guix (as a package manager) be able to improve the situation? As far as I understand, it is hermetic, deterministically repeatable, and redistributable. Just limited to POSIX.
I guess that you do not see conda as a sufficient solution. How would you include and manage binary extension modules written in C which in turn rely on C libraries which might or might not be system libraries? Is it possible to manage that well without cooperation with the host system?
And, as we are at talking about the future, how would you manage native extension modules which are written in Rust?
I'm surprised. I've been using it for ~10 years now, on multiple Linux (CentOS, Fedora, Ubuntu, Debian) and Win7 in the past, and it's always been rock solid and extremely reliable, albeit slower than I' d like (but not too slow as to be unusuable).
I've also collaborated and discussed with hundreds of other conda users, and except for speed and the lack of official lockfiles[0], there were no complaints, only praise.
[0] The functionality was always available, but not straightforward and not named as such.
And pipenv is getting better. Recently thete has been a push of improvement. It still has its issues though. Pipenv or Poetry? Time will tell. Poetry needs to parallelize.
The original author has a habit of abandoning software projects after a while. But I think they managed to get it into the pypa docs (python packaging 'authority'), and so I guess someone has brought it back.
I believe, after some conflict with the community, the original author divested himself of all his F/OSS projects. This resulted in pipenv being donated to the pypa GitHub org. Since then, the project appears to be seeing sporadic development through community contributions.
pip, requirements.txt, and venv are madness. Look at how much suffering they cause! I was dealing with (different) pip issues today myself, and it's not an infrequent occurrence.
Rust's Cargo is a good reference point.