And agreed, there are two separate use cases: development and using the software.
But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that there's much choice, since python also has a shitty story when it comes to backwards compatibility; if you're going with 3.10, there's possibly only one version of a given library that will work.)
Java has a commonly used "fat jar" approach which rolls up all dependencies into a single file. It's excellent. In the python world, this doesn't exist, because virtualenvs aren't portable. If that can be fixed (perhaps a specific section in requirements.txt that captures anything that needs to compile C for the platform) then a distributable virtualenv would become possible. Distros would then scan the application for vulnerabilities (via requirement.txt's manifest), build the distributable-virtualenv, and ship _that_. Python library maintainers don't have to do anything different (except, of course, use the standard way to declare dependencies).
> Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose?
Debian Developer here. Part of packaging work, for Python libraries or anything else, is to verify the reliability of the upstream developers, audit the code, set hardening flags, add sandboxing and so on.
I spotted and reported vulnerability myself and it's not uncommon.
Please don't take it the wrong way, I have a lot of respect for distros packagers and maintainers. I donate to debian, and I report bugs. I think you are basically heroes of the FOSS world, because without your invisible (and frankly thankless) work, mine wouldn't exist.
But come on, there are 300k entries on pypi, 200k more for perl and 160k for ruby. I'm not even counting the whooping 1.3M on npm because I assume this is considered taboo at this point.
You cannot package 0.0001% of that, not to mentions updates.
And unless distros make it as easy to package and distribute deb/rpm/etc than it is to use the native package manager, this means distro packaging will never be attractive to most users because:
- they don't have access to most packages
- the provided packages are obsolete
- packages authors have no way to easily provide an update to the users
- it's very hard to isolate projects with various versions of the same libs, or incompatible libs
And that's not even mentioning that:
- package authors may not have the right to use apt/dnf on their machine.
- libs may be just a bunch of files in a git repo, which pip/gem/npm support installing from
- this is not compatible with anaconda, which has a huge corporate presence
- this is not compatible with heroku/databrics/pythonanywhere, etc
- this is hard to make it work with jupyter kernels
Now let's say all those issues are solved. A miracle happens. Sorry, 47 miracles happen.
That would force the users to create a special case for each distros, then for mac, then for windows. I have a limited amount of time and energy, I'm not going to learn and code for 10 different packaging systems.
It's not that we want to screw over linux distros. It's that it's not practical, nor economically viable to use the provided infra. The world is not suddenly going to slow down, vulnerabilities will not stop creeping up, managers will not stop to ask you to use $last_weird_things. This ship has sailed. We won't stay stuff only published 5 years ago with delays of months for every updates.
Thanks for your work. And I have to say that for internal development, when there is no need for the latest features, it's much easier to develop based on a Debian release as much as possible. A stable distribution provides an easy to track baseline not only for Python libraries, but any other tools that may be needed.
And agreed, there are two separate use cases: development and using the software.
But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that there's much choice, since python also has a shitty story when it comes to backwards compatibility; if you're going with 3.10, there's possibly only one version of a given library that will work.)
Java has a commonly used "fat jar" approach which rolls up all dependencies into a single file. It's excellent. In the python world, this doesn't exist, because virtualenvs aren't portable. If that can be fixed (perhaps a specific section in requirements.txt that captures anything that needs to compile C for the platform) then a distributable virtualenv would become possible. Distros would then scan the application for vulnerabilities (via requirement.txt's manifest), build the distributable-virtualenv, and ship _that_. Python library maintainers don't have to do anything different (except, of course, use the standard way to declare dependencies).