The mechanism, as used by other projects as well, is that when you upload a project to PyPI you can set a minimal required version of Python. So when you upgrade, you get the latest version that supports the current version of Python.
Sometimes maintainers screw this up, but so long as the maintainers do this everything Just Works: people on old Python get old packages, people on new Python get new packages.
Does anyone have any new insight in to plans to resolve pip search which has been broken for over a month now?
xmlrpc.client.Fault: <Fault -32500: "RuntimeError: PyPI's XMLRPC API has been temporarily disabled due to unmanageable load and will be deprecated in the near future. See https://status.python.org/ for more information.">
How does it work for reproducing scientific experiments using legacy Python 2 code?
All of the major scientific libraries and core packages are dropping python 2 support. Are there legacy versions of these packages being maintained or at least preserved?
Maybe if you're reproducing a scientific experiment you should use the original package versions? New releases of pip or scientific libraries won't support Python 2, but the old versions should still work. And you won't be running different code from using up-to-date versions.
> How does it work for reproducing scientific experiments using legacy Python 2 code?
Install Python 2.7, which is by does with pip. Upgrade pip if you want, which will get you the last version of pip that supports Python 2 (the old versions haven't gone away, new versions just don't support py2 and won't show up when upgrading on py2). Install the needed scientific libraries. Run code.
> Are there legacy versions of these packages being maintained or at least preserved?
AFAIK, there is no plan to remove py2 packages from pyPI.
I assumed that the old pip versions would keep working, as long as the python2 packages kept being hosted, and this is just freeing new pip development from being python2 compatible. Does anyone know if this is the case?
If not, it’s a good argument for vendoring dependencies when publishing scientific code.
Indeed. We're hitting this on some Ubuntu 16.04 (supported until April 2021!) build containers which start their work with the suggested self-upgrade of `pip install -u pip` that then bombs out with SyntaxErrors on subsequent operations due to f-string usage.
We (my team) hit this as well with an automated compatibility test running on Ubuntu 16.04. I decided to just change the test to install Python 3.8 from the deadsnakes PPA, and then use get-pip.py to install pip.
This does change the test to no longer use Python 3.5, but considering a) we'd already dropped support for Python 3.5 for the code we were testing, b) Python 3.5 is EOL and c) Ubuntu 16.04 will be soon, it didn't seem worth trying to stay on 3.5.
This approach is okay if you can work entirely in an environment (venv or otherwise) that is unmoored from the underlying system. But usually the reason to use an older OS is because you want to be linking to specific versions of boost or some other dependency that's a pain to build and install from source, so depending on your scenario, installing a deadsnakes python may not work if you're depending on there being a corresponding `python3-xxxx` package that links up system python to bindings for a particular library.
Isn't that a packaging bug? Shouldn't py<3.6 not see pip packages that require 3.6+?
Also, if you are using system-managed Python for stability you should probably also be using system-managed pip rather than upgrading from pyPI, for the same reason.
Yes, I think it is a packaging bug, but it's reflective of an attitude where old stuff is hung out to dry even before the officially-committed time horizons have expired. This would be fine if it were possible for distro maintainers to pick up the reins, but the pip maintainers have deliberately wrested control from them— they've basically decided that their recommended approach is for everyone to use the latest pip regardless of what interpreter they're using, especially in the case where you're using that interpreter in a virtualenv. For example on an Ubuntu 16.04 machine if you create a new venv, the system pip that gets installed there immediately runs off to the internet to check itself and then prompt you to upgrade it:
$ python3 -m venv test
$ test/bin/pip install pyyaml
Collecting pyyaml
Using cached https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz
Building wheels for collected packages: pyyaml
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/administrator/.cache/pip/wheels/2a/d4/92/cf299bdf4162957ca8126b46e913e29f76a4f17ca762c45028
Successfully built pyyaml
Installing collected packages: pyyaml
Successfully installed pyyaml-5.4.1
You are using pip version 8.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ test/bin/pip install -U pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)
100% |################################| 1.5MB 405kB/s
Installing collected packages: pip
Found existing installation: pip 8.1.1
Uninstalling pip-8.1.1:
Successfully uninstalled pip-8.1.1
Successfully installed pip-21.0.1
But then:
$ test/bin/pip
Traceback (most recent call last):
File "test/bin/pip", line 7, in <module>
from pip._internal.cli.main import main
File "/home/administrator/test/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
sys.stderr.write(f"ERROR: {exc}")
^
SyntaxError: invalid syntax
Note that the prompt doesn't just show up in a virtualenv. The system-installed pip has this behaviour as well, and there are some packages which don't install correctly with a pip this old, so the only thing to do is manually track a working version of pip, force-install exactly that version, and ignore the repeated prompts to upgrade.
> it's reflective of an attitude where old stuff is hung out to dry even before the officially-committed time horizons have expired.
As mentioned in another comment, the “officially-committed time horizon” is imposed by Ubuntu, and it does not make sense to apply it on pip maintainers. Python 3.5 was retired by CPython in September 2020.
Unnecessary snark is unnecessary. We already did the bulk of our 16.04 -> 20.04 upgrades last summer, and were planning on mopping up the remainders over the next two months, while there was still time. That's the point of these support commitments with clearly defined timelines— allowing people to plan for this stuff to happen when it suits them.
Now we have a handful of broken support VMs and other things which have to be migrated in a rush instead of according to the planned schedule.
You missed my point. Ubuntu packages pip via apt, not via pypi. You can swap from an installation provided by apt to one provided by pypi using your method, but it means you're off the supported version and any guarantees of support are lost. The quick fix here is to not update pip to the latest, but the latest supported by python3.5 and carry on with your migration as scheduled.
I realise that this is a common practice, and I've been caught out myself in the past with CentOS7+Python3.4, but be aware that it comes with its own set of tradeoffs. Sadly, the packaged version of pip is usually quite old and pip is loud about outdated versions, so the solution is either to upgrade the OS or leverage something that is not as tied to the OS.
If you want to explore the latter, I'd suggest python:3.5 containers if you're building pure python code. Alternatively, I'd suggest manylinux containers if you have C extensions and run on a glibc distro like Debian/RHEL. Both will have the last supported version of pip for your python version installed and neither will randomly break on you.
The final lesson I've learned is to be aware that pip can install the latest version supported by your python version as long as it can access the metadata for the package in the index. That's available on pypi, but not all self-hosted indexes have it, notably nexus.
So I wondered if there was an issue here where pip 8.1.1 is just so old that it doesn't know how to respect that constraint? As an experiment, I started with a new venv and upgraded just to the latest pip 20:
$ test/bin/pip install pip==20.3.4
Collecting pip==20.3.4
Downloading https://files.pythonhosted.org/packages/27/79/8a850fe3496446ff0d584327ae44e7500daf6764ca1a382d2d02789accf7/pip-20.3.4-py2.py3-none-any.whl (1.5MB)
100% |################################| 1.5MB 695kB/s
Installing collected packages: pip
Found existing installation: pip 8.1.1
Uninstalling pip-8.1.1:
Successfully uninstalled pip-8.1.1
Successfully installed pip-20.3.4
You are using pip version 20.3.4, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Now interestingly I don't get any prompts after that, and it doesn't try to upgrade past 20.3, so I think that might have been the last gasp of pip 8.1.1:
$ test/bin/pip install -U pip
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
Requirement already satisfied: pip in ./test/lib/python3.5/site-packages (20.3.4)
Collecting pip
Using cached pip-20.3.4-py2.py3-none-any.whl (1.5 MB)
Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
So yeah, basically the fix is just to hard-code the version you know works. But I think this demonstrates an issue with the PyPA being out of touch with real-world use-cases. No one is upgrading pip version by version— 99% of pip installations are either going to be get-pip.py or upgrading a venv directly from a distro-shipped version. IMO those use-cases should be carefully tested on each release and checked for the versions in popular distros, even at the tail end of the support windows.
The other side of this is Debian/Ubuntu, of course, and with sufficient coordination (knowing the a pip was coming which their "supported" pip would auto-upgrade to and break itself) they could have patched their `python3-pip` package to only upgrade as far as 20.3.4.
Both Ubuntu 16.04 and Python 3.5 are end-of-life. Older versions of pip will work, but newer versions drop the legacy compatability. If you have an out-of-date system you can still use an old pip just fine.
16.04 is EOL on April 30, 2021. Even so, the idea of expecting to run the latest version of a package on an stability-focused operating system released almost 4 years ago seems hopeful.
It's supported until the end of April. Obviously we're in the process of migrating things, but it's either supported or it's not, and as long as it's supported, it should work.
Using the latest pip is the will of the PyPA, and they let you know this on every invocation that touches the network:
You are using pip version 8.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
There are also packages and workflows which break on the older pips, and this has been justified on the basis that everyone can always just use the latest.
> Other than f-string, I'm not sure what critical features were brought in with 3.6.
* type annotation for variables
* asynchronous generators
* asynchronous comprehensions
I can see any of those three seeing use in pip and being worth dropping earlier Python for more than f-strings (or underscores in numeric literals, the other highlighted py 3.6 new feature.)
The major takeaway from this is: if your project is mission critical, then don't depend on free/open source infrastructure.
It's not like there is a "rep" you can call at "python" if this is now breaking your build process.
edit: almost all of these replies have missed my point. Yes, you can still use python2, but if you depended on pip for your build process (which I would consider "infrastructure"), then you are out of luck for this and there is nobody you can call to fix it for you.
Yeah, its not like you access to and the sources of all versions of Python. No proprietary vendor has ever revoked licenses, pulled downloads, version or timebombed binaries, stopped support for eol products, auto upgraded, etc.
Yes open-source is clearly worse for giving you the right and means. Vs being subject to the whims and profitability of a company.
It is literally happening right now. The people who have demanded that everybody switch to python3 instead of simply forking python have proven the FUD true.
If you want forks there are forks. Off the top of my head, Redhat is supporting Python 2 for several more years and there's a project called Tauthon [1] that is "Python 2.8" in spirit. I'm sure there's more efforts I'm not aware of.
The push for Python 3 has been happening for over 10 years at this point. At what point do you expect people to be able to stop supporting a very outdated distribution of any software?
6 years ago my company switched to Python 3.5 and there was a big push in the community at that point to move away from Python 2. I mean Python 3 came out around the same time as Windows Vista, which has been unsupported since 2017.
Good luck calling your "rep" at nearly any commercial software supplier and explaining you're using a decade-old version of their software a year past its EOL date and would they kindly provide you support.
It doesn't matter whether it's open source or not, there has to be a line on these things, and there's been more than enough time given here.
> It doesn't matter whether it's open source or not
It actually does: with open-source you are not bound to the vendor if it isn't cooperative, but can pay anyone else willing and able to do the work. Plenty consultancies will happily fix old bugs or backport fixes in open-source software for you, but can't do much about closed software you depend on.
Well, I'd say no surprises here, because as most other Open Source software / infrastructure, Python is provided under a license that offers the software as-is and doesn't promise "warranty of fitness for any particular purpose". Technically the complaint here is against pip, but I guess they have similar terms and conditions.
On the other hand, you would have enjoyed a royalty-free, world-wide permission to use, distribute, and modify the code base at your will.
So I'd say, not a bad wager if you're willing to do the extra work of keeping your software up to date over time (which otherwise you'd have been paying someone to do, in case of a proprietary infrastructure).
> Yes, you can still use python2, but if you depended on pip for your build process (which I would consider "infrastructure"), then you are out of luck for this
Using python2 is acceptable for you, but using an old version of pip is not acceptable?
My problems are largely coupled heavily into the python2 that my package manager is written in, which as far as my team tells me, is python2. And that it's becoming difficult to integrate against products we have to continue to release for another five years of corporate sunsetting.
