Indeed. We're hitting this on some Ubuntu 16.04 (supported until April 2021!) build containers which start their work with the suggested self-upgrade of `pip install -u pip` that then bombs out with SyntaxErrors on subsequent operations due to f-string usage.
We (my team) hit this as well with an automated compatibility test running on Ubuntu 16.04. I decided to just change the test to install Python 3.8 from the deadsnakes PPA, and then use get-pip.py to install pip.
This does change the test to no longer use Python 3.5, but considering a) we'd already dropped support for Python 3.5 for the code we were testing, b) Python 3.5 is EOL and c) Ubuntu 16.04 will be soon, it didn't seem worth trying to stay on 3.5.
This approach is okay if you can work entirely in an environment (venv or otherwise) that is unmoored from the underlying system. But usually the reason to use an older OS is because you want to be linking to specific versions of boost or some other dependency that's a pain to build and install from source, so depending on your scenario, installing a deadsnakes python may not work if you're depending on there being a corresponding `python3-xxxx` package that links up system python to bindings for a particular library.
Isn't that a packaging bug? Shouldn't py<3.6 not see pip packages that require 3.6+?
Also, if you are using system-managed Python for stability you should probably also be using system-managed pip rather than upgrading from pyPI, for the same reason.
Yes, I think it is a packaging bug, but it's reflective of an attitude where old stuff is hung out to dry even before the officially-committed time horizons have expired. This would be fine if it were possible for distro maintainers to pick up the reins, but the pip maintainers have deliberately wrested control from them— they've basically decided that their recommended approach is for everyone to use the latest pip regardless of what interpreter they're using, especially in the case where you're using that interpreter in a virtualenv. For example on an Ubuntu 16.04 machine if you create a new venv, the system pip that gets installed there immediately runs off to the internet to check itself and then prompt you to upgrade it:
$ python3 -m venv test
$ test/bin/pip install pyyaml
Collecting pyyaml
Using cached https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz
Building wheels for collected packages: pyyaml
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/administrator/.cache/pip/wheels/2a/d4/92/cf299bdf4162957ca8126b46e913e29f76a4f17ca762c45028
Successfully built pyyaml
Installing collected packages: pyyaml
Successfully installed pyyaml-5.4.1
You are using pip version 8.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ test/bin/pip install -U pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)
100% |################################| 1.5MB 405kB/s
Installing collected packages: pip
Found existing installation: pip 8.1.1
Uninstalling pip-8.1.1:
Successfully uninstalled pip-8.1.1
Successfully installed pip-21.0.1
But then:
$ test/bin/pip
Traceback (most recent call last):
File "test/bin/pip", line 7, in <module>
from pip._internal.cli.main import main
File "/home/administrator/test/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
sys.stderr.write(f"ERROR: {exc}")
^
SyntaxError: invalid syntax
Note that the prompt doesn't just show up in a virtualenv. The system-installed pip has this behaviour as well, and there are some packages which don't install correctly with a pip this old, so the only thing to do is manually track a working version of pip, force-install exactly that version, and ignore the repeated prompts to upgrade.
> it's reflective of an attitude where old stuff is hung out to dry even before the officially-committed time horizons have expired.
As mentioned in another comment, the “officially-committed time horizon” is imposed by Ubuntu, and it does not make sense to apply it on pip maintainers. Python 3.5 was retired by CPython in September 2020.
Unnecessary snark is unnecessary. We already did the bulk of our 16.04 -> 20.04 upgrades last summer, and were planning on mopping up the remainders over the next two months, while there was still time. That's the point of these support commitments with clearly defined timelines— allowing people to plan for this stuff to happen when it suits them.
Now we have a handful of broken support VMs and other things which have to be migrated in a rush instead of according to the planned schedule.
You missed my point. Ubuntu packages pip via apt, not via pypi. You can swap from an installation provided by apt to one provided by pypi using your method, but it means you're off the supported version and any guarantees of support are lost. The quick fix here is to not update pip to the latest, but the latest supported by python3.5 and carry on with your migration as scheduled.
I realise that this is a common practice, and I've been caught out myself in the past with CentOS7+Python3.4, but be aware that it comes with its own set of tradeoffs. Sadly, the packaged version of pip is usually quite old and pip is loud about outdated versions, so the solution is either to upgrade the OS or leverage something that is not as tied to the OS.
If you want to explore the latter, I'd suggest python:3.5 containers if you're building pure python code. Alternatively, I'd suggest manylinux containers if you have C extensions and run on a glibc distro like Debian/RHEL. Both will have the last supported version of pip for your python version installed and neither will randomly break on you.
The final lesson I've learned is to be aware that pip can install the latest version supported by your python version as long as it can access the metadata for the package in the index. That's available on pypi, but not all self-hosted indexes have it, notably nexus.
So I wondered if there was an issue here where pip 8.1.1 is just so old that it doesn't know how to respect that constraint? As an experiment, I started with a new venv and upgraded just to the latest pip 20:
$ test/bin/pip install pip==20.3.4
Collecting pip==20.3.4
Downloading https://files.pythonhosted.org/packages/27/79/8a850fe3496446ff0d584327ae44e7500daf6764ca1a382d2d02789accf7/pip-20.3.4-py2.py3-none-any.whl (1.5MB)
100% |################################| 1.5MB 695kB/s
Installing collected packages: pip
Found existing installation: pip 8.1.1
Uninstalling pip-8.1.1:
Successfully uninstalled pip-8.1.1
Successfully installed pip-20.3.4
You are using pip version 20.3.4, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Now interestingly I don't get any prompts after that, and it doesn't try to upgrade past 20.3, so I think that might have been the last gasp of pip 8.1.1:
$ test/bin/pip install -U pip
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
Requirement already satisfied: pip in ./test/lib/python3.5/site-packages (20.3.4)
Collecting pip
Using cached pip-20.3.4-py2.py3-none-any.whl (1.5 MB)
Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
So yeah, basically the fix is just to hard-code the version you know works. But I think this demonstrates an issue with the PyPA being out of touch with real-world use-cases. No one is upgrading pip version by version— 99% of pip installations are either going to be get-pip.py or upgrading a venv directly from a distro-shipped version. IMO those use-cases should be carefully tested on each release and checked for the versions in popular distros, even at the tail end of the support windows.
The other side of this is Debian/Ubuntu, of course, and with sufficient coordination (knowing the a pip was coming which their "supported" pip would auto-upgrade to and break itself) they could have patched their `python3-pip` package to only upgrade as far as 20.3.4.
Both Ubuntu 16.04 and Python 3.5 are end-of-life. Older versions of pip will work, but newer versions drop the legacy compatability. If you have an out-of-date system you can still use an old pip just fine.
16.04 is EOL on April 30, 2021. Even so, the idea of expecting to run the latest version of a package on an stability-focused operating system released almost 4 years ago seems hopeful.
It's supported until the end of April. Obviously we're in the process of migrating things, but it's either supported or it's not, and as long as it's supported, it should work.
Using the latest pip is the will of the PyPA, and they let you know this on every invocation that touches the network:
You are using pip version 8.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
There are also packages and workflows which break on the older pips, and this has been justified on the basis that everyone can always just use the latest.
> Other than f-string, I'm not sure what critical features were brought in with 3.6.
* type annotation for variables
* asynchronous generators
* asynchronous comprehensions
I can see any of those three seeing use in pip and being worth dropping earlier Python for more than f-strings (or underscores in numeric literals, the other highlighted py 3.6 new feature.)
