About a year ago or so I moved to pyenv and poetry. I have used python for over 10 years it was a game changer. I recommend this guide if your still using pip and venv.
Honest experience: I switched to this early on because at the time PyPA was pushing it really heavy, and Kenneth's past experience at Heroku meant it was added to their Python buildpacks far too soon.
At the time, it was unusable and community responses wildly swung between "its not official", its "PyPA endorsed", "don't like it build your own". Which are fine responses beta version of software, not when you convince the community and several companies that its a stable production ready, langauge endorsed package manager. Ultimately, pipenv in Heroku python buildpack was broken for a long time because of it.
Despite being assured it was stable, there were daily Pipenv releases at times, until they stopped for some reason for months.
In summary: avoid pipenv unless you really have to use it, otherwise - use poetry.
Oops. What I meant is, Pipenv manages python versions for you. You tell it what version of python your project uses, and pipenv will make sure to create the venv using that version of python. Great! Solves managing python versions.
With Poetry, the venv is created with whatever version of python that Poetry was installed with. So if you (like me) follow the setup guide on macOS, Poetry will always create python 2.7 virtualenvs until you uninstall it and re-install it under python 3.X. Annoying.
It will only use whatever version is currently activated if that's compatible with the python version requirement specified in pyproject.toml. If it isn't, then it will look for an installed python that is, and create the virtualenv using that instead.
I'm not sure I completely love that behavior; it would be nice if it would default toward trying to use the minimum supported version, even if the currently active one is compatible. But I guess that wouldn't do anything that tox couldn't do better.
You can also explicitly tell it what version to use for a project with a command like `poetry env 3.8`.
Actually, I tried that the other day. It complains about having the wrong Python, but will auto-select a version compatible with your project and run fine. For me, it selected Python 3.7 because that's what the pyproject.toml specified, even though I was running it under Python 2.
What does this tool provide over an in-project .venv/ directory created by venv and populated by pip from an etc/pip/requirement.txt file containing all the packages by version?
There seems to be a lot of activity in this area for a seemingly solved problem. Having solved it, people appear to wish to expand the problem to something bigger (such as multiple interpreter versions, for example)
I believe the most-common case to be:
1 Choose python version and install on operating system., 2 create .venv/ directory. 3, install packages from package system (pip)., 4 freeze versions to local file. 5 use installed .venv/ and do work.
I've got some scripts I use for the above [1] -- must give it some TLC, and I don't understand why people furiously want more than this.
It is a solved problem for other languages. Pip has soooo many issues.
The biggest IMO is pip doesn't provide a way to pin indirect dependencies, so there is no good way to ensure that other people working on a project are also using the same dependency versions. You can do it with extra tooling of course - and that's what poetry and pipenv do.
Even worse-- there's no simple way for pip to print out the full dependency tree (including transitive dependencies). This makes it a nightmare when it complains about some transitive dependency version conflict and you have no idea what's causing it to be pulled in
What do you mean? If you run pip freeze inside a venv it will print every user-installed module and their dependencies. When installing them in a new venv as a requirements file, you end up with the exact same set of modules.
Agree. Works fine for me exactly as you describe. I don't understand why people furiously want more than this for the common case. (Choose python version, create .venv/ directory, install packages, use installed .venv/ and do work.)
Sometimes it works, sometimes it doesn't. This is indeed a problem, but one for which there isn't an elegant solution.
When multiple versions of the same library are allowed to coexist in different dependency trees there is less incentive to solve these conflicts and versions proliferate (see npm).
I'd much rather solve the occasional conflict (usually solved either by settling on some older version of both parent dependencies - and later advancing as possible - or just trying to fix it and send a patch upstream).
Issues start occurring when you, for example, delete dependencies because with pip freeze you cannot ensure that you have deleted their dependencies also. The most common solution to this (that poetry and pipenv use) is to provide a lockfile to track transitive dependencies and their versions, without that (except for manually curating your dependencies) you can't ensure that you get a reproducible environment.
OK, but it does manage the versions of transitive dependencies, and there's nothing in that process stopping deterministic builds.
Adding/removing top level dependencies over time does require the use of two files (the top level requirements and the frozen/locked requirements which lists everything). Or you can list the top level requirements in setup.py and let requirements.txt be the lockfile. It would be nice if pip managed this lockfile automatically, but I'm not really interested in adding any of these newer tools to my toolchain just to manage a lockfile.
There are many packaging and distribution frustrations in Python, I don't think pip's management of dependency lists is one of them
I just so wish that Python tooling would be better, in large part pip is at fault here. If there was no conda I probably wouldn't even touch Python as things look very bleak especially if you need to run anything across different operating systems.
The consensus among non-Python developers is that that is not good enough.
The problems are:
requirements.txt is way too flexible, so users want a lock file to freeze stuff in place.
You can use pip freeze to generate a sort-of-lockfile but it's more manual than tools like Cargo/npm/bundler which do it automatically. And more adhoc as to what you call it. Combined with needing to mange venvs and sourcing etc. people want a standard script that just figures out its context from the now standard "explicit requirements"/lockfile setup.
It doesn't handle versioning Python itself, which is expected of it for some reason, even though nobody cares that npm/Cargo/bundler don't version their languages.
I felt the same way about only needing requirements.txt until I managed a project that needed to be compatible across many versions of python and some of it's dependencies were renamed from one version to another. I highly recommend you take a look at the hypermodern python guide https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
Pipenv and poetry help, but, even with them, it's still not good enough. setup.py is a real challenge. Packages being able to execute arbitrary code at install time can make it very difficult to get a truly reproducible python build. Details of how pip (which pipenv and poetry still use) handles packages means that doing anything that even hints of a monorepo is a delicate subject on the best of days.
And the solution to the multiple interpreter versions problem should be orthogonal to venv like pyenv [1] instead of overlapping with venv like pipenv.
We had a medium size app and it took literally 30 minutes on a MacBook Pro to resolve (not install, just resolve) dependencies. This was a few years ago so perhaps performance improved, but it was abysmal and made it impossible to meet our goals of fast CI/CD.
Seeing all the comments in this thread that people have had bad experiences with pipenv, this contrasts with my own experience which has been pretty good. How has pipenv failed for everyone?
Gonna take a look at poetry, but would love to hear what problems people have had with pipenv?
1. Dependency resolution algorithm failed in places that Poetry handles fine.
2. When Airflow added a dependency choice (based on licences) the setup script required an ENV var choosing one to be set, pipenv swallowed the stdout output and dumped a Pip stacktrace that didn't help diagnose the issue.
3. Mysterious bugs on new pipenv releases that usually manifested as a Pip stack-trace or setuptools stack-trace.
4. Sometimes the bugs were installed Pip version dependent, which made replication hard.
There was other stuff, but can't quite remember off the top of my head.
But yeah, we spent a lot of time trying to figure out why pipenv had broken again. Poetry has been a breeze in comparison.
There is also Pdm now I recently learned, which is supposed to be a more modern alternative. As an alternative to poetry that is; pipenv is yesterday’s solution.
Ah, that uses PEP-582-style package directories, that's interesting! Though I don't know if we need an alternative to Poetry, I'd rather we just standardized on one tool at this point.
There wasn't a full release for a year and it broke my pip on Ubuntu 18.04 as I recall - literally it stopped pip from working after I installed it (can't remember why). Avoid.
I had also the terrible misfortune of trying it on Ubuntu, when dependencies were screwed there were no way to fix it even manually, it was like adding insult to the injury that is pip itself.
Conda's a good replacement for the whole pip/pyenv/venv/stuff-that-isnt-technically-python-that-gets-hacked-in-with-setup.py mess, but, if you're publishing packages, pipenv and poetry go a step further to help with building those as well.
https://cjolowicz.github.io/posts/hypermodern-python-01-setu...