Environments, like the interpreter itself, seem a singleton concept.
I have used a Makefile that sources different Bash environment variables kept in files in etc/ within the venv to switch between, say, a Flask and a Gunicorn startup.
I've used a nested git repository, with the child directory in the .gitignore of the parent. I know that there is some thing about installing git checkouts with pip, but that has not been important to me yet.
I honestly can't even come up with a valid reason for something like nested venvs. At that point it seems more like someone's doing something wrong than 'things get tricky'.
If you have unlimited disk space, sure. Otherwise, it would be really nice to be able to layer venvs so you don't need to have multiple copies of a package installed.
My virtualenv for work comes in at 847Mb (actually a lot more than I expected). I have a 2Tb disk attached to my machine. For all intents and purposes, we do have unlimited disk space.
If anything it just simply seems like a dumb idea to do so. Virtualenv is simple and works really nicely. Why anyone would want to add unnecessary complexity is beyond me
Could someone ELI7 why python is struggling so hard with dependencies and dependency management when in the Java world this is a solved problem and just works with maven, which is working so damn well it last got updated in November 2019, and the biggest fight going is whether you should use gradle as client instead of maven to access the same dependency ecosystem?
I had to set up Python projects for some machine learning classes in college and it was a complete mess.
I am probably in the minority, but my experience is the complete opposite. I had more trouble trying to set up a gradle project in university than I ever had with Python.
I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python.
C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy).
Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.
Or to be precise, it is a solved issue -- several times over. After years of meandering, I think it is safe to say that deployment of Python libraries is pretty much resolved in one, general way: we have the wheel/PyPI/pip toolchain, and that's about it.
On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from the underlying system in a universal way. There are different methods, from virtual environments (both standard and nonstandard ones like conda), to version managers like pyenv and asdf, to "jailers" like pipx and pyinstaller, to containers; but none of them has yet clearly won.
All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. End users cannot set up virtual envs and pip install stuff into them (which sometimes needs external tools like make).
The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.
I've used pyinstaller. My experience was that pure Python was easy; things that needed to use C shared objects increased in complexity exponentially based on the number of C dependencies. If it's pure Python, you can basically run "pyinstaller <spec file>" and be done. I tried to make PyQt5 work in it, and it was extremely painful. I don't think I ever actually made it work, I think I just gave up. That being said, PyQt5 is probably the most complicated thing you'd ever package with pyinstaller, so it may not be a great test.
> All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users.
virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.
Almost nothing should do what JavaScript does, not even JavaScript. Of all the interpreted languages I’ve used, JS has the single worst ecosystem for running production code. If your experience is primarily with JS, do yourself a favor and try to grok other languages and ecosystems.
You are assuming there is a "project", but Python code with massive dependencies can (and often should) be a bare script in an arbitrary folder that runs by invoking the Python interpreter on it, without "installing" anything except missing dependencies.
That's not a bare script then, it's a project with dependencies that need to be installed. This misunderstanding is how you fuck up a developer environment.
I'm surprised this is getting downvoted. This is the only sane way to manage dependencies. They must be relative to the thing you care about.
There are tradeoffs for sure, like the ridiculous sizes of those directories on development machines. But a giant SSD is cheaper than my time resolving dependency issues on user machines.
They essentially are. Not in the actual directory but in the virtual env and not cluttering up your project directory. This is superior to JavaScript in my opinion
No language should replicate anything JavaScript does when it comes to package management
Yeah no kidding. I prefer it as a prototyping language. Unfortunately, I work in networking, and it's become the raison d'etre for our "DevOps". At least we're not using Ansible anymore... :|
> You don't have to deal with virtualenvs or packages from different projects conflicting.
That's only because you're basically assured there will be conflicts inside any single project. Every time I install anything via npm I get a boatload of warnings about insecure dependencies, which are effectively impossible to fix without breaking the whole mountain of hacks.
There are good deployment stories out there. JS is not one of them.
> That's only because you're basically assured there will be conflicts inside any single project.
That's not an issue with the package management though, that's an issue with the quality of the packages themselves. The JS ecosystem does have issues there. But the tooling itself is good.
Babel and Webpack aren't package management tools. I guess there is now yarn in addition to NPM. But NPM works just fine. And furthermore they interoperate.
Java is a self hosted language and has very little dependency on C/C++ or something similar.
Python is glue language for C/C++/Fortran/Rust/etc. projects. Most of the valuable packages in Python in fact C++ projects (like Tensorflow, Pandas, etc.). When you are talking about Python dependency management you are talking about the Cartesian product of the packages you are depending on, their dependency in terms of compilers and header files. It is not hard to run into missing .h files while trying to install these dependencies. This situation was improved significantly a while back with the WHL files.
Anyways this whole article does not make too much sense, I am using Python for 10 years professionally and never run into the concept of nested venvs let alone somebody trying to use that in production.
Compiled dependencies is indeed one of the main reasons. Another one is lack of enforced, static metadata because python started before most other languages.
That's just dangerous band-aid. If there is a mismatch between bundled .h and installed system library, you'll get (in better case) linker error, or runtime problems.
At that point, you should just be installing a binary wheel.
(Nb I’m a maintainer of a python project that’s difficult to build, and the number of issues we get that start: can’t install because foo.h can’t be found is very high, even though there’s other explanations closer to the end of the failed build)
There are several reasons that people don’t get the binary wheels, but a lot of the time it’s either pinned dependencies where they’re requesting an old version while running 3.9, that window between a new python release coming out and our quarterly release schedule, or an unsupported arch for binaries like Alpine, the m1 macs, random android.
Ah I see, that makes sense. I believe Node modules typically fall back to building from source in those cases. Which can be a little slow, but generally works reliably.
I can see that, but we rely on a bunch of system libraries (libjpeg/zlib, + optional libtiff/freetype/etc, and python headers as well). Pretty much by definition, if we don't provide binaries, it's not going to be that simple.
I'd say maybe 25-50% of developer linux workstations have the appropriate packages and will compile out of the box, and if they don't, it's almost always a one line command to install them with a package manager. We've got instructions or docker images for testing for most major linux distros.
MacOS is probably in the 5-10% range for successfully compiling out of the box, some of the headers ship with MacOS, but there are a couple steps to go through. M1 is still a WIP in the python packaging community.
Windows... 0% compile out of the box. You have to set it up, download dependencies and so on. There are at any one time a handful of people who can compile for the windows platform, and a smaller handful of people who can debug problems.
Now, to be fair, I think this project is probably in the most complicated 1% of python projects from a packaging point of view. There are probably more complicated ones, and the only two complications I'm aware of that we don't have to deal with are 1) The chicken and egg of if we break an update we lose the ability to update all the other packages including our own (pip) and 2) GPU drivers.
My secret is to build a new computer every year and keep images of the old ones with all my packages and libraries. Trying to run a tensorflow GitHub library from 2013? Easy just use the computer with that image
People always criticise Python for dependency management, but I've never found it to be that bad. virtualenv with requirements.txt has been completely fine in my experience with medium-sized projects. If necessary you can pin versions with a separate version-locked requirements.txt, and populate that from a base requirements file.
My ELI7 understanding is that python installs required modules (libraries) at the system level by default, and doesn't have good support for installing modules into local directories like node/C/java/etc. does, which leads to conflicts. Users have to find a set of modules that support all of the scripts on their system. The current workaround is to isolate each script's requirements, but there are a ton of competing tools to do this (virtualenv, pipenv, poetry, etc.), so it's hard to choose the "best" isolation solution.
One reason people like PEP 582 is that, by being something built into the interpreter, it will reduce a lot of the confusion over which isolation tool to use.
Part of the problem, which local installs won't completely solve, is that the python interpreter is still changing fast enough that modules are having trouble keeping up. E.g. tensorflow is still working on python3.9 support [1], and now we'll see python3.10 in the next couple weeks. Even with local installs we'll have to be careful to use the right interpreter.
Another piece of the problem, which is also kind of a good thing, is the huge size of the python module ecosystem. I've got python scripts with over a hundred requirements. It saves me from writing a lot of code, but resolving the requirements is a challenge. The pip package manager has recently added a dependency resolver, which has helped a little make sure everything can work together.
The only surefire ways I can think of to make this huge module ecosystem interoperate cleanly are (1) have every module support every other version of every other module and interpreter (insane amount of work, won't happen) or (2) have simpler interop interfaces like the unix shell convention of only passing strings between programs, but that would take away a lot of the ease-of-use of passing rich objects around between modules.
Tell me about it. I had cudatoolkit installed, and then followed every single PyTorch installation instructions, and then still it insisted that I should get the CPU version, not the GPU one.
I ultimately fixed this with going to the conda files list, and choosing exactly what I needed. Not straightforward at all.
CPAN I think only recently got virtualenv-like support; before it was much worse and a pain. Where CPAN modules shine are they're more stable, backwards compatible, and extensible. Modules whose last release was 15 years ago work fine today, and new modules are extended on them.
This was a good article and looks like an interesting project.
Regardless of whether or not you choose to _deploy_ with Docker, _developing_ in Docker containers using the VS Code Remote extensions really solved all of Python's (and Javascript's) annoying packaging problems for me, with the bonus that you also get to specify any additional (non-Python) dependencies right there in the repo Dockerfile and have the development environment reproducible across machines and between developers. YMMV, of course, but I find this setup an order of magnitude less finicky than the alternatives and can't imagine going back.
That's my point. I've always found it hard to set up dev containers. Also each dev container requires its own container, so I may have 10 python docker containers. My implementation only requires 1 image and temporary containers.
I do most of my development in docker. It's tricky to start but once you get the hang of it, it's fantastic. My goal is to one day roll something like RancherOS and provision my entire desktop environment in containers. Then you can just pick up and move it to whatever machine you wish.
Yeah, I read that too and thought he made some good points.
I probably should have been more careful to qualify my enthusiasm above. I’m definitely not suggesting that everyone that has a Python packaging problem should go out and solve it with Docker. As with all things, there are trade-offs and you should satisfy yourself that Docker is the right tool for your particular problem before diving in.
I just wanted to highlight the fact that - as a VS Code user who has some experience with Docker and writes in a couple different languages and ecosystems - I’ve had a great experience with this modality and find that overall it simplifies my workflow.
I wish the tooling around devcontainer.json was available separately from VS Code. Most of my company are die-hand vim users. We’ve rolled our own “devcontainer” analog but it’s not nearly as seemless.
Thanks for giving a look and leaving so many reactions. This project was originally built to experiment on some new PEPs on Python packaging and it turned out working well.
1. What does PEP 582 differ from virtualenv, it is just another "virutualenv"
The biggest difference, as you can see, is __pypackages__ doesn't carry an interpreter with it. This way the packages remain available even the base interpreter gets removed.
2. I prefer Poetry/virtualenvwrapper/Pipenv
It is fine. If you feel comfortable with the current solution you are using, stick with it. I am not persuading that PDM is a better tool than those. Any of the said tools does a great job solving its own problems. PDM, IMO, does good in following aspects:
- Plugin system, people can enhance the functionalities easily with plugins
- Windows support, PDM has completion support for PowerShell out of the box. Other CLI features also work as good as *nix on Windows.
How does PDM deals with binary python modules like foo.so that in turn depend on other binary libraries like mylib.so? This is what conda manager and conda environments are great for, covering the ground for all major platforms.
It is decidedly not true that I want to update my venv for every minor version bump!
I deploy to a cloud service w/ a specific version; my package manager is slow to update; I develop collaboratively using a shared container w/ a fixed version.
Updating shared resources with every new release is not always realistic, and that makes it so that I do need (or at least want) to use a virtualenv.
> It is decidedly not true that I want to update my venv for every minor version bump!
Until you run into a minor version bump that patches a critical security vulnerability, and then you suddenly may very much want to (and in some corporate environments, have a regulatory requirement to)
That’s not every minor bump, though, and is worth the extra effort!
Also, my use case is 100% non critical from a security standpoint, so I can afford to be careless... your point definitely stands in sensitive environments
My first attempt, and I get "AttributeError: module 'toml.decoder' has no attribute 'TomlPreserveCommentDecoder'"
So careful, far from stable.
I reported it (https://github.com/frostming/pdm/issues/247) because the project is great and I want it to succeed, but I won't put it in prod any time soon. I prefer my package manager to be stable than fancy.
It's also why I don't use poetry and so on in training. There is always some bug that students will encounter later because there are so many config combinations they can encounter, but recent tools have not been as battle tested as we think: the enterprise world is a wild beast.
True, bugs can't be exposed by a few users. People have vastly different environment setup for their Python development. System python, in-venv, pyenv, asdf, homebrew, etc. So I post it here to seek for help testing.
The sematic versioning may be a bit misleading, it is just because there was a big breaking change to support PEP 621, and I marked it as "Alpha" stage.
Sometimes you don’t realise how onerous things are until you find a better way. Poetry is that way.
Handles dev dependencies, scripts, virtualenv management, upgrading single dependencies, has a lockfile, is super fast, is built on pyproject.toml, etc etc.
Honestly, just use Poetry. At the very least it’s better than ad-hoc personal scripts built around manually managed virtualenvs.
the venv management is a small aspect. pyproject.toml is nicer and saner than setup.py/requirements. easy example: dev dependencies in poetry. poetry's dependency resolver is great. can upload packages with poetry, twine not required (good for CI). works in fish shell, which virtualenvwrapper doesn't. lock files for deploying exactly the same dependency versions.
those are the ones off the top of my head. use poetry, it's great.
One value add is a reduced cognitive overhead. I don't have to remember where my virtualenv is; Poetry remembers for me.
The "poetry run" command is extremely nice. You can do something like "poetry run pytest" and put that in a Makefile and be done. You don't have to worry about making sure you're in the virtualenv. I basically don't drop into a full shell anymore (although "poetry shell" lets you do that) because I don't like having to juggle whether I'm in a virtualenv or not.
"poetry build" will build your project into tarballs that pip can install. It's not earth-shattering, but it's nice to have it one tool.
The value add is a large number of QoL improvements. I don't think there are any features that other solutions can't compete on; Poetry just makes it easier to do the same things. I would highly suggest it for anything where other people are going to work on it, because of the QoL, but for personal projects I don't think it really matters.
Dependency resolution is the biggest win with Poetry for me - everything everyone else has already said too, but the guarantee that it's only going to install versions of packages that are compatible with each other is fantastic for peace of mind. I can't imagine every using a requirements.txt again.
Poetry shell doesn't work on windows, and it doesn't have named venv which matters for some users. Also it's very slow, require to be installed and use proprietary fields in pyproject.toml.
It's a great tool but there are reasons you might want something else.
Alternative like raw pip, pip tools, or dephell all have various pros and cons.
>doesn't set the name of the project in the prompt
I agree with you there, it's very annoying and I've fallen for it a few times. Having not used Poetry on other platforms yet (still using Pipenv in a few projects), I wasn't even aware that this was a feature.
Not a solution, more a workaround, but on Windows I highly recommend developing using WSL, especially for Python. WSL + pyenv + poetry + VS Code with Remote WSL works amazingly well.
A virtualenv is just a bunch of packages, you end up with the same thing whatever you want to call it. It is probably a more roundabout solution that it needs to be, but I don't find many problems myself (also using poetry).
I haven't used Poetry and most of my workplaces have done their own thing with versioning Python packages. So my virtualenv experience has been on my own or with toy projects.
I really like writing one-off or sporadically used commandline tools. Loading a virtualenv or navigating to a directory before executing kind of sucks. Generally, I install into my homedir. I haven't read through or tried this out, but it seems like it might better address this use case.
You're right, this seems to be very similar. To me, it looks like it removes one abstraction layer of pushing-popping the environment.
You could add that bin/ to your PATH, but you'd also need something else to load the venv before executing it. You could write a generic wrapper that would load the env and call the Python binary and put that in a more generic bin/, but that's the kind of thing this proposal seems to make irrelevant.
Another issue with wrappers is how quickly it executes. It's not a big deal for most executions, but for --help or building up a command pipeline it's really annoying to wait a second or two on each execution. That slow startup time also shows up when you start integrating it into automated processes.
He also said virtual envs install their own copy of Python. No need for that either. Coming from other languages the whole virtual env thing seems alien especially the way it messes with your shell.
A virtualenv (venv) does not install its own copy of Python. Instead, when you create the venv it puts symlinks for the calling python in the <venv>/bin directory. When you activate the venv the <venv>/bin is added to the front of your PATH so that any "python" commands use the symlink'ed version of Python corresponding to your venv.
Indeed, a virtual env is really not that exciting or complicated. Even "activating" a virtualenv is just changing your shell's PATH so it picks the virtualenv binaries, nothing else. You can also just call those binaries directly from their installed location in the env without ever having to "activate" or "workon" anything.
It's not optional. It's required for things to work properly.
A basic example of something that would break if the environment is not activated is binaries, like mypy.
You have a script that calls "mypy"? Gotta be in the virtual environment to invoke it, otherwise the executable might not be found, or worse, it might run another executable.
It's absolutely not required, I assure you. If you run the Python binary inside the venv you have the same environment as if you'd activated the venv, but only for that execution and not affecting your shell. Try making a venv, running python, and looking at sys.path.
I assure you 100%, it is required for everything to work, it's not just here for the show.
One of the thing that the virtual environment does is modify the PATH to lookup executables from the virtual environment first.
This is to ensure that when a script calls binaries like mypy/python, it's going to find the same binary from the environment it's currently running in.
Typically if the environment is not activated, python is going to lookup to /usr/bin/python and might be something else.
This is most noticeable when there are different versions of python on the systems and when having scripts that invoke themselves.
I see we will not agree, and this thread is mostly dead anyway, but for posterity: it sounds like you haven't installed mypy into the venv you're using. I can see how that would not work. I don't try and reference python scripts that I haven't installed using the venv so perhaps that's why I've never encountered this situation. If it's part of my project it's installed into the venv (and if it's a dev dependency like mypy, which I do use, it gets installed as a dev dependency so it doesn't end up in releases).
Thanks for writing this. I've always felt that virtualenv was bit of a placebo. I still use it from time to time though because others expect it. In fact I have used it this week on MacOS, and was annoyed to find that I can't install ipdb to my user packages and still see it in a virtualenv. Seems it's either global or local. (I think the --system-site-packages defaulting to off has it backwards, but it is probably reassuring if you don't know how the module finder works).
Regarding nested virtualenvs, last time I checked there was no such thing. When nested, they always linked back to the system level python, and the cloned python had loads of dependencies on the global system. I'm not aware of a valid use case for nesting virtualenvs.
There is one handy feature of virtualenv that I'm not sure __packages__ solves though: console_scripts and entry_points. setuptools will automatically create a console script under /usr/local/bin or similar which hooks in your module under site-packages. virtualenv includes a ./bin directory and adds it to your path so that these console scripts still work locally. Any thoughts on this?
__pypackages__ also has a bin/ directory and you can run the console_scripts with pdm run my_bin or by absolute path: /path/to/__pypackages__/3.8/bin/my_bin
Curious, what about environment variables? I like the ability to put my environment variables in my activate script. It seems like if you did things the way the author does, you would need all of your env variables to be global, which could be hazardous
props to the author of this tool. based on what I read from the blog, it's already a step above poetry, which a lot of people rave about recently, but still not good enough not to not use pip. question is, are we going to be nuking the __module__ folder like we nuke node_modules every now and then ? other than I will become an early adopter of the tools. currently using miniconda for my virtualenvs
> are we going to be nuking the __module__ folder like we nuke node_modules every now and then ?
Two thoughts:
1. Part of the good aspect of the node_modules is its nukeability. It's a good thing that we can blindly nuke and `npm i`
2. With node and react (I have 5 node projects and 2 react projects that I work on regularly), I really rarely have such error that I need to nuke the node modules. `npm i` to update dependencies and restarting the server because it got stuck in a weird state is usually the solution. I nuke a node_modules maybe every 3 months, probably less. On react-native, yeah it's still very much a thing though
Not sure about ruby, but most of the languages I know about, C#, javascript, php, etc don't need virtualenv because they install packages in a project locally (and globally if required). so envs of one project do not clash with another.
while python install packages globally and only globally. so a virtualenv is needed to isolate different projects from one another
the thing is python cli has no concept of locally scoped libraries. you can install libs wherever you want, but all python.exe will refer to the same global location.
this is why you need to activate virtualenv every time you open a new command line, so that it can set ENV variables for the current session and force python to use the locally scoped folders.
While in C# et all. the compiler knows to check files locally. and you don't need to activate (or deactivate) environment at all.
That's an optional environment variable, it is not set unless you set it. Meaning you could change it to anything, per process. Also you can change sys.path. Not an issue.
For development, the "install everything locally" that virtual env provides. This avoids conflicting, incompatible versions of packages being installed in a global or per-user location.
Python isn't at all the only language to suffer this problem. e.g. "DLL hell" and its variations like Haskell's "cabal hell". With Ruby, if its packages are installed with bundler, then these packages are installed locally in the project directory.
People use RVM or rbenv to manage Ruby versions. The virtualenv analogue is Bundler, which provides something like requirements.txt and causes gems to be installed locally instead of globally. https://bundler.io/
The basic facilities are baked into python tools too, you can direct installation of packages to arbitrary folders and then just add them to your PYTHONPATH.
But it's not as quick and clean as "python -m venv venv", so everyone uses that (and pyenv for different versions).
well, a lot of "system" utilities are implemented in Python and I'm not aware of many in Ruby (they must exist? I just checked to see what on my system depends on Ruby and it seems like the only things are a screenruler utility, some texlive packages I've never heard of and gnome-code-assistance, whatever that is). Pip installing random crap is a great way to completely break some Linux systems.
Absolutely... I usually do python setup.py bdist_rpm if I want to install some package systemwide. Not sure it's really any safer, but it sure feels safer...
Conda is so slow that I resorted to using it for env creation and then just pip install everything. I don't have 10+ minutes to wait for package to be installed, only to be told it can't resolve conflicts. Package version conflict is not such a big deal, just pip install a different version of the offending package!
This looks interesting, but personally I don't have any problems with virtualenvs. I use pipx to install "global" tools and virtualenvwrapper to make project envs. It all works fine really.
As I wrote in the post, you cannot make two tools isolated in their own environment work together(work together meaning one needs to import from the other). The only way to make it is to install one tool in the global site-packages and create the other virtualenvs with --system-sitepackages and this flag is off by default.
Regarding the installation of self-contained tools, this is a problem already solved by pipx, which is very simple to use. I wished such a system was included as part of official python installers.
Never had such a requirement.
Environments, like the interpreter itself, seem a singleton concept.
I have used a Makefile that sources different Bash environment variables kept in files in etc/ within the venv to switch between, say, a Flask and a Gunicorn startup.