This is great, but sometimes I think that python needs a new package manager from scratch instead of more tools trying to mix and mash a bunch of flawed tools together in a way that's palatable by most of us. Python packaging sucks, the whole lot of it. Maybe I'm just spoiled by rust and elixir, but setuptools, distutils, pip, ez_install, all of it is really subpar. But of course everything uses pypi and pip now, so it's not like any of it can actually be replaced. The state of package management in python makes me sad. I wish there was a good solution, but I just don't see it.
Edit: I don't mean to disparage projects like this and pipfile. Both are great efforts to bring the packaging interface in line with what's available in other languages, and might be the only way up and out of the current state of affairs.
I think python packaging has gotten LOTS better in the last few years. I find it quite pleasurable to use these days.
From binary wheels (including on different linux architectures), to things like local caching of packages (taking LOTS of load off the main servers). To the organisation github of pypa [0], to `python -m venv` working.
Also lots of work around standardising things in peps, and writing documentation for people.
I would like to applaud all the hard work people have done over the years on python packaging. It really is quite nice these days, and I look forward to all the improvements coming up (like pipenv!).
I'd suggest people checkout fades [1] (for running scripts and automatically downloading dependencies in a venv), as well as conda [2] the alternative package manager.
+1. Relatively to what we have before, it's so much better. But compared to the JS/Rust ecosystem, we are behind.
Now it's hard to compete with JS on some stuff : it's the only language in the most popular dev plateform (the web) and it has one implicit standardized async model by default.
It's hard to compete with rust on some stuff : it's compiled and is fast, can provide stand alone binaries easily and has a checker that can avoid many bugs.
But this. The package manager. We can compete. And yet we are late.
It's partially my fault since it's a project I had in mind for years and never took the time to work on. It's partially everybody's fault I guess :)
I'd strongly disagree with Python being behind relative to JS. They're roughly on par (with Python being slightly ahead due to smarter caching and more deterministic builds). Unfortunately neither does namespacing nor good versioning, especially in the cache. Take for example Maven (from Java), it caches namespaced and versioned dependencies and is pretty reliably deterministic in the build process.
Verses node which makes horribly assumptions that allow dependencies of libraries to be resolved and dumped into the same library namespace as your application. It also allows users to use the shrinkwrap feature, which imposes the dependency's settings on me (say, for example, I want to use my own npmjs proxy, this is bypassed...)
One suspects it's you who hasn't distributed or installed many modules on either python or node. So many of the problems that python has, simply don't exist for node, because it finds modules in a bottom-up hierarchical fashion. That allows a single app or module to use modules that in turn use different versions of other modules, and not to worry about what other modules are doing, or how other modules are installed, or how node is installed, or what version of node is installed. This prevents the traditional "dependency hell" that has plagued devs for decades. Thanks to tools like browserify and webpack, the browser may also benefit from this organization.
On top of all that, npm itself just does so many things right. It's quite happy to install from npm repos, from dvcs repos, from regular directories, or from anything that looks like a directory. It just needs to find a single file called "package.json". It requires no build step to prepare a module for upload to an npm repo, but it easily allows for one if that's necessary. package.json itself is basically declarative, but provides scripting hooks for imperative actions if necessary. At every opportunity, npm allows devs to do what they need to do, the easy way.
In a sense, node and npm are victims of their own quality. The types of "issues" (e.g. too many deps, too many layers of deps, too many versions of a particular dep, deps that are too trivial, etc.) about which anal code puritans complain with respect to node simply couldn't arise on other platforms, because dependency hell would cause the tower of module dependencies to collapse first. node happily chugs along, blithely ignoring the "problems".
Personally, I used to be able to build python packages for distribution, but since I've been spoiled by node and npm for several years I've found I simply can't do that for python anymore. It is so much harder.
npm has its own special problems. disclaimer: what I'm talking about in this post is at least six months old, which in node/npm/js world is ancient history.
> it finds modules in a bottom-up hierarchical fashion. That allows a single app or module to use modules that in turn use different versions of other modules, and not to worry about what other modules are doing
To my understanding, if your app transitively depends on package foo-1.2 in thirty different places [0], there will be thirty copies of foo-1.2 on disk under node_modules/ . Each package reads its very own copy of foo-1.2 when it require()s foo.
On a large app, that adds up to a lot of inodes ("why does it say my filesystem is full? there's only 10G of stuff on my 80G partition!" because it's used up all its inodes, not its bytes.) and a _lot_ of unnecessary I/O. The second through thirtieth copies of foo-1.2 don't come from the kernel's block cache "for free", they come from spinning rust (or if you're lucky, the dwindling number of IOps your SSD can choke out. Do you pay money for provisioned IOps?).
[0] and thirty is a lowball number for some projects, especially given the community's preference to require "leftpad" or whatever instead of writing a couple lines in their own projects
...what I'm talking about in this post is at least six months old...
Haha npm@3 was out June 2015. b^)
I agree that it would have been better, on balance, for previous versions to have created hard links to already-installed modules. Actually that wouldn't be a bad option to have even now, since debugging is often easier when one has a deep directory structure to explore rather than hundreds of random names in the top-level node_modules directory. That is, if I know the problem is in module foo, I can pushd to node_modules/foo, find the problematic submodule again, and repeat until I get all the way to the bottom. [EDIT: it occurs to me that having all these hard links would make e.g. dependency version updates easier, since un-updated dependencies wouldn't have to be recopied, unix-stow-style.]
To me, the more amusing file descriptor problem is caused by the module "chokidar", which when used in naive fashion tries to set up watches on all 360 files and directories created by itself and its own 55 dependencies. At that point it's real easy to run out of file watches altogether. Some of the utilities that call chokidar do so while ignoring node_modules, but many do not.
Your concern is somewhat ancient history in the node/npm world. NPM >= 3 flattens the dependency hierarchy as much as it thinks possible. (3.0.0 was released to beta in Jun 2015; NPM >= 3 has been bundled with Node LTS since "Boron" in Oct 2016)
Subsequent versions of NPM continue to improve upon this flattening effort.
This is actually one of the big problems, I think: Python packaging involves knowing a number of different things and reading various resources to get the full picture.
Recently, I built a small CLI tool in Python, and learned all of the bits needed to build, test and package my application "the right way". I knew Python syntax before, but it was a lot of effort to set this up. The difference in the experience between Python and Rust or .NET Core is actually shocking, and most it isn't down to anything that Python couldn't do, just the current state of the tooling.
Python best practice: figure out the correct directory structure by reading docs and looking at GitHub repositories, learn how to write setup.py & setup.cfg & requirements.txt & MANIFEST.in files, setup py.test and tox (because Python 2 still lives), write your README in RST format (as used by nothing else ever), and for bonus points: write your own Makefile. Get depressed when you realize that target platforms either don't have Python or have the wrong version.
Rust: type "cargo new", README and doc comments in Markdown, type "cargo test" and "cargo build".
I'm being deliberately snarky, but you get the point: there has been a slow accretion of complexity over a very long time, and most of it is not the language itself.
The Python best practice: you do not need "correct directory structure" to start programming in Python--there is no boilerplate structure to create (and that would require a tool): REPL, Jupyter notebook in a browser, a single file (module) is enough in a simple case.
There is no single "correct directory structure" for more complex cases either e.g., a Django web app and Ubuntu's Appindicator would probably have different structures and it is likely that there are tools that create corresponding project skeletons such as "django-admin startproject mysite".
Usually, the packaging/deploying are not nice & neat and differ depending on your project, chosen platform--either you follow an example or you have to study a never ending list of incomplete/contradicting standards/semi-documented rules & conventions--it is unrelated to Python (even if tools such as Ansible, Salt are implemented in Python). Naturally, there are things that can be improved.
btw, reStructuredText is a great text format for documenting Python projects (it scales from a simple README.rst to a whole book). You can always convert README.md written in your favorite Markdown flavour using pandoc or a similar tool. Markdown is great for a short human-generated plaintext content common on Github, Stack Overflow.
> there is no boilerplate structure to create (and that would require a tool): REPL, Jupyter notebook in a browser, a single file (module) is enough in a simple case.
Rhetorical question: is Python a scripting language or an applications language?
If it's a scripting language, then yes, a single file is all that you need. Otherwise, you need a directory structure with various support files to be able to create and build your CLI tool or library, and you need to follow conventions if you are going to publish your work as Open Source. Which in turn requires you to know which tools to use, and what they require.
Go raised the bar here, I think. Rust and .NET Core both follow the Go idea of having one top level tool that gives you what you need to test and package your library or CLI application. That means that new users can get success with at least two basic use cases very rapidly and with little effort. Which in turn means that the previous status quo is no longer so tolerable.
> btw, reStructuredText is a great text format for documenting Python projects (it scales from a simple README.rst to a whole book).
Sure, the big problem with RST is that it lost the fight. Every developer who uses GitHub has to know a little Markdown, not RST or something else. For larger documents, AsciiDoc is probably the thing. I used to be a Textile loyalist, myself, but stopped fighting against the tide and switched to Markdown some years ago.
Python is a general -purpose programming language. No size fits all.
You've ignored "in a simple case" and ignored the next paragraph that discussed "more complex cases." If I have a task: publish Ubuntu PPA with a package that does X, or deploy a web app on a PaaS, or add a component that massages some data for a numerical simulation on a cluster or something else. I do not see what Go, Rust, .NET do here that Python can't.
Obviously, there could be specific use-cases where some language/framework has advantages over other languages/frameworks and (as I said) there may be use-cases that can be improved that why they should be named explicitly -- that is why I've asked in the first place -- so far I don't see "shoking difference" in the general case (I don't doubt that you had experienced the shoking difference for your specific (unnamed) task).
My original comment was specifically about building a CLI tool, which I did as a learning exercise to practice writing a multi-file Python project that could act as a library as well as a CLI, and be packaged as a wheel for PyPI.
Please bear in mind here that I did this because I wanted to learn and follow current best practices, with a project that I specifically picked because it would not require 3rd-party libraries or big frameworks to muddle things.
I don't think that it was dissatisfying because Python is a general-purpose programming language, but because the process currently happens to involve an assortment of different tools, piecing together scattered information, and being willing to suffer a few paper cuts like Python versions and multiple old file formats. That was my experience building a simple Python 3 application, and it didn't leave me eager to do it again.
We can discuss how Go-style tooling and packaging can also help with the cases that you mentioned, but I didn't really want to stray off into an extended analysis. My original points were just that the overall developer experience that new Python users currently get compares poorly to some other languages, and that is totally fixable. I hope that things do improve, and feel that Pipenv is a very welcome move in that direction.
> You says "X is bad", "Y is better" but you don't say how exactly X is bad, in what way Y is better.
I gave you a breakdown in my earlier comment, but I'm thinking that writing more isn't going to make this better:
* Your experience: Python does everything that you need, and quickly, because you are already very proficient in the language and the tools.
* My experience: Shopping for a programming language, knows Python syntax but no attachment beyond that, has tried out a number of languages and has working knowledge of their tooling.
I'm not suggesting that Python is a bad language, just that I think that the tooling and developer experience for new users is not competitive with some other languages. If you are proficient in Python, and haven't tried other languages recently, you won't have experienced the jump: Python is slightly more hard work than Ruby or Node.js, but Rust, Go and .NET Core have a level of slickness and integration that older platforms just don't have.
> What is so bad about "a multi-file Python project that could act as a library as well as a CLI, and be packaged as a wheel for PyPI."
Again, see my previous comment for a list, including some bits that you are omitting. I found that document and read it, but it assumes that you have already created something with docs and tests (laid out in an appropriate directory structure). Since I didn't actually submit the package to PyPI, I didn't have to think about twine, but that's another paper cut. Which is my main point: if someone is not already sold on Python, every small quirk and special case becomes another obstacle, and the Python tooling has collected more than a few.
It shouldn't be that hard to demonstrate "shocking" difference in experience.
Could you provide an explicit list of what do you mean by "a breakdown in my earlier comment"? (obviously, I wouldn't ask for more information in my comments if your previous comments were specific enough) e.g., in language X I can do Y using command Z but Python requires A,B,C instead (implying that Z is "shockingly" better than doing A,B,C).
As I understand, the issue is that you want the structure of a project in a language X to be generated for you by a single command (whatever the project) or that this command must be available by default.
> It shouldn't be that hard to demonstrate "shocking" difference in experience.
Please try installing Rust with the rustup tool, creating a "Hello world" skeleton CLI tool, and compiling a deployable executable for it. It will take you (not joking) perhaps 10 minutes or less if you have never done anything with Rust before. Then look at how you would add docs and tests. The Rust language is not simple, but the tools are really good. Try the same exercise with Go if you prefer.
If you are willing to give this a try and invest the 10 minutes, you'll understand why I didn't use the phrase "shocking" casually: listing off features doesn't really convey how huge the usability gap is.
> Could you provide an explicit list of what do you mean by "a breakdown in my earlier comment"?
It was this:
> "Python best practice: figure out the correct directory structure by reading docs and looking at GitHub repositories, learn how to write setup.py & setup.cfg & requirements.txt & MANIFEST.in files, setup py.test and tox (because Python 2 still lives), write your README in RST format (as used by nothing else ever), and for bonus points: write your own Makefile. Get depressed when you realize that target platforms either don't have Python or have the wrong version."
Okay - maybe I'm missing something, but pip is the only Python package manager I've ever used. And it's basically "pip install xxx", "pip install --upgrade xxx", "pip show xxx", "pip list", "pip uninstall xxx"
I'm curious what I've been missing about pip that makes it problematic - I've never used the other tools you mentioned (setuptools/distutils/ez_install) - so I can't comment on them, but, on the flip side, I've never had to use them, so maybe my requirements are somewhat more simple than yours.
One things is a good dependency management. Right now if you want to upgrade your Python version, or one of your packages, it's a mountain of manual work. There is nothing in the stack helping you with the dependency graph.
Another thing is providing a stand alone build. Something you can just ship without asking the client to run commands in the terminal to make it work. I use nuikta (http://nuitka.net/) for this. It's a fantastic project, but man it's a lot of work for something that works out of the box in Go or Rust.
One last thing is to generate packages for OS (msi/deb/rpm/dmg/snap). Your sysadmin will like you. Pex (https://pypi.python.org/pypi/pex) is the closest, but not very standard.
Other pet peeves of mine:
- you can't easily move virtualenvs;
- creating a setup.py is very hard for a beginner and has numerous traps;
- setup.py are executable files. Meh.
- what's with this setup.cfg containing 2 lines ? And the MANIFEST.IN being a separate files. Why do I have to put conf also in tox.ini ? And one for each of my linters ? I want ONE setup.cfg file with all config for all tools for my project inside and be done with it. TOML can handle rich sections, just stop creating new files.
- one place to have __version__, please. I want it readable in my code AND in my package metadata, without having to use regex or have side effects on imports.
- remove the "can't build wheel message" when it's useless. It scares newcomers.
- README is the long_description. Don't make me read it manually.
- how do I provide vendors in a clean way ?
- install_requires, extras_requires, setup_requires, tests_requires... Make it one require with hooks and tags and be done with it.
- creating a setup.py test config is way harder than it should be and breaks in CI on strange edge cases.
- can we get a PEP on the standard project structure and built it in our tools to be done with it? We all have src/package + setup.py on root anyway.
- pip installs packages in the site-packages dir of the python executable it's installed for. It makes sense, and I think Python deals pretty well with the fact you can have various versions installed on the same machine. But God people are confused by this. Now you can recommend to do "python -m pip", but it's very verbose and it assumes people know what version of Python is behind the "python" executable. On windows it can be any, and they must chose with... yet another command ("py")! pipenv just bypass that by assuming you want a virtualenv, and be able to access it. It's a very good call.
- pip install --user will create commands you can't use unless you edit your PATH. This makes newcomers go mad.
Oh my god, you've described every single one of my issues with Python packaging.
The whole setup.py/setup.cfg situation really is ridiculous. Having to import the __version__, read() the README, no markdown support on pypi, MANIFEST / MANIFEST.in files, tox.ini, what a mess.
This. Particularly the need for a minimum standard project structure.
Pipenv shows its pedigree and looks like a great tool...that also overlaps significantly with conda. What are the use cases that Pipenv addresses better than/in lieu of conda?
It looks like Pipenv does not handle the python install itself or associated non-python libraries. With Conda I can tell it to install Python 3.6 along with FreeTDS (for mssql). Conda lets me do this in one environment.yml file and have it work cross platform. Separate homebrew or apt-get steps are no longer necessary.
That said pipenv still looks awesome. Any improvement to the python packaging world is welcome gift.
You don't need to install (ana|mini)conda just to get a package manager, would be why I would use Pipenv over Conda. Miniconda alone requires somewhere close to 400MB of space and comes with a whole bunch of extra things I don't need just to manage packages and virtualenvs.
The miniconda bootstrap of conda is ~20-30 MB (compressed) depending on platform. It contains only conda and its dependencies, like python and requests. It's how you install conda if you want only conda. The 400 MB number is for the Anaconda Distribution, which is a self contained, single-install, get-all package primarily aimed at scientists and engineers.
pipenv allow you to completly ignore the virtualenv. Like node_packages. It seems a detail, but giving a lot of python and js trainings, I came to realize newcomers needs little help like this.
Then you can let people like Kenneth build a big friendly wrapper on top of it.
The only problem with those is that it must be pure Python. Otherwise you will have problems. Wheel are not bullet proof, and while you can get away with tinkering for dependencies, you can't with your package manager. It should work out of the box.
But creating "cargo for Python" is a very, very hard job. And nobody will remember you for it.
Note: you must run this after you've installed any packages into the environment. If you make an environment relocatable, then install a new package, you must run virtualenv --relocatable again
Nice list! We manage to avoid most problems with distribution by using Docker containers, but it brings its own set of problems and downsides. I would love to have a better solution!
Honest question: have you seen how the Perl world handles this stuff? Now that I've mostly moved over to Python, the Perl experience (overall package mgmt) seems much, much better.
Note, it doesn't feel terrible in Python land, to me at least. But it was almost a joy working with Perl's packaging system.
>One things is a good dependency management. Right now if you want to upgrade your Python version, or one of your packages, it's a mountain of manual work. There is nothing in the stack helping you with the dependency graph.
pip-tools doesn't solve the problem at all. It will update things to the last up to date version, cascading from package to package.
That doesn't guaranty your setup will work.
Dependency management suppose to create a graph of all requirements, lower and upper versions bound for the runtime and the libs, and find the most up to date combination of those.
If a combination can't be found, it should let you know that either you can't upgrade, or suggest alternative upgrade paths.
pip-tools will just happily upgrade your package and let you with something broken, because it's based on pip which does that. They don't check mutually exclusive dependencies versions, deprecation, runtime compatibility and such. And they don't build a graph of their relations.
How can you have an upper bound on compatibility? When a library is released, it knows that it works with version 1.3.2 with its dependency, but how can it ever know it doesn't work with 1.4, unless the developer goes back and re-releases the app?
If the library follows semantic versioning, then you can always declare that you work with everything from the current version to before the next major version.
That's what I usually do (although I pin minor, because you never know). I should also be better about following semver, but it just feels wrong to have your library be at version 5.43.55 :/
I strongly agree but this can work now and is a big improvement over what we currently have. So while I would literally pay to see somebody work on a better package manager (which can generate exe, deb and use a conf file indead of .py), this is a good filler.
debhelper pretty much automates the process of packaging any standard distutils or setuptools package, Red Hat distributions have templates for packaging Python libraries as well (and rpmdev-newspec python-mypackage will automatically generate an appropriate .spec file).
Windows and OS X are always a pain in the ass, but that's more an issue of the platforms lacking in package management than anything else.
I would hardly call it a pain, it takes me all of 3 minutes to write a .spec for most python packages and from there it's basically 'tito release'. Sure, if I wanted to package for debian-based distributions it'd take a little more time, but it's worth it to make a quality package that a distribution itself can decide to pick up (packagers love other people doing the work for them, though they won't refuse doing it themselves) with minimal effort.
And it took me all of 60 minutes to learn, packaging isn't anywhere near as hard as people make it out to be. If you can use a build system of your choosing to build and test your project, you can learn how to write an rpmspec or debian control file in under an hour.
It took me months to learn the best practices to create a clean Python package. And nothing is particularly difficult. Just information to gather and sort out. Given deb/rpm are old systems with a lot of legacy, I seriously doubt your 60 minutes claim, even if it's infinity better than Python status quo. Which I doubt as well since all GNU standard are usually a spaghetti of past requirements mixed with FOSS politics.
Do you have a setup.py? No, then it's not going to be on PyPi anyway so ignore the rest of this since you'll need to figure out how to package this special snowflake.
0. Run `yum install -y rpmdevtools mock`
1. Run `rpmdev-newspec python-mypackage`.
2. Run `python setup.py sdist`
3. Modify the .spec as appropriate to add dependencies the package requires as well as set the version correctly.
4. Run `mock -r fedora-25-x86_64 --buildsprm python-mypackage.spec`
5. Run `mock -r fedora-25-x86_64 --rebuild python-mypackage.f25.src.rpm`
Done, there's your RPM, debian packaging is roughly as simple (I'm less a fan because of how they try to fudge a Makefile into an adequate tool for package building, but hey, most everything can be built automatically by dh so the average makefile is ~3 lines).
The biggest problem with learning this is simply finding the right documentation, Fedora has some pretty great guides for new packagers on the wiki.
I would look at this comment[0] by sametmax for a critique of pip. My main gripe with virtualenv is that it's required at all: other interpreted languages, like node and elixir for example, have figured out how to handle non-global dependencies without a third-party package. Beyond that, it's frustrating to deploy because its non-relocatable (in our build/deploy scripts at my last python job we had to use sed all over the place to fix paths), and I find it semi-annoying to have a bunch of different copies of interpreter and all that goes with it (though this is mostly a minor annoyance -- it doesn't take up that much space and it doesn't matter if it gets out of sync.
Also notable, IMO, is the lack of a tool like rbenv or rustup for python. I can't tell you how many times I have had to try to figure out which python version a given pip executable worked with.
> like node [...] have figured out how to handle non-global dependencies
Node would be the last place I'd look for a good solution in. Not sure if there was some progress recently, but it was hell some time back. Modules were huge, taking thousands of other modules with them, majority of those being duplicates. There was no deduplication, no version wildcards I believe either. It wouldn't even work with some tools because the path would end up being hundreds of characters long.
Since npm 3 (about 18 months ago), the node_modules dir tree is now 'flat' and de-duped (where possible).
There have always been version wildcards as far as I know. Long paths caused by the deeply nested tree were a problem in Windows only, addressed (I believe, I can't find open issues on it) by the flattening in npm 3.
Oh cool, I actually hadn't seen pyenv before. Looks like it does indeed solve my problems (from a glance anyway, though I didn't see anything about pip in the readme).
> Beyond that, it's frustrating to deploy because its non-relocatable (in our build/deploy scripts at my last python job we had to use sed all over the place to fix paths)
pip does cache the wheels so instead of moving the virtualenvs around, just recreate them. This also ensures the virtualenv is up to date. Using tox this is fairly easy to do.
Sure virtualenv is a bit of a hack but it's not that bad.
I'm mostly talking about moving between different machines. I would like to be able to tar up my source code and venv, distribute it to multiple machines, untar it and run it. However that's not possible with virtualenv unless you do a lot of hackery in your build. In particular, creating a virtualenv on each server during deploy is not an option.
> My main gripe with virtualenv is that it's required at all: other interpreted languages, like node and elixir for example, have figured out how to handle non-global dependencies without a third-party package.
venv is in the stdlib since 3.3. (Though I agree with the annoyance at the need.)
All you've said here is "python packaging sucks", with no explanation why, and with no alternative. Not a substantial comment, and I'm disappointed that it's been voted to the top.
I'd ask for your reasoning but it seems sametmax has done a good job of that for you:
Comments that make a single unambiguous point are fine. It's no problem to leave detailed support to replies.
A more controversial statement with the same content probably wouldn't have been voted up, but thread parent is objectively true even though it doesn't contain its own proof.
"This is great, but sometimes I think that python needs a new package manager from scratch"
Ha, that's what I came here to say!
Or better - a new packaging paradigm.
Maybe it's extremely uncool to say ... but I think Java still has the best packaging paradigm of all languages. Jars rule. Of course 'gradle' is kind of a confusing mess so they don't have dependencies worked out very well ...
Nevertheless I do feel that Python's packaging and dependency/versioning woes create a much bigger systematic problem than many realize.
I gave up populating requirements in setup.py. I just use multiple requirements.txt. This article has been debated for years already and there is absolutely no right / wrong.
Hey everyone. I'm Kale, currently the lead developer on the conda project. It's been mentioned a few times in this thread, and I just want to make sure that any questions about it are answered accurately. Feel free to ask me anything about product conda. Thanks!
Often people have a requirements.live.txt, or other packages depending on the environment. Is that handled somehow? Can we use different files or sections? [ED: yes, different sections]
Still wondering to myself if this is worth the fragmentation for most people using requirements.txt ? Perhaps the different sections could have a "-r requirements.txt" in there, like how requirements.dev.txt can have "-r requirements.txt". [ED: the pipfile idea seems to have quite some people behind it, and pip will support it eventually. Seems it will be worth it to standardise these things. requirements.txt is a less jargony name compared to Pipfile though, and has a windows/gui friendly extension.]
Other tools can set up an environment, download stuff, and run the script. Will pipenv --shell somescript.py do what I want? (run the script with the requirements it needs). ((I guess I could just try it.)) [ED: doesn't seem so]
Why Pipfile with Caps? Seems sort of odd for a modern python Thing. It looks like a .ini file? [ED: standard still in development it seems. TOML syntax.]
With a setup.py set up, all you need to do is `pip install -e .` to download all the required packages. Or `pip install somepackage`. Lots of people make the setup.py file read the requirements.txt. Do you have some command for handling this integration? Or is this needed to be done manually? [ED: seems no considering about this/out of scope.]
You can also mark the dependencies in your myscript.py as comments. The use case is for quick one off scripts which may require a bunch of different requirements. The integration with ipython is nice for experimenting too. I think pyenv could be useful for that use case of little one off experiments too.
One enhancement I'd like for it to also run modules or entry points like:
Setuptools entry points like used in console_scripts to work would be nice too:
fades --dependency pyenv -m pyenv.cli:cli
Or the best::
fades -m pygame.examples.aliens
So it can create a virtualenv, see that it needs the package (pygame here), and then install that package with pip, and then run the module or entry point.
Anyway... just some dreaming. Can I haz pony? All this is exciting.
It's a very similar set of tools. I use pip-compile which allows me to put all of my dependencies into a `requirements.in` file, and then "compile" them to a `requirements.txt` file as a lockfile (so that it is compatible with pip as currently exists).
This looks great, though, I'm excited to check it out!
We use pip-tools on all our Python projects and it works great. I believe the requirements.in compiled to requirements.txt approach is much more sane and less error-prone.
I think an app should not expose end users to its dependencies. That leaves the end user with a lot of pain figuring out versions of dependencies and god forbid you need to compile some dep then you need a build environment and its dependencies any of which can fail in this chain leaving a very unpleasant and even hostile end user experience.
Ruby and Node apps are particularly guilty of this pulling in sometimes hundreds of packages some of which need compilation. Compare that to a Go binary which is download and use. These things can get very complicated very fast even for developers or system folks let alone end users who may not be intimately familiar with that specific ecosystem.
Very similar. I think Pipenv improves on Bundler by leveraging Virtualenv and Ruby doesn't have a per project equivalent to Virtualenv that I'm aware of. You can set the path config variable of Bundler to not place the project Gems in a central location which I think is cleaner and try to remember to always do now.
It would be _super_ interesting if the Python and Ruby communities got together to harmonize every last detail of their packaging toolchain. Who is in?
Bundler does proper dependency resolution using Molinillo[0] which is also used by CocoaPods[1]. This is definitely something that other package managers can stand to adopt.
This is exactly what I was trying to get at with my original comment. There needs to be a lot more sharing of package management tools and techniques across language boundaries. Generic platform- and language-agnostic tools and algorithms are a step in the right direction!
> Ruby doesn't have a per project equivalent to Virtualenv that I'm aware of.
The nearest equivalent is to place a file called '.ruby-version' in the top level directory, containing the version number of the Ruby you want to use. Version numbers come from https://github.com/rbenv/ruby-build/tree/master/share/ruby-b.... rbenv, chruby and rvm all support .ruby-version.
One difference from virtualenv is that the Ruby version managers share single installations of each version of Ruby. My understanding from occasional use of Virtualenv is that it copies the python installation into a new per-project subdirectory, which seems a bit wasteful to me.
> You can set the path config variable of Bundler to not place the project Gems in a central location which I think is cleaner and try to remember to always do now.
Yes, this is what I do. It gives me a shared 'clean' Ruby installation of the right version, plus a project-specific copy of all the gems the project depends on. To me this provides the best trade off between project isolation and not duplicating the whole world. You can set bundler up so this is done automatically by creating '~/.bundle/config' containing
Great to here you also do what I do with the path config variable. I was just reading the documentation and Bundler has an opinion about deployment as well. I must look into that. I will also start making sure I am using/setting .ruby-version .
Seems to be a python specific nix-shell like tool?
With nix[OS] you just run `nix-shell -p python[2,3] python[2,3]Pacakges.numpy ...` to get an environment with the required packages.
Of course this requires that the python library is packaged in nix, but in my experience the coverage is quite good, and it's not very hard to write packages once you get the hang of it.
It also possible (but currently a bit clumsy in some ways) to set up named and persistent environments.
I will definitely be trying this out. Python version and package management is a dumpster fire that wastes gobs of my time on the regular. I'll try anything that promises to end the pain.
Finally someones does it!
I was using:
pip -t .pip
in my code, avoiding virtual-env completely, but that was not enough and incomplete.
As this is not cross platform and it would be nice to switch between Linux/Windows while coding to maintain platform compatibility, can the virtualenv envs be created with a os platform & subsystem prefix ?
for example, having multiple envs at once:
I always wonder if this could be done once and for all languages, instead of Ruby making bundler, Haskell Cabal sandboxes or stack, Perl Brew, etc. Is this where Nix is going?
I don't know, different languages and environments need different things. I'm not sure that "package management" is as generic as it seems at first glance.
You make it one tool, and sysadmins will instantaneously lock it down. These package managers, in most cases, are developer tools built to get around system-wide locks on libraries; the more you centralize them, the more likely it is they will get locked down, and then someone will build tools to get around that, and so on and so forth.
You can easily get a nice isolated python environment with some packages in nix without using pip, pyenv, etc. `nix-shell -p python pythonPackages.numpy ...`
So far I think it works quite well for most languages as long the needed packages are in nixpkgs.
Some of the tooling could be better, but the underlying model seems sound.
I'm not really convinced language-specific package managers are needed. Nix isn't perfect yet, but it has come a long way.
normally I use virtualenvwrapper and that makes a virtualenv directory for all virtualenvs you create with it. before that, I always create my projects' venvs inside my project hierarchy.
I had a dilemma about it. But after all, you can not move your venv directory unless you use `--relocatable` option. So, anyone have a strong argument about creating venvs inside your project directory?
I've always found the whole virtualenv stuff so superfluous. Do we really need all the machinery with shell scripts and static paths?
We just use a directory where we keep our dependencies. It's a matter of:
mkdir libs
pip install -t libs <package>
# then to run
PYTHONPATH=libs python app.py
From what I can tell, this accomplishes everything a venv does (except bringing the Python interpreter itself along) without requiring any extra tools or conventions to learn.
- you still need to change PYTHON_PATH to recognize libs from `libs`
- packages have bin scripts sometimes that most likely, will be needed by the project
A more proper command:
- pip install -t ./libs --install-option="--install-scripts=./bin"
But still, does not solve the PYTHON PATH issue, it won't be solvable because all python cli tools and all scripts in ./bin must be aware of PYTHON_PATH including ./libs
This is what venv does and pip alone cannot easily solve, replicating an entire python environment that is aware of project local pip packages
My commands do set PYTHON PATH when running the app.
As for the cli tools, fair enough, but the packages I use with such tools/scripts are full applications, not dependencies to be included in another project. Which kinds of packages do you see as both dependencies and containing CLI tools?
I just keep venvs in the project folder and add them to .gitignore. --relocatable never worked well for me, I just keep my requirements.txt up to date so it's trivial to blow away a venv and recreate it if necessary (python3 -m venv env && source env/bin/activate && pip install -r requirements.txt).
I find tools like virtualenvwrapper and this one from Kenneth tend to solve issues I don't really have. A little bit of repetitive typing here and there is ok to burn knowledge into my mind; and less leaky abstractions I have to deal with, the better.
- beginers don't have to understand the whole virtualenv shenanigans. I use pew myself to replace virtualenvwrapper but I will switch to pipenv just to ease the pain of team member joining in.
- it enforces good dependency management practice with the toml file and lock file. This is an issue in almost all project I worked on, including ones from Python experts. We all use only requirements.txt file out of convenience, and never lock.
- it's one tool to do all the packaging stuff. No need for pip and virtualenv and a wrapper. You got one command.
I was really not a fan of the last "made in Reitz" project Maya. But this, I really can get along.
The whole things make it way easier to get started for a beginner. Now more activate. No more wondering about virtualenv. Automatic lock files are great since no project I know of use them since they are not well understood.
It's like node_packages (easy and obvious), but cleaner (no implicit magic).
LinkedIn has a similar open source project that is much more mature. It builds on Gradle features to manage complex dependencies and build Python artifacts. If you include this LinkedIn Gradle Plugin [1] you can automatically run tests in a virtual env and source a file to enter the project's virtual env.
PyGradle [2]: "The PyGradle build system is a set of Gradle plugins that can be used to build Python artifacts"
> --three / --two Use Python 3/2 when creating virtualenv.
I use Python 2.7, 3.4, and 3.5 on various projects. Is there a way to choose between 3.4 and 3.5 using Pipenv? I'm using something like this with virtualenv:
Is that true? His command would evaluate the path at the time which is run, but yours seems to evaluate the path at the time python3.5 is run. Subtle, but very different results.
virtualenv resolves the interpreter when you run virtualenv, and if you specify a different interpreter with `-p` it will resolve the path to the interpreter and then run virtualenv again using it.
First, you need to take care of the virtualenv manually. This tool avoid that. You can completly ignore the virtualenv. You don't even have to populate requirements or learn about pip freeze.
Secondly, this tool allow you to freeze your requirement list at specific versions. So in your req file, you have the name of the packages you depend on. Bon on the req lock, you get all the pulled dependencies recursively with the version you are using right now. The first one let you dev more easily. The second one deploy more easily.
I haven't really used requirements.txt because I found that I could install 'extra' and 'test' specific content based on args to setup() in my setup.py. It seems more like the Right Thing than requirements.txt, from what I can tell.
At first glance, this doesn't seem to offer anything beyond what I already see from setup(). What am I missing?
It's unfortunate that CPython gave us distutils and took a very long time to converge on a built-in successor (setuptools?) that gives the right composability.
> Otherwise, whatever $ which python will be the default.
This is a bit strange because the python binary is always supposed to be Python 2. The Python 3 binary is supposed to be named python3. Some distributons don't follow this, but they're the weird non-conformant ones; it's not a behaviour that should really be relied on.
This is not correct. It's a symlink to python2 on systems that rely on calls to python to be python2. On modern systems, python is usually a symlink to python3. This is the case on Arch Linux and I believe other recent distro releases.
Arch is the oddball. I'm not aware of anybody else who's made the switch.
Given that they're not mutually compatible except in rare cases, it's a very silly thing to do. You can upgrade GCC with the same name because you know it will handle most of the same input. If you do that with Python you're breaking tons of existing scripts for very close to zero benefit. Why would you do that?
It's not silly. One of Arch's primary draws is being the first to get updates and integrate new technologies. Other distros will follow soon enough if there really aren't any others.
It's not as if python2 suddenly doesn't exist, it's not terribly life-changing to add a 2 in the places where the scripts are still on the last version.
I'm glad to be on a system where the default Python version isn't one that will be officially unsupported in three years.
Edit: It's worth pointing out that this switch was made 6 years ago and the world is still spinning.
They can adopt Python 3 and drop Python 2 without adding this link which encourages a convention of /usr/bin/env python for Python 3 on their platform, creating unnecessary incompatibility with other platforms.
As you said, it's old news now, and not the end of the world. But I'm hoping others don't follow.
Non-conformant? PEP-394 says that scripts should only use python in the shebang if it's compatible with both py2 and py3, and be updated to work with both, or to use python2 otherwise.
Edit: I don't mean to disparage projects like this and pipfile. Both are great efforts to bring the packaging interface in line with what's available in other languages, and might be the only way up and out of the current state of affairs.