I just enter the directory and the environment is created/'activated'.
In all the years I never had any need to use the real activate method. Just running Python and scripts directory from the bin/ in the virtualenv is enough to run 'inside the env'.
When I get the choice I use Nix and ignore as much of the "python" packaging infrastructure as possible - it's all a vague and unreliable toy compared to a real package manager. It all particularly falls apart when it comes to non-python dependencies.
`pip-tools` has served me well, but an annoying gripe that pipenv/Poetry resolves is that you can't install git links in non-editable mode, i.e.
-e git+https://github.com/...
Works, but
git+https://github.com/...
Doesn't.
Also can you pin your dependencies to a specific hash? Last I checked you cannot. A major advantage of pipenv/Poetry is that the lockfile protects you against compromise of the package manager; if someone swaps in a new binary for yourdep==1.2.3, the lockfile will fail to validate the hash and you'll get an error.
Using --user here, no per-project virtualenv, used to have a virtualenv in ~/.env, where i keep everything up to date and have many -e installs from ~/src. For production deployments: --user in containers.
This means all my projects have to work with the same versions of dependencies.
This means all my dependencies have to work with the same versions of dependencies.
All my dependencies have to be up to date: all versions must be latest, or next to-be-released.
When don't support new versions of other dependencies: I deploy a fork and open a PR.
At such, I'm not spending time trying to make obsolete versions work, rather, I'm spending time making new versions work together.
My perception is that this strategy offers better ROI than all others which I have tried, not only for me, but for the whole ecosystem. That's also how I made my Python 2 to 3 transition, and have been using 100% Python 3 for a while (once you've ported one big software, porting others and their smaller dependencies is easy).
I'm extremely happy about this strategy and my life has just been better since I started working this way, and highly recommend it.
For my own (countless) packages: I use setupmeta which simplifies auto-updates (I think it succeeds what pbr tried to achieve).
Not to mess with system packages, but I sometimes do so in containers. I just started to switch to --user in containers so that my users would not have to rebuild python dependencies from scratch when they use it for development: with source bind mounted into /app, which is declared as the home of the app user in my container, as such, .local remains between builds on the developer checkout (like node_modules)
There are a few highly dismissive comments here. While the python community has (clearly) been getting by with requirements.txt/setup.cfg/setup.py, the project/package management story is far less usable than more recently developed systems like npm and pub.
With poetry+pyproject.toml, I have one file that is flat, simple, and readable. @sdispater has done incredible work on this project, and I hope they get more resources for development.
Your comment does not explain why you think that the package management story is better in npm, it's merely like saying "node-gyp is a pile of debt" and not going into details, but I can surely tell I have no idea why node_modules ends up so fat and why my npm install eats a lot of bandwidth, time and disk space ... is npm trying to workaround incompatibilities by installing different versions of a same package in the same node_modules ? I hope not, because that would be just perfect to accumulate debt in the whole ecosystem. Not to mention that I was just handed a frontend source code where building the source requires node-gyp which requires g++ and python2 :)
Quick question since you seem to know npm very well, is there a better solution than this to automate npm package publishing nowadays ?
sed -i "s/GIT_TAG/${CI_COMMIT_REF_NAME/v/}/" package.json
It took me a while to learn to love pipenv. It also bothers me how many developers "blame pipenv" for any problem they can't easily explain.
Anyway, has Poetry caught up to the dependency resolution provided by the pipenv lock command yet? Last time I tried it (~6 months ago), it couldn't produce working environments for my requirements.
That's funny, Poetry's README [1] claims exactly the opposite (that it resolves dependencies better than pipenv). And looking at the git history, that section of the README is 2 years old, so that claim was made when you experimented.
Was the problem with dependency resolution you had similar to what Poetry describes in it README? Or are there other problems the README fails to mention?
The description they make got me interested, but up to now all alternatives I tried to build moderately complex python projects failed to deliver on their promises, so definitely interested in what you have to say.
I do really like the workflow pipenv provides. But last time I checked, the mayor downside of pipenv is it doesn't and won't support multiple Python versions [0]. For libraries, this can be a dealbreaker. For applications, especially ones you're deploying to a controlled environment, this isn't an issue.
Been meaning to look into poetry. Not a huge fan of TOML, but hopefully all the tooling supports pyproject.toml now (I know black does, but not sure about flake8, isort, pylint, pytest, coverage). I know there's still some hacks required for tox + poetry though.
It seems like pipenv and poetry fill different niches: pipenv is intended for application deployment, so you're likely to only have one version of python, while poetry is intended for building libraries.
I found the tox support a bit hacky, but not bad, and got 3.5 to 3.8 testing just fine.
Poetry does make the publishing process pretty seamless for the simple case. If you're starting a new library, poetry is great.
I think poetry needs a "fuzz option" to test random valid solutions to the version constraints, to check if the older versions are really valid.
I use pyenv to switch between python versions (or just the `py` launcher when on windows), and pipenv will follow suit. It does require regeneration of the lockfile for each python version/environment in the scenario you describe, but you can also keep them organized in a folder and swap them out for each environment. You're right that it doesn't come included in the tool though.
What don't you like about TOML? It seems to be the best language for simple configuration files. I think YAML is often overkill for basic configuration, and JSON isn't ideal. pipenv's Pipfile is also in TOML.
What config language would you prefer to use instead?
To be honest, I don't know. All I can say is YAML parsers exist and it works well enough for many applications, although I do understand the issues with it. Sure, it isn't perfect, but no need to re-invent the wheel for marginal improvements.
I roughly agree with the arguments laid out in PEP 518, even though the array of tables thing is not at all "obvious" to me. And I actually do think an "official"/semi-official YAML subset would be great for loads of use-cases - yaml.safe_load already is that in practice.
Not even with the `pipenv lock` command? It seems to work out solutions in multiple rounds, including backtracking, but I might be wrong on that observation...
My experience was the opposite. pipenv always told me something couldn't be installed because of version mismatches and poetry has never given me any trouble.
When you use `pipenv lock --verbose` you trigger holistic dependency resolution with reporting on the process and I have yet to encounter _any_ issues with that.
It is important to apply pipenv as intended; I only use `pipenv lock` and `pipenv sync`, and edit Pipfile manually.
The problem with other commands such as `pipenv install` is that they don't apply the holistic dependency resolution as `pipenv lock` does.
How are you supposed to know you're not meant to use `pipenv install <package>` and should instead manually edit Pipfile? The documentation starts at `pipenv install <package>` and never gets to "but that's not applying pipenv as intended so don't do it".
Does pipenv create cross-platform lock files? That has been my primary requirement when looking at solutions, more than the usability issue of Python using a bunch of tools and files for the whole development process without a coherent story.
iirc Poetry does have cross-platform lock files and a coherent story. I'm just blocked on some PRs due to the original author getting busy and not yet having delegated to lieutenants.
But it won't lock the linux version. Similarly, if you lock on Linux, it throws out macfsevents. And `doit` is not doing anything strange[1] to report these extra dependencies.
In the past Pipenv has failed to resolve dependencies for my project before, where Poetry (run last week) succeeded. Not a fair comparison as I've not tried Pipenv recently, but that at least lets you know that Poetry is now better than where pipenv was a few months ago.
The big win seems to be the lock file - like Cargo in Rust, or yarn in the JS world. It's really, really hard to lock down dependencies reliably in Python, especially when you are talking dependencies of the primary packages you are installing (and their dependencies, ...).
One solution at the moment is to run 'pip freeze' and put that in as your requirements file, but that very much feels like an 'and now I have a different problem' solution.
For dependency management needs, 99% of what you need can be done easily with vanilla virtualenv, vanilla pip, vanilla setup.py, and a Makefile. A small shell wrapper that sources bin/activate and runs your Python app is all you need to allow anyone to run your code without additional steps.
The biggest problem with Python packages isn't a dependency manager, it's that Python developers don't reuse and extend existing projects. They just keep churning out new similar projects with different features. All of the most commonly used Python packages are basically one-offs that use a bunch of other one-offs. This creates a cycle of reinventing the wheel and friction in ramping up on existing technology, due to all the extra integration work and split development, and later lift-and-shift into completely different dependencies that implement the same thing. PyPI is awash with duplicate abandoned one-offs to the point that just trying to just find a useful module to do what you want can take a long time.
I can't find a good comparison to Conda - is the main distinction simply that Poetry uses the official repos?
For people working outside of scientific Python: conda is a package and env manager maintained by a private company that's become the go-to because it's really good at handling binary dependencies.
I use conda as a python manager and poetry as an environment manager. For every project I just create a new conda environment with the bare minimum python 3.x version I need. Then I use poetry to handle the dependencies and manage the environment for me. For stuff that are strictly scientific or numerical in nature and I need things from conda (ie MKL accelerated numpy) I skip the part with poetry and just use conda to install my packages to avoid any potential issues.
As far as my understanding goes, conda focuses on providing virtual environments for interactive use, but to build and distribute a package on conda forge, you still need to rely on setuptools/distutils.
My impression here is that Poetry aims more at _replacing_ the pip + setuptools toolchain. Users of your package could still install it using conda, if relevant. It seems a bit limited in what it can do at the build step, unfortunately, so it is not a replacement yet.
Coming from the Java world, I pass my days dreaming of a "maven for python", and this project definitely goes into that direction. I will definitely keep it on my radar.
Well conda packages don't need to be python based, so technically yes you can build and package up whatever your heart desires.
conda build recipes are essentially just shell scripts that conda runs in a sandbox, taking care of library paths and so on. You could, if you really wanted to, develop a poetry based Python package and then have your conda build script use/run poetry to build and package the python package, and then add whatever lines are necessary to the build recipe to make that the package that gets installed by conda.
That basically confirms the feeling I was trying to express, in that conda and poetry seem to solve different problems: conda focuses on easing the distribution to the end user, whereas poetry tries to simplify managing the build on the developer side.
Worked a little with and it looks really nice and promising, but my biggest concern is number of issues on github and that vast majority of commits is done by one person.
From what I've gathered, the project grew faster than the original dev had time to build up other people to also monitor PRs and has since gotten very busy, slowing down the review process. I know I have several straightforward outstanding PRs. I hope this is able to be resolved so we can have a sustainable community.
That is also my worry. I've opened issues that were show-stoppers for many people and we couldn't even get the author to give us an update on when the fix would be released.
I wish he'd run the project a bit better. Other than that, it's stellar software.
I like how transparent poetry is about what's happening when you run it, and how well presented that information is. I've come to loathe pipenv's progress bar. Running it in verbose mode isn't much better.
I can't be too mad at pipenv, but all in all poetry is a better experience.
Python is the worst thing to happen to SWE (specifically SWE, not ML, Education, etc).
Python makes huge sacrifices in readability and efficiency for a hypothetical win in writeability. It's also fundamentally at odds with the multicore world we're living in an will continue to live in, an issue that many people have tried to fix and failed at.
I can't count the number of times I've had to spend my entire night reading through an older Python service without type annotations, desperately trying to understand the types of inputs and outputs. It's just a mess, and the bias of programmers to believe that they are writing readable code (when in reality it's only readable to them) exacerbates the use of names that provide little context to solve the problem.
Python is awful, and it's an uphill battle to fix it. Asyncio can solve some performance issues, but negates the benefit of Python's simplicity by destroying the sequential mental model that Programmers can easily grok. It also requires a complete rewrite of libraries and a split between non-asyncio land and asyncio land.
Type annotations can make Python more readable and thus more maintainable, but gradual typing still leaves lots of unanswered questions when you're interfacing with untyped code.
Python is really not good unless you are using it to prototype or build something on your own. It's led to a world of slow, buggy software that is impossible to rewrite. It's downsides are easy to measure, but its benefits are difficult to quantify.
How do you start a new Python project in 2019? I mean, there's so much stuff. Poetry, pip, virtualenv... oh, my. With PHP, I just composer init and that's it.
Neat, definitely something I would want to give a try.
I see though that it only supports pure python for building packages, does that mean that it doesn't build if you are dependent on compiled libraries?
Is there also a plan to add some of the functionality of bundling tools such as web-pack into this build phase? like automated css optimization, image compression... Could be handy for some django/flask projects.
They can't support reading the poetry data because it's not generic. Nor are most practical implementations of tools using pyproject.toml because they all use custom tool-namespaced fields in pyproject.toml: it's far from a standardized format yet, despite the push for using it from the people that made it.
I still wish poetry would support setup.cfg in parallel. It has been working for 2 years, is compatible with pip, setuptools and the whole legacy stack, and most fields are standards and documented.
For pip, I'm not sure what's involved in getting everything to work generically. This is probably not simple. It has the consequence that when we don't have control over the install command, we end up needing hacks like using `extras` for ReadTheDocs (e.g. https://github.com/readthedocs/readthedocs.org/issues/4912#i...)
I was excited by this project, and pipenv, but unfortunately I haven't had consistent results cross platform with either. I have ended up sticking with venv because of this.
Adding dependencies takes a (little) bit of time, because of the lockfile. But deployment (ie installing dependencies) is as fast as any dependency manager (pip, pipenv, ...).
Venv doesn't do dependency management, it just provides an isolated python environment. By default, poetry uses venv for environment isolation, but also does dependency management.
what about pip freeze to requirements.txt for manually but solid dependency management under venv? pip takes care of dependencies be it venv or poetry, requirements.txt will record what's need so you make your project reproducible and portable. what else does poetry bring to the table? If there is a strong selling point I would like to try poetry again.
yarn came out great but these days I switched back to npm for the sake of simplicity of my workflow, especially npm absorbed yarn's good features.
same thing happens to parcel/webpack, the neat new tool beats the old players, but then the old dominant one catches up by taking in good stuff from its smaller competitors, quickly.
- vanilla virtualenv: I don't even bother with the wrapper most of the time.
- vanilla setup.py/setup.cfg: Its just really not that bad. Forget about project.toml or whatever the next big thing is.
- pip-tools: ditching pipenv and using this for pinning app requirements, has made my life so much simpler.