I did not expect this to be on hackernews this early. I want to point out that I have not made a release yet and it's not yet feature complete. Mainly I want to ask for feedback on the general design.
Armin, my feedback is in the form of my own version of this library.[1] I have been working on this on and off for some time and like you have not made a release. That said, I have been using it quite a bit both in my research and at work and I think it helps.
The main idea behind it is make it easy to write a "getopt" style program with arbitrary command nesting. I have found that although argparse and sisters are nice libraries they don't allow me to do many of things I like to do in my interfaces. For instance sometimes like options such as
foo -x a -x y -x q ...
where I would process that into like so:
extras = list()
for opt, arg in opts:
if opt in ('-h', '--help',):
util.usage()
...
elif opt in ('-x', '--extra'):
extras.append(validate_or_die(arg))
I also believe that you should have "fast fail" validators. So I have several in `optutils` which are like:
util.assert_dir_exists(path)
which if a directory doesn't exist on the path it creates it. If there is already a file there and it isn't a directory it dies with an error. When it dies, I try and have unique exit codes for various errors (for testability) and provide usage information immediately. This style is nice because it provides immediate feedback to the user with no fuss. I think a lot of "option parser frameworks" miss the point in having lots of things for parsing ints and things. Most of the time I deal with files, directories, and "string" parameters which these libraries don't help with.
In general, the standard libraries make it way to hard to write really nice command line tools. I like some things about your library, but I think that you need to increase the flexibitly for how options are processsed to you can do whatever you want with them. I also think that option parsing and configuration should be integrated. I am working to support that but I am not there yet. (see optutils/conf.py for my current ideas)
> I have found that although argparse and sisters are nice libraries they don't allow me to do many of things I like to do in my interfaces. For instance sometimes like options such as
?
This is trivial to do with argparse (or optparse for that matter):
import argparse
def a_prefixed(string):
if not string.startswith('a'):
raise argparse.ArgumentTypeError("%r does not start with 'a'" % string)
return string
parser = argparse.ArgumentParser()
parser.add_argument('-x', '--extra', action='append', type=a_prefixed)
print parser.parse_args()
> I also believe that you should have "fast fail" validators.
Isn't that what the callback parameter is for, especially with is_eager=True? Or ParamType if you need either something more reusable or something more extensive.
> Most of the time I deal with files, directories, and "string" parameters which these libraries don't help with.
@masklinn good point. I haven't used argparse (mostly out of compatibility requirements with 2.6). So I may have mis-characterized the state of the art. It does do many things well. One note, is that the `prefix parsing` functionality could cause weird behavior when doing partial parsing (which I do a lot of).
> Isn't that what the callback parameter is for, especially with is_eager=True? Or ParamType if you need either something more reusable or something more extensive.
yes. I think almost everything should work like this. I think the getopt style makes this a bit easier to understand.
Not at all the same. I never said my programs were going to open the files themselves. I often have to write automation scripts around other things. In these cases I need to make sure files and directories are sane but I don't open them. I just canonicalize them and pass them on.
The big thing is the lack of integration with configuration files which is something I am still working on myself.
Checkout the github issues there for some features I want to add, like bash completion support for example. I think we can steal ideas from each other!
One nice thing about dart's annotations vs. python's decorators is they can be placed on a function's parameters as well, so it allows the option/flag/argument declarations to be a bit more DRY.
I think the creation of a group by decorating an empty function seems awkward. Why not just have the group be an ordinary class, i.e. something like "cli = click.group()"?
Gorgeous. This guy has a full time job, a wife, and wrote Flask and Werkzeug, ItsDangerous, Sphinx, Markupsafe, Jinja2 (and Jinja) and that's just some of the well known stuff. Handsome and young too. Bastard. He probably saves the world in a dinner jacket in his spare time.
Armin Ronacher is absolutely amazing at python. If you want to learn some things dive into his source code and dig away. He really thinks through code architecture.
You're right - looking on the http://docopt.org/ page it's not immediately obvious how it works.
The general idea is that you don't specify any code, you just give the help message as text. Docopt parses that to figure out all the options etc and convert those into the parameters coming in to your system.
It's a really clever idea. You specify the human interface, docopt converts that into the code version (so long as you adhere to a few common conventions). I haven't seen a cleaner system anywhere.
If you need to parse something as simple as `foo --option spam --option eggs` remembering both spam and eggs, docopt breaks down. You are also unable to compose an interface using docopt, if you want to allow plugins in a plugin system to add their own options for example.
docopt is much simpler because it's only capable of creating very trivial interfaces. That's perfectly fine if that's all what you want to do but it also makes docopt unusable, if that's not the case.
I haven't pushed it too far so I'm sure there are other cases where you'd need to create a different command line api to have it work (that may or may not be an issue depending on your use case).
Regarding the other criticism, seems valid. In the docopt api can you have it do the parsing for you so you can edit the rules afterwards before you run them?
I've used docopt for some tools, but argtools (and click it seems) is much more convenient, imo.
It's very easy to remember the 1-2 decorators and probably most importantly, the command arguments and everything else are where the command is defined, not somewhere else.
When using docopt i spent much more time defining the docstring and figuring out the right way to write it, especially more complex scenarios.
I love docopt. My big draw is that I write the documentation only once - in the doc string where it should be. Click looks great, but there is no documentation in the code itself. You need to run the code to print the documentation.
It seems to me that there are a few projects similar this. Here is another, https://pypi.python.org/pypi/Cogs/ (conceptualized as a "Makefile" replacement).
I'm wondering if there could be a breakout at the next PyCon to see if we could discuss approaches and come up with a unified way to do convert Python libraries into command line scripts?
"There are many alternatives to click and you can have a look at them if you enjoy them better. The obvious ones are optparse and argparse from the standard library.
click is actually implemented as a wrapper around optparse and does not implement any parsing itself. The reason it’s not based on argparse is that argparse‘s design does not allow proper nesting of commands by design and has some deficiencies when it comes to POSIX compliant argument handling."
What I love about Pocoo is they always have stellar documentation and give clear rationale - from the beginning.
Given that argparse has known problems and that not all optparse code can be ported to argparse I doubt optparse will ever go away. It's also a very small module and if optparse will ever disappear only a subset of optparse is needed to make click work.
Kudos again to the pocoo team for creating such a simple and useful library. It sure beats optparse :)
On a side note, however, does anyone know why the team prefers to wrap functions in decorators? Flask also uses them, but what's the design decision behind them?
Decorators are so simple and minimalistic and elegant and easy to understand. They can surface an arbitrary amount of extra functionality with a single statement. For example, ensure a function is only accessible to authenticated users by simply saying something like @requires_authenticated_user. Another example would be to create a logging decorator to wrap a function or a class to create a log of every time the function is accessed @log_function_access. Or to wrap a function to ensure that every access to the properties of that function are pushed through some sort of validation or transformation @validate_set or @transform_get. Decorators allow encapsulation of functions within other functional concepts with almost no code, that's the design decision for using them.
Yep, it's best us case is basically acting as pythons anonymous function (since lambdas suck for everything but the most trivial things).
Where in js flask might read app.route('/', function(){//do something}); python uses a decorator right above a regular function definition. The alternative is to manually add it after it's defined (in flask that's add_url_rule), but that kind of hides the intent of the function and is more cumbersome as you need to manually pass in the fn name.
On a related note. Which python library would you recommend for text-based interfaces? I have never used ncurses, so I don't know how complex it is. What I would like to achieve is having a user launch my script from the command line, use the text based interface to select a source and destination folder, set a few parameters and show a progress bar.
That syntax is usually meant for packages that you want to edit after installation (with -e).
If you plan to release your stuff, the dependencies in your req.txt should be as pinned as possible. The classic example is Requests: when it changed the API fairly significantly, umpteen installers broke... just because people did not bother with specifying a version for that lib.
(btw, it seems bad practice to add such a link as a dependency in your setup.py... usually you'd do it for temporary shallow forks, but otherwise I think it'd be better to also upload your shallow fork on pypi ...I guess you can just remove it from pypi if it won't be needed anymore)
Why is it that (nearly) every description I read about some random new Python command wrapper fails to get the "python" and ".py" out of the command examples, even in Linux?
I don't blame this particular offering, since I don't think release was actually planned just yet and any number of other projects have made the same subtle mistake.
Command Name Extensions are Harmful. Don't expose such an implementation detail in every example, lest everyone actually follow them. Use the "#!/usr/bin/env python" or whatever at the top of your scripts. And yes, you can keep the .py if what you have is a library, not just a command (but it's nice to then make a wrapper the doesn't expose the implementation language). And obviously in other OSes where the command extension can be omitted and still work this isn't such a big deal.
But in Unix/Linux, commands should be reimplementable in a different language without making some .(extension) a like, retained to keep from breaking other things that depend on it. Just say no :-)
It actually shows you how to set it up so you don't even use a #!, but rely on setuptools to make an executable for you, that'll work in a vitualenv or on windows. And the script name doesn't have .py at the end in his example, though he doesn't call that out specifically.
So many more to choose from. Now we get to evaluate Click. Seems like the reason Armin wrote Click was to load options dynamically, but that's what Cliff does via stevedore (https://github.com/dreamhost/stevedore).
I also really like argh, though it seems the provided link documents an api the mandates decorators and in fact what I like about argh is that functions can remain clean. http://argh.readthedocs.org/en/latest/
You're never forced to use decorators, they're just syntactic sugar.
Their example:
@click.command()
@click.option('--count', default=1, help='number of greetings')
@click.option('--name', prompt='Your name',
help='the person to greet', required=True)
def hello(count, name):
for x in range(count):
print('Hello %s!' % name)
Could be written as:
def hello(count, name):
for x in range(count):
print('Hello %s!' % name
hello = click.option('--name', prompt='Your name',
help='the person to greet', required=True)(hello)
hello = click.option('--count', default=1, help='number of greetings')(hello)
hello = click.command(hello)
The command decorator is nice, but the group decorator feels wrong, why create a function that does nothing just to decorate it? I'd rather use `cli = click.Group()`.
The biggest gripe I have with using Python for rich command-line tool is the startup time. One of the first reason I like to write a nice command-line tool when I start a project, say foo, is to be able to do `foo --help` to quickly see and remember what the project can do (I have a very bad memory and doing this makes it possible for me to jump back faster to a project, even well documented. I can forget what I was doing in just a few days and so I add a lot of small commands).
In short running `foo --help` should be instant and if it loads all its modules to list the different sub-commands and their respective description it is really too slow.
A possibility is to cache some information (e.g. generate a text file or a small Python script).
Hm, it'd be pointless to argue this point (if you think it's slow then it's slow) but I've created many a Python CLI and have never noticed them being slow to start. Perhaps it's a case of importing certain modules globally instead of on a per-function basis?
See my answer to coldtea. Yes you can organize you're code to help making it load faster but I don't think it is fast enough. Maybe my terminal is slow, maybe my machine is slow (and maybe I'm overly sensitive to the problem) but the difference is telling.
>The biggest gripe I have with using Python for rich command-line tool is the startup time.
What startup time?
I even have a python script running on every shell prompt drawing (that checks mercurial on top of starting the python interpreter), and the latency is negligible.
> time python manage.py --help
real 0m0.473s
user 0m0.240s
sys 0m0.148s
Half of a second is very noticeable (and annoying). I had similar perception in my previous work.
You can try to only load enough to display the help text without really loading everything, but that doesn't work that well (you have to organize things differently, and `--help` requires to load a lot of stuff. `subcommand --help` needs less but it still has to see if the subcommand exists).
As a reference:
> time python -c 'print "hello"'
hello
real 0m0.041s
user 0m0.024s
sys 0m0.012s
> time echo hello
hello
real 0m0.000s
user 0m0.000s
sys 0m0.000s
Actually even the first (0.041s) is not instant, while the second one does (I mean as I perceive it visually).
> Half of a second is very noticeable (and annoying). I had similar perception in my previous work.
That's because it has to load django and the whole django project. Here's click:
> time python test.py --help
Usage: test.py [OPTIONS]
Options:
--help Show this message and exit.
python test.py --help 0.06s user 0.02s system 95% cpu 0.078 total
and argparse:
> time python test.py --help
usage: test.py [-h]
optional arguments:
-h, --help show this help message and exit
python test.py --help 0.03s user 0.02s system 93% cpu 0.054 total
reference:
> time python -c 'print "hello"'
hello
python -c 'print "hello"' 0.01s user 0.01s system 88% cpu 0.025 total
Calling a shell built-in isn't really fair -- you should at least compare to /bin/echo (on my system the relevant times are [edit: 0.001s] for echo, 0.007+0.004=0.011 for python -S -c "print 'hello'". Without dropping site-packages (without the -S) python jumps to 0.014+0.012=0.026 -- marginally above 200 ms which I suppose is a perceptible difference between instant, and not-quite-instant).
Sadly, both pypy and python3 are slower than python2.7. Also worth nothing that the sys time fluctates for me -- in other words when it is > 0.00s it doesn't appear to have anything to do with the command run.
Still, your second example is an order of magnitude faster than the example with Django. That's likely because Django loads the world on startup, parses all your model files, checks your database settings, etc.
I'm completely clueless about this, but would it be possible to have a persistent python interpreter always running in the background somehow, and submit the commands to it?
You can keep ipython kernel running in background if you want. Quick unscientific test showed that it didn't reduce ipython console startup time significantly. But it should be possible to make a leaner console to connect to the kernel.
edit: my test was flawed, with more accurate test there is almost a full second time difference:
If loading the rest of the code and initialization time really matters, yes. But then you're back to square one: you now have to create a CLI to access the server and that new CLI must load quickly. Which would also be the case if you just wrote it that way first, then loading the rest of the code once the command is determined.
And if you're in some project similar to Django, you would have first to query the server to know the possible subcommands...
So what you say makes sense when other parts of the code are slow to initialize and must be accessed frequently, not for the CLI itself.
Ahh.. I really like pocoo's products. But I've found manage.py (https://github.com/Birdback/manage.py) to be far leaner and simple. What do you guys think?
I'm curious about that decision given that optparse is deprecated. It says it's because argparse doesn't allow nested commands, but I don't really see why that feature is so desirable as to warrant using a deprecated module. I probably need to have a play with it to find out.
Click can be extended lazily which helps execution time a lot when working with many, many plugins. It also can be extended at runtime as per configuraton.
argparse requires the parser to have full knowledge of everything which makes it slow if you add many commands. Biggest problem though is that it's parsing system is a bit broken when it comes to escaping. Options with arguments cannot have values starting with dashes which is problematic for delegating subcommands to other things.
For what I wrote Click for I could not find any alternatives that worked that way besides optparse itself but that is hard to use.
Can you provide an example of what you're saying? I've found that argparse.ArgumentParser.parse_known_args() and sub-parsers can do everything I can see click doing. Including lazy loading of plugins and the like.
user@uweb1:/$ pip install click
Downloading/unpacking click
Could not find any downloads that satisfy the requirement click
No distributions at all found for click
