The go flag package has a nice API, but the user interface it creates is straight from its Rob Pike's Plan9 roots, and Linux users familiar with getopt command line interfaces will find it strange and perhaps even ugly.
I wanted to create command line programs that look like traditional GNU/Unix utilities, so I wrote the https://github.com/ogier/pflag package. It's a drop-in replacement for the flag package, just change the import line. It also adds some basic features, such as the common pattern of a flag with a long-hand form usable with a double --dash and a shorthand character with a single -d.
The Go flag package has nothing to do with Plan 9. Like Unix, Plan 9 had single-dash single-letter options ONLY. The long option syntax of getopt is a BSD/GNU-ism, not from Unix.
The Go flag package's command lines adopt the observation made by the Google C++ command line flag parser (https://code.google.com/p/gflags): the long/short --/- two-names-for-every-flag dichotomy was useful for backwards compatibility when recreating original Unix commands, but for new programs, it doesn't pull its weight. At the cost of requiring that each option be specified with a separate command-line argument, you can drop the distinction between - and -- entirely. (This is similar to what X11 programs do, although I believe they do not admit --x as a synonym for -x.)
It's not clear to me what is "ugly" about not having to keep alternating between - and -- as you spell out options on the command line.
> the long/short --/- two-names-for-every-flag dichotomy was useful for backwards compatibility when recreating original Unix commands, but for new programs, it doesn't pull its weight.
Short is more usable for users on the command line. Long is more maintainable in scripts. That's why it makes sense to have both.
> At the cost of requiring that each option be specified with a separate command-line argument, you can drop the distinction between - and -- entirely.
Again, that cost is high for users on the command line, one of the few places where every keystroke does matter. It's a lot more pleasant to be able to do:
$ rm -rf
Than:
$ rm -r -f
Or, worse:
$ rm --recursive --force
Orthogonality is definitely an important principle, but when it comes to usability, it often makes sense to offer multiple ways to accomplish the same goal for users in different circumstances.
Ultimately, you're interfacing with a greater primate. Even with all of our nice features, we still don't support USB or web standards. Instead, it's all lossy OCR and weird muscles. You should expect a certain amount of... squishiness... when interfacing with something like that.
Re command line vs scripts, I am not convinced that the mental overhead of knowing two names for each flag and flipping back and forth when you context switch between the command line and a shell script really makes that much sense. Certainly common flags should be short and less common flags need not be, but personally I'd rather have one way to spell each. Certainly having just one name per flag is more in keeping with Go trying to keep things simple.
My officemate is a long-time Git user but was surprised the other day to see me type "git pull -r". He didn't know that -r was short for --rebase, and in fact I didn't know that --rebase was long for -r. This kind of mutual incomprehensibility is a tiny variant of the observation that every C++ programmer uses only a small subset of C++, but the problem is that each programmer chooses a different subset. Not having two names for a flag can be a feature.
Re typing, I am also not convinced that the overhead of rm -r -f vs rm -rf is really that high. Of course, rm is a bad example: if you are implementing rm, you MUST use the Unix conventions and accept -rf. But for new commands not burned into decades-old muscle memory, it's not clear that it's needed.
I think there are multiple valid design decisions that could be made here, and we've tried to tilt the balance toward simplicity, but of course other balances could be struck, and maybe in other contexts it might make sense to strike a different one.
I tried to make sure it was possible that someone could write a package (say, flg) with the following API (and nothing else):
// Package flg implements a getopt-compatible command line flag parser.
package flg
// Parse parses the command-line flags defined in package flag.
// In contrast to flag.Parse, Parse imposes getopt semantics:
// - single letter flag names must be specified with a single dash: -x
// - longer names must be specified with a double dash: --long
// - the argument to a single-letter flag can follow it immediately:
// -xfoo means -x foo when -x takes an argument.
// - multiple short flags can be combined: -xyz means -x -y -z,
// when neither -x nor -y takes an argument.
// - name aliases can be introduced by calling Alias before Parse
func Parse()
// Alias records new as an alias for the flag named old.
// Typically old is a long name and new is a single-letter name or vice versa.
// For example, Alias("r", "recursive").
func Alias(new, old string)
I believe it is, although I haven't seen one. That would let programs still use package flag as the data definition, so that they can reuse any other "plug ins" for the package (like custom flag.Values) but still provide a getopt-style parser if desired.
I would argue (at the risk of being down-voted) that if you are scripting tools then it is important to use the long option, not just because future-you might forget what the options are (relevant XKCD http://xkcd.com/1168/), but also for searchability, it's rather difficult in almost any engine (except the wonderful new code-orientated ones that are cropping) up to search for `-xvzf`, but rather simpler to search for `extract verbose zip filename +tar`, for example.
That's really what munificent was arguing too though -- that the long-form options are preferable for scripting, but that no one prefers "tar --extract --verbose --compressed --file" to "tar -xzvf" when you just have to extract a file on the command line (in fact, many people don't even bother with the "-", "v", or "z", opting for "tar xf"). This optional brevity is important for utilities.
> It's not clear to me what is "ugly" about not having to keep alternating between - and -- as you spell out options on the command line.
It's about the principle of least surprise.
Regardless of whether double-dash opts came from UNIX or GNU (and obviously you are correct in that they came from BSD/GNU), they're the convention nowadays and it's less confusing for end users to just embrace the convention.
Go has bucked the (recent) tradition on a lot of different fronts, which I love it for, but the flag package -- the interface to the users of the software, not the authors -- is one place I don't really think it's quite as appropriate.
Certainly, Google is able to make these sort of changes in its internal world, and reach a high enough percentage of software that the new choice becomes normal, but Go will probably never have enough mindshare in the extraGoogle for its CLI parsing to not be "a little weird".
---
As an addendum, I happily use the builtin flag package for software that is consumed primarily by my team. For software I expect other teams to use, who don't necessarily care about the fact that it was written in Go, I prefer pflag.
People insisting that "XYZ is not UNIX", Unix or however you want to capitalize it, is kinda pointless. For all practical purposes Linux is Unix, FreeBSD, OpenBSD and OSX are Unix.
One would think that if any OS would foster the idea of sameness it would be those belonging to the Unix family.
As for old vs new style command line options: that's now Unix in much the same way that Dave Gilmour was a proper part of Pink Floyd and not merely "the new guy".
The original OS we call Unix has not been relevant to the discussion for 20-25 years.
Unix now refers to a family of operating systems -- not a single OS.
Thinking of Unix as a specific OS and a specific trademark in 2015 is just odd since the majority of people using various forms of Unix today were not even born when the distinction was even relevant.
And just as with other families, the peculiarities of a single member does not automatically apply to the entire family.
There are so many existing scripts, books and guides that use "rm -rf", "du -hs", "ls -la" and so on that those will have to be supported forever.
For many people, those sorts of commands are 80% of what they run on the command line. Having it so that the other 20% consisting of new programs works different is confusing and annoying.
There is a difference between writing systems in an environment where you have a centralized flag parser and flag inheritance across google3, and shipping a binary to users who expect things to behave like getopt. Also, let's be honest, flags in google3 are a nightmare. They are certainly well-engineered but lost any semblance of sanity a long time before you and me got there. The kernel patch springs to mind.
Having to rethink my entire career of something simple like "how to pass flags to a binary" had better bring significant improvement, and flag does not. flag regresses a lot of convention about the operator interface to a program. This:
I understand the argument for self-explanatory command lines but at the cost of unexpected (and far more verbose) behavior is a tough trade off. One thing I haven't seen discussed is how "required" options are encouraged by flag because of how difficult positional arguments are. That's in addition to everything in the article, which I agree with.
Python argparse gets this nearly perfect, subcommands and all. flag is near the top of what I hate about Go as an SRE. Java too.
The maximum command line size allowed by the Linux kernel used to be 128kB. This might sound crazy, but that limit was a real constraint for some Google services. My guess is that this is referring to a kernel patch to increase the maximum size.
This is a somewhat educated guess. A long time ago at Google I used to work on the program that was used to launch most jobs on the clusters. To help mitigate the command line length problems, the launcher preferred generating command lines with -nofoo instead of --foo=false, and -foo instead of -foo=true. This saved a few characters per boolean flag. Of course it also caused no end of trouble for people who'd defined their own "three-state boolean" flags that could e.g. be true/false/autodetect. Not the finest hour of engineering, in retrospect.
Shouldn't programs simply support a @args.txt argument which references a file containing more arguments to the program? Most build tools I've seen do that and it makes command-line length limits mostly a moot point.
Packages like this are where I find the debate between idiomatic Go usage and general use cases difficult to side on. I would err on the idiomatic side by using stdlib flag.
"Use stdlib unless you can name a reason good enough to incur the cost of importing another package, which shouldn't be underestimated" is probably the short version of the idiomatic Go position. If you've only got three flags, there probably isn't any reason to use anything other than stdlib; you can't be winning big enough to pull in another package. If you're solely being driven by external scripts, it's harder to win than if you expect interactive usage. Of course, if you're writing a grep replacement and you've got half the alphabet in both capitalizations being used and half of them take arguments and a couple of things switch "modes"... well, by all means don't write that yourself, go get a package that does it well. Most of the stdlib is designed to hit the 95-99% use case, but if you're really pushing something hard you may need something else.
Probably the only truly controversial bits of the Go standard answers are A: pulling in external packages isn't free and extremely frequently underestimated over the life of the product, they better be bringing some real value beyond merely preventing the developer's nose from crinkling a bit [1] and B: if you've only got three flags right now, don't go leaping to the conclusion you're in the process of writing a grep replacement; wait until you actually have the situation I described before dropping another package in (YAGNI).
[1]: Contrast this with something like Node or Ruby that seems to encourage pulling in dozens of libraries, many of which may literally be single-digit lines of payload code, and it is considered totally worth it for developer tastes to alter significant elements of the language/runtime/stdlib. And I really do mean contrast this with... I am currently expressing no opinion as to which philosophy is "better".
Of course if the Go stdlib produced something which looked more like almost every other Unix-like tool out there, then we wouldn't be having this discussion - but it doesn't, and leaving your app's commandline parsing until it becomes a catastrophe to manage (or your users complain) is technical debt whichever way you spin it.
For a language like Go I also strongly question why keeping with the stdlib is necessary. There's value in a language like Python where you the environment is dynamic - Go produces statically linked binaries. Who cares what libraries you build them with? That's kind of the point.
"leaving your app's commandline parsing until it becomes a catastrophe to manage (or your users complain) is technical debt whichever way you spin it."
I wouldn't deny that.
But I do agree with the Go philosophy (now I am expressing an opinion) that says you should also not underestimate the ongoing technical price of taking on a 3rd party library. Most developers consider it "free". In practice, it isn't. It is entirely possible for the value to exceed the cost, but developers rarely make a sensible decision... they just account the costs as zero and go grab the library, and often don't even notice what they're paying, and certainly don't notice what they're costing the team.
(Bear in mind we're talking "stdlib" vs. "grab a library" here, so NIH-syndrome is not currently on the table.)
And you're also really prejudicing the discussion, and indeed probably your own mental visualization, by speaking as if every Go program is a budding "command-line catastrophe" waiting to happen, when in fact BY FAR the most common case is that your program takes zero command line parameters, and the vast bulk of the remainder is five or fewer with no complicating factors like "modes" or something. Those who are programming the remainder are free to grab any of the 5 libraries I saw mentioned in the discussion yesterday (and probably more since I last looked), and nobody is going to complain.
"Who cares what libraries you build them with? That's kind of the point."
The maintainer. Go is generally focused on "serious" programs who will be maintained by teams of developers over periods of time that typically involve 100% turnover (though, hopefully, not all at once). If you don't have that problem, well, Go is going to be less appealing to you and I invite you to consider other options. If you do have that problem, you either A: already know how big a problem that is and will tend to appreciate some of what Go has to say, even if you don't quite agree with the solution or B: in another five or six years will fit into A.
In general I am inclined to agree with you. Given two equivalent libraries, if one is standard and one is pulled from github, the standard library should be preferred. If it were just a matter of developer ergonomics, I would agree with you in this case too.
But the command line interface is the fundamental user-facing piece of a command-line program. If it's not intuitive and clean, the usability of the whole program suffers. The internal API is secondary (this is why I didn't try to "improve" anything here -- go's flag package is good enough and I'm not trying to reinvent the wheel). It's all about a good experience for users, and I don't think Go's stdlib flag package provides that.
Commandline programs written in Go shouldn't feel like idiomatic Go programs, they should feel like idiomatic commandline programs.
If idiomatic Go has you preferring using packages that provide terrible, confusing, non-standard[1] UI to users of your program, then either idiomatic Go is wrong, or needs better interpretation.
It's interesting that you acknowledge that struggle but settle on what I'd consider to be a poor choice for your users. Potato/potahto, I guess.
[1] Whether you like it or not, GNU/BSD-style getopt/getopt_long is the standard these days, by a wide margin.
Also: getopt is pretty easy to write (if you're going for strict Unix style, you're presumably not looking for a featureful getopt), and that interface style is abused far more often than it's deployed gracefully. Some of the worst Unix command line interfaces† are the product of getopt, or the belief that there's an idiomatic Unix argument handling convention and that the idiom is getopt.
If your argument parsing needs are complicated, you should break out an actual parser.
Finally: I see the benefit of having an auto-generated usage message that is better than a bad auto-generated usage message, but if you really want artisanal command line tools, shouldn't you be writing artisanal usage messages? C programmers as a rule didn't count on the library to generate the usage message for them.
I think this is missing the point of the article. The author doesn't seem to care too much about complicated argument parsing needs, just about the interface that Go's flag library presents to the user.
You touch on the auto-generated usage message: personally I think the one that Go's flag lib generates is terrible. I agree wholeheartedly with the author. C programmers indeed don't count on libc to generate the usage mechanism (though, frankly, if it did, I certainly wouldn't mind), but we're talking about Go programmers here. And if idiomatic Go is to use the flag library, then you get an auto-generated message. And it's bad.
I think the other part of the author's gripe is just as (if not more) important: flag has decided to do away with the difference between '-' and '--' options, something everyone has come to expect as standard. In the Go model, it's weird to have multiple options for the same thing. For frequent users, typing "rm -rf" is better and easier than "rm -r -f", which is still better than (what I would think Go would encourage) "rm --recursive --force". The short options (and ability to chain them without a dash per option) is great for command-line users, and the long options are great for scripts where you want clarity and self-documentation.
Being able to make up complicated parsing rules is potentially another problem, sure, but I don't think it's as important to users of a Go program as either of the above issues that the author raises.
(For the record, I absolutely love Python's argparse. I think they've settled on a fair compromise between giving a lot of power, but making the simple cases simple, while providing behavior and UI that most users of my programs would expect. Not perfect, but I've been happier with it than with any other parsing library. Of course, implementing an API like that in a non-dynamic language that doesn't support optional, named function arguments might be difficult or impossible.)
But why should you have to do extra work to override a default that's sub-optimal from a standardization and user-familiarity standpoint, and is trivial to fix "upstream" where everyone should benefit?
I think the ideal argument parser allows you basically to write a spec for your program, and then automatically generates a parser that will both turn options specified by the user into a map, as well as print the relevant parts of the spec when invalid input is given.
Could you elaborate on why you believe flag is preferable to something like what I've described? (I'm coming from Ruby-land where Trollop basically does that.) I'm struggling to understand what's preferable about needing to roll your own parser and write documentation when you could just describe your program and have the parser/documentation auto-generated.
Trollop seems like a "better getopt". Conceptually it's still a series of flags. For simple programs, flags might be all you need, and there's value in cohering with all other Unix tools that communicate strictly through flags.
But when your needs get more complicated, when the information being communicated on the command line starts getting more dense, the concept "flags" becomes an obstacle. Far too many programs keep leaning on flags far past the point where the abstraction breaks down. You start to feel like you're writing small programs in the confining language of command line flags.
A great example I think is "tcpdump", because it exhibits both behaviors: it has a hideously overcomplicated getopt configurator and a relatively graceful parsed language for describing capture filters (those filters being the most important option given to tcpdump). You can see how tcpdump would be better off if most of the flags were hoisted into that filter language.
(They can't easily be, of course, because that filter language is actually parsed --- and then compiled --- by libpcap, not tcpdump).
What I'm saying is that flag feels like a "worse getopt," and you seem to be going from the assumption that if you need more than 2 flags you should roll your own DSL parser, which sounds to me overengineered and harder for others to pick up. Most programs are not tcpdump.
It's OK for the standard library to satisfice or even constrain flag parsing, because programs that need "complicated flag parsing" are better served by something more sophisticated than flags anyways. Super-awesome flag libraries just trick programmers into building bad UX.
A lot of programs would also be better off if they reduced the numbers of flags they had, and made the "value" of those flags more sophisticated. In other words: taking some of the weight off the flag parser.
Most programs aren't tcpdump. Most programs don't have a lot of flags. But most programs that do have a lot of flags would probably be better off with something other than flags as their command line idiom. "tar" springs to mind.
It's hard for me to look at my zshrc and see all the aliases with their -fNqC's and -avz's and think "this is a good command line user experience". Many of these flaggy CLI programs have terrible UX.
There's maybe a "Gettysburg Address In Powerpoint"-style satire to be done of, say, awk commands expressed entirely in flags.
I cant agree more with the importance of consistency UI in CLI programs. It's unfortunate that newer programs overlook it.
I'm glad for the recent changes from the flag package which has also been annoying me previously ;)
When i type blah -h or --help i expect help to pop up.
I expect it to tell me about USAGE with [optional] and <required>. I expect the same spacing and readability as the vast majority (which is basically dictated by GNU getops.
Else, its a pain. Heck, even git doesn't follow this and as a result is a little more painful that needs to be.
Imagine a GUI program with every window having a different close, resize, etc button/menu. Hell.
I always use go-flags [1] in my Go programs. It has support for GNU-style options, pretty help output, subcommands, automatic parsing of more complex arguments (e.g. time.Duration) among other things.
I do wish they would improve the included flags package though. There are lots of nice third party packages, but having a proper one in the standard library would've been better.
Came here to say this. go-flags is an excellent package, and makes great use of struct-tags. An extra bonus is that it can parse .ini files and command-line flags into the same underlying config struct, allowing one to override the other.
Well, I don't think the writer proved the importance of UI.
But I think it serves as an excellent reminder that, for a cmdline program, the flags and arguments are the UI, they impact how people perceive your software and how productive they will be with it.
I like the present (older) output because it's absolutely unambiguous. If I have to read a textual description to understand whether it's a default argument, whether it takes an optional parameter, or how precisely it is used, it's less clear especially when the text is badly written. The current output is probably also perfectly suitable for parsing (by a program).
The flags package is one of the few Go stdlib’s where I haven’t been won over; I felt that the cost of my understanding the abstraction was higher than just implementing the logic.
I wanted to create command line programs that look like traditional GNU/Unix utilities, so I wrote the https://github.com/ogier/pflag package. It's a drop-in replacement for the flag package, just change the import line. It also adds some basic features, such as the common pattern of a flag with a long-hand form usable with a double --dash and a shorthand character with a single -d.