Things like httpie, jupyter, etc should be provided as pyz and binaries as well, not just pip installable, but python devs don't do that really often.
It's a shame, because cli tools should be black boxes, people should not have to think about the implementation. I regularly see cli python tools that output stack traces on errors, and it should not happen.
Anyway, if you want those tools to be always around, I really encourage you to grap a usb stick, and put them on there as pyz and binaries. It's always nice to have them at hand.
Here is an example procedure for both (run them in a venv).
For pyz files:
python -m pip install shiv --user # (Yes, I realize how ironic this is).
Then:
python -m shiv httpie -o http.pyz -c http
(download httpie, and creates a http.pyz file that will contains httpie and all it's dependancies, plus code to trigger the http command when called)
This will let you do "python http.pyz" anywhere you have a modern python and enjoy httpie without the need to pip install it.
For an exe (this is an example on linux, the commands are a bit different for windows and mac):
python -m pip install nuitka httpie --user
You need to have a C compiler and python headers around as well (gcc on linux will do, E.G: sudo apt install build-essential python3-dev on ubuntu).
Now you have a nice stand alone AppImage you can call with ./http.bin to be used on any linux system, and not care if Python is installed to play with httpie.
Here are the resulting files (available for 5 days) so you can see what it looks like:
Basically Rust ignores SIGPIPE, so if you don't use write!() or handle SIGPIPE yourself, large amounts of data going through a shell pipe can result in a panic.
There's truth in that, but it depends a bit how you install it. The OS packages are very reliable, and creating your own bundle (through a venv, or through something like pex) is pretty reliable too.
(EDIT: talking about Python packages specifically, not so sure about other things)
My experience with Python has been that once you get past the initial setup (which may be tricky if you haven't figured your Python installation out yet), everything just works.
As you say, if something doesn't work due to conflicting dependencies, usually creating a virtualenv solves it. I don't bother these days, as just installing everything with --user tends to always work.
Yeah, I just read about it after seeing it around for years, and it looks fantastic. I might try it, but I've never had a problem installing apps, so I don't have a pressing need.
To be honest, the number of apps installed with python isn't really that high so you can get away with it most of the time. I like the fact that pipx solves the problem so I just do it once and forget about it.
The real problems start when people write scripts and notebooks etc that use libraries that might just happen to be installed or, even worse, install new libraries. The only solution to that is often a scorched earth approach where you vow to never install anything outside a venv again.
I also like Rust tools but Python is quite manageable using either pyz or pipx: I use the latter to install each tool in its own virtual environment. Periodically I run “pipx upgrade-all” to install updates (including injected plugins) but otherwise spend almost no time on it.
I never looked into Homebrew as I know that its mainly for MacOS.
Since you are sort of making the connection for Homebrew to work without fuss on Linux and Windows as well, if implemented in Rust, I am asking myself what it would offer that a user of Linux and Windows is missing?
Ordinary Linux distributions already have a package management and there is scoop and chocolaty for Windows.
Yeah, I can see how you could take that from my comment, it was a bit of a throwaway last line.
I mainly want it to be rewritten in a compiled language for performance reasons, it's a sluggish beast. I haven't tried to use it on Linux (where it has some support), and definitely not on Windows (winget/chocolatey would be my port of call there).
Examples of tools I use across operating systems that were written in Rust/Go, that pretty much are drop an executable somewhere and run:
ripgrep, starship, tailscale, terraform, vault, etc.
Whereas when something is written in Python, Ruby or similar, I'm rolling the dice, and I'm only an apt-get or dnf away from it breaking when I really need it.
I didn’t read their comment that way. Although Homebrew does also exist for Linux, I think the absolute majority use it on macOS.
Personally, being a user of Homebrew on macOS I’ve been thinking about wanting brew to be rewritten in Rust too. My own reason being that I think it would be faster when run, compared to the version they have of it currently which can be quite slow sometimes. And because I like Rust as a language and for its memory safety features. So I’d rather see brew implemented in Rust than in say C or C++, if a compiled language was to be used.
I don’t think that homebrew will ever get rewritten in anything other than ruby. It may be feasible to rewrite brew itself, but all the formulae are written in ruby, so moving away from it would insta-delete the entire ecosystem.
This is great! I've also come to really like curlie (https://curlie.io/). It is written in in Go and provides an `httpie` like interface with `curl` like performance.
Curlie is actually way better because it doesn’t mess with the default curl arguments. You don’t need to (re)learn httpie’s overly verbose command line parameters if you already know curl, and you don’t need to rewrite scripts or snippets that currently use curl.
No pretty colors, no JSON formatting, some kind of security vulnerability every month. Maybe the last one isn't quite fair though given curl's usage and the number of security researchers looking at its code.
I think the biggest value of this vs httpie is that it doesn't use the system TLS libraries (openssl/gnutls/whatever) but instead uses rustls. Less attack surface, though admittedly if someone has a compromise that works on arbitrary HTTPS traffic, you're probably pretty boned anyway.
The binaries of Rust/Go replicas should also be updated but to my experience few package managers provide packages for apps written in those languages.
So now you need Python + pipenv + the actual program? Versus just the binary with go/rust (or any compiled language for that matter)?
I think that's exactly the kind of complexity arusahni was referring to. Being able to deploy a single binary with `go get` or `cargo build` without worrying about anything else is really convenient.
I guess it depends on the use case though, I use Python extensively at work and have little complaints.
More like `pipx install httpie`. To update all python apps : `pipx upgrade-all`. For setting up pipx, you can use a pyenv managed python, like latest version, independent of whatever version is installed on system/distro. And you do this only one time. pipenv is for managing virtual envs not python itself.
`go get` and `cargo build` don't "deply" the binary but merely build it. It's `sudo make install` vs `apt update` all over again.
Point is, from the complexity of app management POV, those language don't magically solve anything that python doesn't, not to mention the bigger size of the binaries built with Rust/Go.
I do choose rg/fd/fzf over python-equivalents all the time but I acknowledge that I have to manually keep track of installing/updating these programs.
> I do choose rg/fd/fzf over python-equivalents all the time but I acknowledge that I have to manually keep track of installing/updating these programs.
No you don't. I'm not sure why you're spreading misinformation. And there is no viable Python equivalent to ripgrep anyway.
Well I'm on Xubuntu 18.04 and none of them are available on official repos and I do manually fetch the binaries from their repos (thanks for your awesome work and for providing `deb`s for rg btw ;) )
but yeah my observation was based on empirical evidence on older distros...things seem to have changed...
Still I find quite a lot of Rust apps that I like to use but there is not a distro package for them and the repos don't provide a binary or a deb or PPA. At least with Python repos, because of the nature of scripting vs compiled languages, I can simply isolate anything with a simple `pipx install app`
Isn't `cargo install` similar to `make install`? Usually it is advised against handling binaries in system paths independent of distro package manager.
Also, sometimes there is no `make uninstall`. Does Rust enforce an uninstall routine in the manifest?
With pipx, every app is contained in its own venv automatically and does not interfere with distro-managed packages.
In the sense that it is not a replacement for a real packager manager, yes. It is both slightly more restrictive and also not; as the sibling says, it installs to a particular directory, and that means uninstalling is easy, however, there is an install hook, but no uninstall hook, so if that puts files somewhere, which in theory it can, they won’t be removed on uninstall. This also kinda sorta means people don’t do that in the install hook.
For any Arch user running into this thread, wanting to install this: the project went through a rename (to xh), and a package has been added to the Arch "community" repo.
Adding HEAD as one of the methods would be welcome so we could get headers without pulling the whole page. Also, I did find the arguments for the -p switch in the source, but that should probably go in the README too.
I took a stab at something like this a few years back (https://github.com/saghm/rural), but yours is much fuller-featured. I'll definitely try this out soon!
Hate to be that guy, but it's a bit weird to pick the HT part of HTTP, which stands for HyperText, when the tool only deals with the Transfer Protocol.
I’m intrigued by the current version number and what seems to be a quick pace of development. I see that you have a “HTTPie feature parity checklist” (issue #4) on the issues list. Is that the sole indicator (aside from bugs) of what a version 1.0 would be like?
Also, please post another Show HN when the backlog is almost done.
Thanks, I will try post it one more time once the tool is stabilized.
>I see that you have a “HTTPie feature parity checklist” (issue #4) on the issues list. Is that the sole indicator (aside from bugs) of what a version 1.0 would be like?
Yes, that pretty much sums it. Also The list might expand if I notice other HTTPie features I forgot about.
% hyperfine 'http get EXAMPLE.COM' 'ht get EXAMPLE.COM' -w 10 -r 100
Benchmark #1: http get EXAMPLE.COM
Time (mean ± σ): 200.8 ms ± 7.4 ms [User: 178.9 ms, System: 21.1 ms]
Range (min … max): 166.5 ms … 221.0 ms 100 runs
Benchmark #2: ht get EXAMPLE.COM
Time (mean ± σ): 22.7 ms ± 3.2 ms [User: 8.7 ms, System: 5.6 ms]
Range (min … max): 15.6 ms … 31.3 ms 100 runs
Summary
'ht get EXAMPLE.COM' ran 8.84 ± 1.27 times faster than 'http get EXAMPLE.COM'
Yup, performance was one of the reasons I decided to port it to Rust. The other reason being that Rust gives you a single binary that is easier to deploy compared to python.
Nice work! As shared down-thread, I really loved and wanted to use `httpie` but the non-trivial amount of startup has put me off. Very happy that you made a Rust alternative because I really don't like `curl` that much -- it requires quite the amount of incantations for non-trivial requests. `ht` and `httpie` definitely improve ergonomics at important places.
Why do you need more performance from a CLI test tool? I'm honestly curious.
It's basically curl (fast) plus simpler and easier interface and pretty printers. For performance in shell scripts you can use curl, and for troubleshooting the IO time dominates anyway.
Why use something slower when an equivalent faster tool is available?
There's definitely a noticeable delay on my machine with starting up the python interpreter. Enough that it dominates most actual request times to fast servers. (`http get www.google.com` is ~460ms while `ht get www.google.com` is ~130ms)
For a tool I'm constantly using to check APIs I'm developing, I really appreciate snappy commands that give me results that feel instantaneous.
It adds up. I once tried to scrape an internal company website / API because we had no PDF exports and wanted to download + export to PDF, with `httpie`. It's an amazing tool but all the startup times compounded pretty badly. I switched to `curl` (which is a bit more pain until I got the full command line right, granted) and the script finished in ~95 minutes as opposed to the ~202 minutes it took with `httpie`.
With about 0.5s startup time, it means every 120 requests add a full minute to the final time. And I had to scrape ~125K URLs back then.
For a daily casual flow 0.5s startup time might not be much (although people like myself get irritated by that as well but I do recognize it's a minor inconvenience). But when doing mass-scripting such delays can very easily compound to non-trivial time inefficiencies.
Still, it was an one-off thing and I didn't want to turn it into a project. Using a for loop, `curl`, `wkhtmltopdf` and `parallel` eventually did the job.
I for one am totally on board with writing atrocious grep/sed/awk pipeline mashups, doing whatever it takes to get a one-off job done with as little programming as possible. Sometimes the important parameter to optimize is brain cycles spent, or time to result, not CPU time or optimal usage of network resources.
Absolutely. I'd even argue that the brain cycles spent should be the first and most important metric for one-off tasks.
That's why I recently asked people here about what to use if not shell scripting. I think I'll either use a less confusing scripting language that is almost like bash/zsh -- namely `rc` -- or I'll brush off my OCaml skills.
That need was borne out of my frustration that I routinely spent non-trivial amounts of time on one-off scripts so at one point I got fed up and started looking for alternatives.
I currently have the dynamic language blues – Python to me just doesn't feel like as much fun as it used to. Maybe it's getting too big and/or too crufty. At the moment I'm having great fun with Rust instead, which feels as magical as Python and C did when I was twenty years younger.
That said, bash/sh are really good at setting up pipelines to pass around lines of text between Unix tools. That is a really cool thing, and most languages don't really make this as first class concept. Because bash does, and I have twenty years of experience with it, I mostly put up with Bash. It's not a great language but it gets the job done, literally.
I recently read some stuff about Julia which made me interested in it as a potentially better Bash (and hopefully more fun than Python) –
Same, I used to be a Ruby (and Rails) fanatic a long time ago but this has subsided. Nowadays I love Elixir and I reach for it in many scenarios where I don't expect CPU-bound workloads but I am acutely aware that the startup times (when scripting) are atrocious, plus working with it in business context for 4.5 years made me painfully aware of the problems caused by a lack of static typing.
> At the moment I'm having great fun with Rust instead
Same, I love it more than I loved almost any other tool I tried in my life. Just not sure how economical it is to reach for it for throwaway one-off tasks. I suppose with more experience in it it would become feasible.
> It's not a great language but it gets the job done, literally.
Agreed, I have achieved a lot with bash and `parallel` alone. But at one point it became a death by a thousand paper cuts thing; I know I can do the task but I can never quite remember all the little annoying details when e.g. iterating numbers, or file list from a previous command, or was there anything specific about piping inside the scripts (there is not, but you have to be mindful of config like `set -euo pipefail`) etc.
It might be me getting stupider and less tolerant with age but you know what? I am okay with that. These tools / languages should be intuitive and easy to remember!
I am not abandoning bash and zsh scripting, ever. But my tolerance towards banging my head against the wall was severely reduced lately. And, as mentioned in other comments of mine here, I'll either use `rc` or re-learn OCaml -- since it's extremely terse, compiles to very fast binaries (faster than Golang and only slightly slower than Rust) and has static strong typing.
But time will tell.
---
Thank you for the fantastic links. Opened them on my phone, they'll definitely be researched.
Yes, in the context of my comment higher in the thread about using libraries. This gives you a lot more control of the connection pool and queues, which can get a lot more performance than simple cURL usage. From the cURL manpage:
curl will attempt to re-use connections for multiple file transfers, so that getting
many files from the same server will not do multiple connects / handshakes. This
improves speed. Of course this is only done on files specified on a single command
line and cannot be used between separate curl invokes.
So using a library gives you more flexibility in reusing connections (and avoiding the expensive TCP/TLS handshake) than the simple patterns you can use with the cURL CLI.
Oh believe me, I am aware. I just didn't want to write a program for my task back then (even if it would have likely taken almost the same amount of time as it turned out in the end).
That's why I am searching for an alternative to the classic shell scripting and I have almost settled on OCaml.
Python is alright as well but never really liked it: just when you might need a tool that's quick to launch, you'll hit the Python interpreter startup overhead. Same goes for many other interpreted languages.
It was supposed to be a one-off task and I was confident at the time that I can script it quickly.
I still scripted it but it took at least half a day.
I'll still behind the idea that one has to be able to do one-off tasks without starting a dedicated programming project. It's what scripting languages are meant for.
But I did misjudge the speed at which I'll be able to finish the task, that much is true. Ironically I knew exactly what to do from the start but several small and annoying quirks of the shell scripting languages lost me quite a bit of time.
Because slow startup time is obnoxious for any command-line script. There's no reason to start up an entire scripting virtual machine just to make an HTTP request. No one should be writing serious CLI tools in an interpreted language.
And yet for decades serious CLI tools are being written in POSIX shell, Bash, Zsh, Awk, Tcl, Perl, Ruby, Python, and no one really complained about speed until now, when hardware is so fast, finally people are starting to notice slow startup times??? C'mon. Python is not Java.
I can definitely tell a difference, and it's irritating. The authors of these tools might not notice or care but I actively avoid them. Magic wormhole is one such example.
Sorry about asking this, I don't mean to be mean or anything, please don't take offense on this - I'm honestly intrigued.
I would like to understand from where a question like this comes, I understand different people have different experiences and backgrounds, but being on this site I would assume this would be kinda obvious.
Can you share anything about your experience that can help me understand it?
FWIW as a developer whose worked in both interpreted and compiled languages (but not Python) it's obvious to me that the compiled version will be faster, but it's not obvious that it'd be meaningfully faster. I'd actually expect network latencies to dominate here in many cases.
As another commenter mentioned I suspect what's being measured here is the Python startup time rather than performance once things are running.
My guess would be a faster to start interpreter (like lua or v8) would rival ht performance, while a slower to start compiled language like the JVM would be as "slow" or slower than httpie.
I know many people who wouldn't necessarily understand that compiled programs tend to be faster, because it just hasn't come up, or because in their domain it's not really important or even clear.
If you've been working primarily with Javascript, for example, or even worse, Typescript, the question of "is this thing compiled or interpreted" is "yes". Compilation becomes a piece of either build tooling, where the artifact isn't a concrete binary, or even an attribute of runtime.
It really wouldn't be too hard to imagine a shop built on Javascript and Python, where compilation just doesn't factor into the system in a meaningful way.
These days I try to avoid interpreted CLI tools, you never know when they’re going to stop working.
Rust/Go CLI tools basically work on Linux, Max, Windows without too much fuss.
Wish someone would rewrite Homebrew in Rust!