What kind of cost? Obviously, backups on the author's server are not something I'd ever do, but the "offline only mode" is there, apparently.
TBH I was thinking about doing this for a while now. History of my shell, which now is 3.8MB in size, is one of my competitive advantages as a developer (and a very nice thing to have as a power user). It accumulated steadily since ~2005, and with fzf searching the history is much faster than having to google, as long as I did what I want to do now even once in the distant past. I even wrote a utility to merge history files across multiple hosts[1], so I don't have to think "where did I last use that command", as I have everything on every host. The problem with this, however, is shell startup time. It started being noticeable a few years ago and is slowly getting irritating. The idea of converting the histfile into sqlite db crossed my mind more than once due to this.
This is interesting, thanks I'm checking it out. I was thinking this could be a flash in the pan thing that would then be annoying to maintain, but obviously global history of everything you've done is definitely a super boon to productivity. How do you handle maintaining this as you transition through jobs, machines, etc? 3.8MB is obviouslly trivially small in size so you could store it on a potato, but like what is your workflow around maintaining these one off ad hoc "developer boost" type tools?
> How do you handle maintaining this as you transition through jobs, machines, etc?
Currently, the tool reads ~/.mergerc, which is a JSON file with a list of SSH hosts to SCP history to and from. As long as the history file is in the same place (it tends to be on hosts that I setup, and otherwise I check in default locations) and the host has an entry in ~/.ssh/config, the tool will work. It's really just a wrapper for a few SCP invocations plus a history file (extended) format parser.
Changing servers is just a change in the config file, but it's also helpful for changing jobs, because I can quickly add a bit of filtering before the merging happens. I had to erase some API keys and such a few times, adding `filter` call here: https://github.com/piotrklibert/zsh-merge-hist/blob/master/s... took care of it.
> what is your workflow around maintaining these one off ad hoc "developer boost" type tools?
Good question. I don't have such workflow, at all. When I commit to write something like this, I try to make sure that it has a scope limited enough so that it can be "completed" or "done". In this case, the tool builds on SSH/SCP and a file format that hasn't changed in the last 20 years (at least). So, once I had it working, there was nothing much to do with it after that. The only change I had to do recently was changing `+` to `*` in the parser, because somehow (not sure how, actually) an empty command made it into the file. But that's all I had to do in 5 years time.
I'm not as extreme, but suckless.org philosophy appears to work well here. Here's another example: https://github.com/piotrklibert/nimlock - it's a port, done because I wanted to do something in Nim, but it worked for me for years and I suspect it still works now (after going full remote I stopped needing it). There's nothing much that could break (well, Wayland would break it, but I don't use it), and so there's not much you need to do in terms of maintenance.
As for language choices - these are basically random. I made the zsh-merge-hist in Scala simply because I was interested in Scala back then. I have little tools written in Nim, OCaml, Racket, Elisp, Raku - and even AWK (pretty nice language actually) and shell. That's another reason why making the tools any more complex than what's absolutely necessary would be a problem: the churn in the ecosystems tends to be too high for me to keep track of, especially since I'd need to track 10 of them.
EDIT: I forgot, but obviously the most important "trick" is not giving a shit if these things work for anyone else but me :D
> I'm checking it out
If you have Java installed, `./gradlew installDist` should give you `./build/install/bin/zsh-merge-hist` executable to run. The ~/.mergerc (on the host the tool runs) should look like this:
One such cost could be database size. I currently have 45k history entries in my database and it sits at roughly 15MB in size due to database indices along with the additional data we store
But it's a cost shared by all open shells, right? Well, even if it was 15MB per shell session, it should still be worth it if the startup and searching is faster.
For comparison: I use extended ZSH history format, which records a timestamp and duration of the call (and nothing else), and I have ~65k entries there, with history file size, as mentioned, 3.8MB. It could be an order of magnitude larger and I still wouldn't care, as long as it loads faster than it takes ZSH to parse its history file.
There's currently no paid option. Obviously one should assume such a free service can't be sustainable so you'd be correct to think we should have one.
Currently we rely on Github sponsors as well as our own additional funding
With the server, is it primarily a gateway to the hosted SQLite databases?
eg receives incoming shell history to store in the backend, and maybe do some searches/retrievals of shell history to pass back? eg for shell completion, etc
If that's the case, then I'm wondering if it could work in with online data stores (eg https://api.dbhub.io <-- my project) that do remote SQLite storage, remote querying, etc.
We currently use postgres. The server is very dumb, verifies user authentication and allows paging through the encrypted entries.
There's a PoC that allows it to work with SQLite too for single user setups - and we are thinking of switching to a distributed object store for our public server since we don't need any relational behaviour.
Interesting. Yeah, we use PostgreSQL as our main data store too. The SQLite databases are just objects that get stored / opened / updated / queried (etc) as their own individual things. :)
One of our developer members mentioned they're learning Rust. I'll point them at your project and see if they want to have a go at trying to integrate stuff.
At the very least, it might result in a Rust based library for our API getting created. :D
> we are thinking of switching to a distributed object store for our public server
As a data point, we're using Minio (https://github.com/minio/minio) for the object store on our backend. It's been very reliable, though we're not at a point where we're pushing things very hard. :)
Can you tell me if my understanding of this issue is correct?
Let's say I run a command where I've pasted in a credential from my password manager: ` some-cli login username my-secret-password` (note space at beginning)
Normally this would prevent the command from getting saved in any meaningful way in my bash history, so that if I later run a malicious script, it can't collect secrets from my bash history.
With the bug here, it sounds like atuin would prevent that entry from being stored in the sqlite store, but it would still be in my shell history?
If so, this is really significant, and would stop me from using Atuin. Not letting users know about this behaviour is incredibly negligent, and honestly erodes my trust in Atuin to consider user security in general.
it's not serious for most people I guess, but if you rely on bash's HISTIGNORE and don't disable bash's built-in history mechanism when you adopt Atuin, then this is as serious as you are paranoid
