There's a way to configure readline (which bash and many other use) to use up arrow history, like Matlab or Julia do by default, where you hit the up arrow key to search the history for commands that start with what you've typed so far. It is a small thing, but I use it all the time.
The only downside ist, don't over-rely on history muscle memory! Regularly every few months I use p+arrowUp and expect to get "ping" but get the exceptional "pip" instead.
Well, having read (or rather: started to read/tried to read) many of Stephen's blog posts, I would argue that not every single one of his musings should be recorded. :-)
Thats interesting to learn. I think he is on to something.
The one thing that computers are really good at is storing data (especially text) so I believe it should be a core function of any computer that it should never loose anything a user has input.
Sadly the reality is far from the truth. The number of times I have entered something into a webform or an app, only to have it disappear for some weird reason...
I find it peculiar to ask this question but then also deduplicate your history file because then you lose complete history and context. Sometimes I've had to issue series of commands to fix a particular problem (changing a kerberos password, fixing the expiry date of a gpg key, etc.) that I can backtrack through the history file to find what I did and how I resolved it, later having to do the same, and according to their settings the first instance would be removed.
Personally, I can't remember the solution to everything I do so having that deduplicated log gives me the full history of what I did and how I resolved it, amongst other uses. Sure, removing duplicate 'ls' entries is fine for example, but for every command? Not so sure.
You know what, you're right. And it's actually something that's been bugging me for a while, I just never got around to fixing it. I'm going to update my config now. Thanks!
I also don't understand why they're doing it. On my long-running full history duplicates cost me a factor of 2 in space, and presumably the same in search time (but still perceptually instant).
to ensure that my shell history file isn’t deleted.
Being append-only, it also ensures that some type of timing issue in my shell which means my HISTFILESIZE or the equivalent isn’t set to something larger than default, meaning my history isn’t deleted.
If the shell is unaware that the file is append only, does it result in appending not just the commands run from this session but undesirably also ends up appending a full copy of a lot of the previous history?
I use zsh-histdb which stores the history in SQLite with context. Then I have few different queries/ways to interact with it. For example I have a line completion (which also shows as grayed out text while I type) based on most frequently used commands with give prefix with additional weighting depending how close in the directory tree they were used.
oh, incredible. this is exactly what i wished existed for years but assumed was too absurd/overkill to have ever been made, so i didn't even search for it
Prefixing these utility scripts is a nice tip, I used to do that as well.
Some time ago I found sd [1] though. It's a light wrapper around your own scripts which provides namespaces, autocompletion, custom help texts + some other QoL enhancements around that. It improves discoverability and usability a lot, very happy with it.
Prefixing your scripts is an awesome tips. A colleague told me he prefixed all his commands with ',' and I've started to do the same. '_' might clash with 'hidden'(?) shell expansions etc.
Sadly git doesn't allow for aliases with an comma-prefix, so all git aliases starts with a dot.
Nushell has a SQLite history mode which captures more metadata than is typical. I haven't looked into extending it, but it seems like a good setup for adding enough so that some of these cognitive requirements could be recovered from history rather than thought of up front.
Like if I tend to go down the wrong path, fix it, and then go down the right path, maybe that's a pattern that can be identified and then when I search history it finds two or three commands which together set me on the right path in the first place.
I mean, thinking ahead is always gonna be better, but I don't always.
I do a lot of literate coding for shell work, using org-mode. I have param substitution, command chaining, textual explanation and saved output from the last time commands were run. I can save the .org files for reference, or add them directly to my blog.
The article briefly mentions atuin at the end. I've tried atuin but found it a little bit too heavyweight for me - instead I use zsh-histdb[0] (together with the fzf extension for it[1]) which allows you to easily answer this type of question - can highly recommend it.
I tried atuin as well and switched back to my fzf setup. I setup my history to ignore duplicates. Makes it also easier to use the up arrow key. But that can have downsides depending on the user. For me it works great for years. The only reason I don‘t share the history along with other dotfiles is the problem that some commands read API keys and token from the options. And I don‘t yet have a system for that so that. Because other means like env variables can make the it harder to repeat said command month later.
I use—and love—atuin. It beings the power of SQLite to your shell history with sync across multiple devices. It’s simply awesome and has helped me unravel historical install and configuration details that would otherwise be lost.
Bringing this advice up again because (unlike a lot of what I say) it’s actually something that I have personal experience with and I think it will greatly improve your life: https://news.ycombinator.com/item?id=33187749
My .zsh_history doesn't go back far enough, but I can see my stack overflow history from exactly 1731 days ago (the search is surprisingly nuanced, and allows filtering by user and date range right down to the day - to see yours just replace my user id with yours in the following url):
> This feature can also be be enabled in macOS by adding the following line to ~/.zshrc:
> setopt HIST_IGNORE_SPACE
Except that I think it's called `histignorespace` (I haven't caught up with when and how much zsh cares about lowercased names), I think this is just about zsh, not about macOS specifically.
Everyone's memory is better than mine: I can't even remember the command I typed, so I need to hunt it down from the context: current directory, date or previous commands.
Lines 11-21 can be ignored, they detect if the folder you were in got moved/deleted from another shell, to avoid the confusing behavior you get in that case.
zsh has a comprehensive hook system available for these types of tasks¹(zshaddhistory in this case). It gives you more options to control how/when the history is preserved, and allows you to be more selective in when it should be written.
Possibly worth noting for others that you'd want to think this through if you're using any of the history eliding options(hist_ignore_space² for example), as one history file may contain secrets when you're really expecting that it wouldn't.
There is also a better interface to work with hook functions through the add-zsh-hook mechanism³, which allows stacking multiple hook functions together.
The log files are noisy with ANSI codes, but they can be stripped, and I only refer to the logs extremely rarely (but when I do, it's extremely useful).
With a bit of prompt hacking I send information (shell session, pwd, command, timestamp) to a fifo, which is read by small Tcl script to ingest in a sqlite database. Aturin seems nice, probably does something similar.
It's a hacky script but the idea is sound. Good thing about SQL is that I can fine tune the query before feeding to fzf. A combination of order by recency and frequency of commands with and without context (pwd) works very well for me. With proper indexing I am not close to feeling slowdown, but if that happens I can probably just export data to duckdb and use that.
I've been doing something similar for years. Originally I tried to keep everything in .bash_history, but over the years and different machines it inevitably gets wiped sooner or later.
But the long-term (multi-machine) solution is to keep another append-only copy, one per machine:
function storehist() {
# Bug in mac version of bash stops HISTCMD getting set in function called from PROMPT_COMMAND...
CMDCNT=`history 1 | sed -e 's/\w* *\(.*\)/\1/'`;
if [ "$CMDCNT" != "$LASTCMDIND" ] && [ -n "$LASTCMDIND" ] # First time, no prev, do not log random
then
DATE=`date '+%Y%b%d %H%M'`;
if [ "$LASTCMDIND" != "$CMDCNT" ]
then
echo "$DATE $HOSTNAME:$BASHTTY $CMDCNT" >>~/.custom_history;
fi
LASTCMDIND="$CMDCNT";
fi
LASTCMDIND="$CMDCNT";
}
export PROMPT_COMMAND='storehist; ....
A little bit over the top and crusty but it has worked like that for years so I've not updated it in a while. There is a separate sync script that puts the .custom_history from each machine / container into a folder on each machine with the orginating machine name in the file-name.
This keeps ^-R working over a long period of time, but eventually that stops being a good way of find obscure commands from long ago and then something like this works:
The overall cost of running some script on each prompt is negliable on a modern machine and as the OP notes the disk space does not amount to much. It becomes very useful over time. I have about 20MB of shell history that stretches back to '06. It is one of things that is easy to setup, and yet the value accrues steadily over time. Across the dozen or so machines involved I doubt that the archive would have survived long-term without the step to export it outside of the shell's real history file.
It seems my history preceeds "recorded time", because timestamps start in 2016, which is probably when I discovered HISTTIMEFORMAT. 110k commands btw
the "exec env*" is because emacs tramp uses that when remotely logging into a machine and fill the logs with junk.
The fist thing I do on any machine is to set HISTSIZE and HISTFILESIZE to large numbers. It is wonderful to refer back to how you set up a machine from scratch for instance.
How did I set up this. what did I set up. What packages did I install.
I've always kind of wanted to create some sort of "annotated history" program. The idea would be, to ask you what you are doing (maybe in a separate window to the side) so you could properly document something fiddly.
Perhaps an edge case, but what about all the commands you mistype or use the wrong flags for etc. If I want to recall a niche or complex command I used months ago I probably won't remember which of the 10 results that pop up was the right one.
learning about ctrl+r on zsh feels like magic for me -- I'm not a heavy terminal user but I am pretty comfortable with it, and like the author, I have a lot of common commands with specific flags I need to add depending on if I'm doing a push to a local repo or a public one, etc.
I like features like this that build off of how _i_ use the shell, and it's up to me to figure out how I want to use it, and it's so simple but it makes using the shell all that much more efficient and it's easier for me to express my intent to the shell. just very cool, like learning a bit of the coreutils to make every day stuff in system administration easier.
and then freak out when their browser crashes and trashes the session. If you're that dependent on it you should probably add one of the many tab extensions that will save your list of open tabs to a backup every few minutes.
I do this. I use sidebery which creates snapshots of all open tabs. I usually have between 100 and ~600 tabs open. Once a month I go through the list and close those that are not relevant anymore (can easily select and close multiple tabs in sidebery).
It works pretty well for me, it often reminds me of things I was working on but haven't yet finished.
Since I have so many tabs, I also created a firefox extension to pin tabs on the right, so that they are close to the "new tab" button.
In my experience, the default shell history settings don't work well with multi-shell environments such as tmux. What I have done is in ~/.bash_profile, I create a subdir of ~/.historydirs and put in each of my history files (HISTFILE, LESSHISTFILE, PY_HISTFILE, etc.) which will be unique for a given tmux pane. This means that I can not only search for an exact command, but I can also get useful contextual information related (e.g. What folder did I last cd to?)
Just a general recommendation from someone who keeps a long history: if your shell loads a histfile, don't let it grow unbounded.
It isn't massive, but there's real shell startup latency associated with loading lines from your histfile. A few hundred isn't really noteworth, but tens and eventually hundreds of thousands of lines can add up.
(In my case, i store them in a database and I synthesize histfiles from the db. Keeping the permanent storage in a separate file also probably works.)
Totally agree with this. I use https://github.com/larkery/zsh-histdb slightly modified to work more smoothly for me. If I remember correctly, I tried Atuin but it messed up multi-line commands. Zsh-histdb handles them well.
Can fzf or the shell history file answer this question? zsh records timestamps and runtime if extended history is enabled. bash needs HISTTIMEFORMAT to be configured properly. I haven't used fzf so not sure if it can search by timestamp, I only see a line number for the matching command (presumably) in the gif.
Here’s the ZSH configuration I used for many, many years:
possibly cannot fully answer the question as there are settings to scrub duplicate commands ... meaning the target day will show commands unique to that day but not necessarily all commands from that day.
It's a small quibble but potentially an issue for those that really need to Indiana Jones their command history for some specific clue.
I agree, especially if the focus is not on the command itself but the date. Atuin has a CLI switch for filtering command history by date, but I don't see any support in the ctrl-r UI eg. with an operator like "before:" or similar.
In my own case large clumps of similar commands hinge upon what was the CD <current directory> at the time ... and the command most likely in many cases to be scrubbed as duplicate is a cd <one of 50 common working base directories>.
This leads to "I can see I tidied up a bunch of project files first monday of last month ... but which project was that??"
A nice thing that zsh (and bash-preexec) has is that you can add arbitrary functions to be called on command execution.
I work at a large facility with a shared filesystem and many computers, so often I need to answer questions like “What did I run the last time I was sitting at computer X”. I log: command, timestamp, PID (to separate terminal streams), hostname and cwd.
I’ve set up ctrl-r to search this “.fullhistory” file instead.
if [[ -n ${ZSH_VERSION-} ]]; then
# ZSH doesn't split command over multiple variables
preexec_custom_history() {
echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $1" >> "$CUSTOM_HISTORY_FILE"
}
else
preexec_custom_history() {
echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $*" >> "$CUSTOM_HISTORY_FILE"
}
fi
# Add it to the array of functions to be invoked each time.
preexec_functions+=(preexec_custom_history)
It's harder to search for multiline but I haven't gotten to fixing that yet. I've thought about putting it into sqlite (pwd makes the file large) but since everything is on network filesystems I don't think that's a place sqlite has any guarantees; screwing up a single line of a bare log doesn't break anything.
Edit: The fzf is kind of hacked together but is here: https://pastebin.com/sTxsvZAf . It has a few bugs but nothing annoying enough to have made me fix it.
Atuin (the tool mentioned later in the post) stores timestamps and CWDs for commands in a database, which is probably why the title is what it is.
I've been using it for 6+ months and it's actually good. I also like having an sqlite database with my commands so I can search it even more freeform if I want.
A little known secret is that you can actually search not only backward (ctrl+r), but also forward (ctrl+s). The problem is ctrl+s is bound by default to (obscure terminal thing) that stops your terminal from outputting anything. This can be disabled with stty stop ^-, and then ctrl+s will work as intended.
I recently reinstalled my personal laptop to distro hop. the only two files I migrated over were my pgp key and .bash_history, the bash history is around 8000 lines I don’t want to loose.
sometimes I ‘sort | uniq’ the bash history to filter about duplicates
My setup is so sporadic it would be hard to remember to save shell history. Is there any way to upload this to a server automatically across multiple computers?
The other day there was another post about a bash history in a db, years worth and searchable and available on all machines.
I do not get this. I have over 30 years certainly cobbled some complicated commands, but I never need an ancient one.
If it's complicated and used more than one day, it's a script.
Complicated or not and used a long time ago, it's no longer valid.
These days "so old it's no longer valid" is barely a year. Back in the sco Unix days you could have 10 year old commands that are still valid references, but Linux never had that except for trivial things.
There are exceptions but they are exceptions and so by definition not reasonable to focus on.
I think that building essentially mini apps right on the command line and then carrying around years of shell history to reuse them is solving a problem the wrong way. A stupid and backwards way.
It's losing sight of what the problem actually is and like growing one weird finger muscle to the size of a leg instead of realizing maybe you're not supposed to still be using a finger for that job.
The author mentioned this objection and suggested that since the cost of keeping a long history is so low, there’s really no reason not to. Additionally, you may not know a command will be useful in the future at the time you execute it.
I suppose everyone’s workflows are different though. I keep a long history but rarely use it.
It's useful for several other things. Automatic logging is one. Another: I compute the commands run most often (candidates for automation).
> These days "so old it's no longer valid" is barely a year
I don't think this is true (generally). At my work, we use old boring technology for most things (as we should). I don't see how it could be true enough to warrant not keeping a command line log.
I do a lot of analyses as a series of shell commands. Sometimes the command is complex, other times it isn't but the specific arguments I'm using matter a lot. When I need to revisit an old analysis being able to see the commands I was running is incredibly useful for picking up where I left off or answering people's questions about exactly what I did.
These are rarely commands so complex that they represent serious programming effort, but that's not why I want to save them.
> If it's complicated and used more than one day, it's a script.
That was a great solution prior to tools like fzf. It's more overhead creating such a script than using the OP's solution.
In ye old days, I would sort/organize bookmarks. Bookmark search solved that problem. And for a long while, Google solved the problem of even having bookmarks.
Same idea here. Why go through the trouble of making a script (and remembering its name), when you can find the command within seconds?
Predictive search/auto completion is incredibly useful.
In the browser context, auto completion search from history in the search/URL bar hugely reduces the need to even make bookmarks for top level sites. It's just much easier to type the first few characters of the site I want then hit enter.
I have a very common use case. I have had old machines that I have had to recreate or upgrade from scrach, and the shell history is instrumental in supporting this.
In ~/.inputrc: