I use a simple prompt_command hook to push my history (from all my machines) to a central MySQL database(https://github.com/fredsmith/history-db). Entries are de-duped and a tally is kept so I can do simple ad-hoc reporting:
It's pretty useful (I search through my history constantly with ctrl-R or using bash's history-search-backward hotkeys), and having a 56K line history file doesn't seem to impact history searching or shell startup any noticeable amount.
I agree with the general idea that contributing to these projects is good. However, the link is slightly misleading:
"GNU programs are used billions of time every day all over the world. However, the intellectual property owners, the Free Software Foundation, don't make a cent from it. What a raw deal! This project aims to fix that by monetizing many popular command line tools that GNU maintains."
Maybe true, but keep in mind that the main coreutils maintainers (including the parent) are funded by Red Hat. There are plenty of FOSS projects that do suffer from a lack of funding; coreutils, GCC, etc are certainly not one of them. As such, if your goal is to help the FOSS community at large, you might want to consider channeling the money to other important projects.
Maybe the donation link already does such channeling; I am curious about this.
It's a joke that's very much not obviously a joke. This is social commentary about the state of free software and/or Hacker News, I'm just not sure what the social commentary actually is.
This is just from my work machine (~4 years old), my history is not centralized but after seeing all this I think I'm going to work it into my centralized dotfiles system.
Also, I can confirm that gigantic bash histories don't affect the speed of anything.
erasedups will erase any earlier instance of an indentical command, keeping your history file lean. Setting HISTSIZE and HISTFILESIZE to something nonnumeric makes for no limits. histappend appends to the HISTFILE when the shell session exits. And my prompt command appends each executed command to HISTFILE immediately after it's executed. With histappend, this may be redundant. (The prompt command also gives me a blank line after each command finishes.)
That mysql setup is pretty cool too, it's a rich man's
$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head -10
14365 vi
11006 sudo
6976 ls
6898 svn
5322 cd
3869 cdb
2809 svnvimdiff
1846 ggrep
1474 git
1285 ssh
Stuff like swiftkey automatically recommends next words, it might not be that difficult to have a similar system for a shell. Sounds like a fun idea actually.
FYI this has been a side project of mine for a while. I call it Bashhub. Check it out if you're interested! Bash history in the cloud. Indexed and searchable.
this must be some kind of record, in 15 words you took something great, started adding unnecessary, creeping features, and ended up with bloatware no one wants to use anymore :)
It seems all you do is git pull, push and check disk usage!! Do you really check disk usage almost as often as you do git pull and push? You do push and pull more than commits?!! I cant understand what is that you do if you dont use other commands like grep, find, vi, top as often?
It's not bullet-proof, but since I never manually touch the directory it's worked without hiccup for the last ~6 months. When trying to remember the syntax to a rarely used function (or even the name itself) or coming up with a gnarly awk/sed/grep command, it's been incredibly helpful to have fairly granular search capability to go back and find them weeks later.
Also ctrl+g in a terminal toggles between per directory history and global history which makes context switching easier when switching to a project not touched for a while.
I recently set up my bash history to collect into an sqlite database. Besides the exact command it records the working directory, return value, starting time, ending time, a shell and a login session id. It doesn't handle background jobs well right now though.
Do you have any links to resources you used to build it or better yet a link to your implementation?
I've been pretty happy with my bash history solution, but thinking about the return value, one big thing I think would be handy would be to have a default filter for searches that removes commands that returned an error code. They might be handy to see every once in a while, but for the most part, I wouldn't want to re-run one of them.
I was interested myself too so had a quick search. There are a couple of implementations of the idea out there, https://code.google.com/p/advanced-shell-history/ looks the most promising of the ones I looked at. I might have to play with it later.
It was mainly for an exercise to get familiar with sqlite. Maybe later very rarely I can use the session data (hunting down sessions where you only remember some irrelevant command). Getting the duration of some commands is useful like long running simulations, compilations. It's especially handy where you don't expect it to be too long and forget to time it beforehand.
Having said that I didn't use it too much yet, I have my regular history in place too.
I maintain almost all my bash history and I often use it to reconstruct a sequence of events in the past when I feel like maybe a mistake had been made.
I guess the downside of this is that ctrl+r would only search the current day's history? Personally I at least increased the history kept with HISTFILESIZE=500000 and HISTSIZE=5000
Another issue is that it looks like would start a new history file only when a new shell is started. Some people log in fresh every day, but I've got terminal windows on my home machine that have been sitting around for many months...
I guess you'd have to set the history-size etc large enough to hopefully last for the maximum expected shell run-time.
EDIT: hmm, the man page says you can just unset HISTFILESIZE or set it to a non-numeric/negative value to avoid truncating the history file at all.
It can also be nice to use HISTCONTROL=ignorespace
If you're planning on keeping a lot of history, ignorespace makes it easy to avoid inserting commands with sensitive info. A well-placed $(cat) can work too.
$ PASSWORD=hunter2 # sensitive info doesn't go into history
$ curl -O http://name:$PASSWORD@server/path # command with correct syntax does go into history
You can still use ctrl+r, but you need to provide the history file in one piece, using "history -r <file>" command. I do it in the following way, maybe it can serve you too:
cat `find "${HOME}/.history" -type f | grep -v ".allhistory"` > ${HOME}/.history/.allhistory
history -r ${HOME}/.history/.allhistory
I keep a pretty long bash history, but I don't have it unified among machines or anything.
Others have mentioned several handy tweaks for doing this, but I didn't see all of the ones I use so I thought I'd share them here. One of the important bits is the HISTIGNORE. I am interested in commands that I might want to search for and run again one day, so I filter out commands I consider to be clutter.
# Configure my history preferences
# Load history substitute into readline rather than immediately executing
shopt -s histverify histreedit
set histappend
# don't put duplicate lines or lines with leading spaces in the history. See bash(1) for more options
export HISTCONTROL=ignoreboth
export HISTIGNORE='&:bg:fg:cd*:clear:ls:pwd:history:exit:make*:* --help:'
export HISTTIMEFORMAT="%m/%d/%y - %H:%M:%S "
I love all of these new Go tools for managing your *nix systems.
I started using Homemaker recently https://github.com/FooSoft/homemaker which is a very clean/lightweight replacement for GNU Stow, Homesick, and other dotfile managers. The Go versions of these utilities are always simple and single-purpose.
Modern developers are returning to building Unix style systems programs. Time is being spent on that instead of the usual bulky python scripts that have been popular recently or the million personal-productivity web apps people were creating.
ctrl-r (find something) enter (execute)
ctrl-r (find something I have to change) home/end (change, then enter)
with fzf ctrl-r and selecting always gives me just a string where I then have to press enter, it just behaves like dmenu for the console basically if you look at the bash bindings, which unfortunately is a bit less flexible unless I am missing some option that would allow this
Here's something interesting - We collected bash history of around 15000 users and listed frequented used commands for fun :) http://www.webminal.org/fulc/
Kind of interesting, might be more interesting to see what the other end of the distribution looked like, i.e. the really esoteric commands that people rarely use. We all might learn something from that!
We have common mistakes list http://www.webminal.org/common_mistakes/ but didn't figure-out something like which is the least-used valid command. Good idea, we will dig deeper into their bash_history :)
You should also analyse commands that were run without 'sudo'. For me, using 'apt-get install' without sudo has been the most frequent mistake I have made. Luckily, 'sudo !!' saves me every time.
Thanks so much for creating autojump! autojump + Crl+R are saving me so much time that would be lost searching around.
There is just this impedance mismatch between our brains hazy recollection of things and the exactness that is required by computers. I find that fuzzy search combined with stats like most-commonly-used bridge this quite well!
The value is in when you remember you did something before, but not exactly how. So you just grep or scan through your history, and the files you edited / commands you ran are likely to refresh your memory even if they don't exactly capture the entire workflow.
Also means you don't have to retype complicated commands too often.
Practical example: I remember I cleaned up some calibre converted epubs with perl a couple months ago, because they had a piece of text injected by some kind of generator on every page. It took me some time to figure out the exact replacement patterns, and if I were to do it again, I just need to do:
In practice I've had better luck with shell history than SSH keys when exploiting arbitrary file reads. The history tends to reveal interesting locations.
bash history had never seemed to work for me between tmux windows. Commands are 'trapped' in their respective window and the last one to close 'wins' and writes its history over the others.
So I end-up with never-closing task-based window sessions and occasionally copy-paste commands to a safe file for future reference.
I really should investigate the root cause some day...
Some people dislike it, but I LOVE it. I like being able to run a command, open a new terminal, and be able to control-r to find that command again. It ensures that I don't have race conditions over which terminal writes its history to disk first, as well, so I lost far fewer (zero?) history elements.
You've conflated terminal and shell there. It's an important distinction because, in fact, there's still one race condition with history files, even if you only have /one/ shell running. It's down to the Bourne Again shell not doing atomic updates when it rewrites the history file. It's one of two bugs, the other being systemd sending multiple signals in very quick succession to kill a terminal login session, that in combination cause the effect discussed at https://news.ycombinator.com/item?id=10151861 . Far from losing zero or a few history elements, it can in fact result in your losing all of your history.
Also, if you want fresh history in an already opened window (assuming you added the above), do "history -n". E.g. if you typed "longcommand" in window1, then switch to window2, do "history -n" and uparrow, and "longcommand" will be there.
This is IMHO the real benefit to zsh over bash. You can set zsh to share history between sessions and they are interleaved based on the time the commands were run. No more closing a session in the wrong order and losing half of your history for the day.
A much better system would be to have a user daemon centralize history (among other things). The benefits would be numerous:
- Better performance
- (optional) Sync between concurrent shells
- No history overwriting problems / race conditions
In addition, this daemon could be used to enable other programs such as autojump. These programs work with the current system, but are based on a hack and as such have needlessly poor performance.
It's not just the history feature that's archaic. Bash itself is quite the abomination, also operating on bytestreams is really error-prone, and then there is the whole tty system..
As much as I like my command line, I think it would be very worthwile to rethink this whole system from scratch.
Did not know that, for me HISTCONTROL=ignoreboth is set in my .bashrc. I think Debian's adduser creates a default .bashrc file where this line is present, I'm not sure though.
I believe that command-line parameters are available for all to see in the output of ps, so typing parameters on the command-line is a bad idea anyway.
I think it's better to simply learn the commands than to have a tremendous amount of history logged. If you have to type a lot of command parameters a lot you can always automate that with aliases anyway.
I have <M-p> and <M-n> bound to history-search-backward and history-search-forward respectively, so
ls <M-p>
inserts the most recent command in history that begins with 'ls ' in the command prompt. I don't know why this isn't the default, as it's extremely handy. Also it works with any CLI program that uses readline.
If you want to try it out, add this to your ~/.inputrc file.
I use fzf https://github.com/junegunn/fzf combined with bash and zsh for doing some fuzzy completion of history when pressing CTRL-R. Also opens the list in a tmux pane if it's installed. Plus, it's very useful in Vim for opening files in the current buffer.
· --merge immediately incorporates history changes from other sessions.
Ordinarily fish ignores history changes from sessions started after
the current one. This command applies those changes immediately.
Well if it includes seconds, the likely hood of me typing something on a machine, switching to another and running something with the same pid within a second, is pretty low.
Mostly because I can't switch to another machine within 1s
It includes hostname too. You can only get a collision if you do all of the following within one second:
1. Start an interactive shell.
2. Enter a command you want saved.
3. Exit this shell.
4. Start another shell.
5. Be unlucky such that the new shell has the same PID as the previous one.
I do something like this in PowerShell too. Forgetting handy one-liners by not realizing they were important, or losing them when Windows restarted were the impetuses for me to start.
Why even store your bash history? It's like you store everything you ever said... Would you lose your precious time to go through the old stuff? I wouldn't.
I'm a computational biologist. I often need to know exactly how some piece of analysis was done months or years ago. In an ideal world all the details would be in scripts with carefully written READMEs cross-linked from my lab notebook, but it's not an ideal world and sometimes we're in a hurry. In those cases the history saves my bacon.
I'm not a computational biologist, but I've run into this once or twice. I only use the shell for quick summary statistics now though. I started doing most of my analysis in R, which made life easier on a lot of different levels.
Try binding the <UP> and <DOWN> keys (or whatever keys you want) in you ~/.inputrc to the readline commands "history-search-backward" and "history-search-forward". See bash(1) in the READLINE section details.
Instead of simply scrolling through your recent bash history when you press <UP> (which still works the same when your prompt is empty), it filters out any command that don't match the partial command you've already typed. For example, typing "cp " and then pressing up will find the last "cp" command that I typed.
