'!!:n' selects the nth argument of the last command, and '!$' the last arg
A lot of people know about "!$" (which is shorthand for !!:$), but that's just the tip of Bash's history expansion. I use these things all the time. One of my favorite keystroke savers is adding :h, the head modifier, to !$. For example:
Once you understand how each component works it's easier to put them together into new (to you) combinations. For example, once you know that !$ is shorthand for !!:$, it's not a huge leap to reason out that you can use !-2:$ to get the last argument to the 2nd-to-last command. Or !ls:$ for the last arg to the most recent `ls` command.
I also prefer to do substitution with the :s modifier rather than ^ as suggested at the link, for consistency's sake:
$ echo "foo bar"
foo bar
$ echo !!:s/bar/baz
foo baz
$ echo !?bar?:s/foo/qux
qux bar
Another nice ending is `:p` to print the command instead of executing it. I use this if I'm doing something complicated and I want to make sure it's right. Or if I'm saying `!-n:foo` with n>2. Then just up-arrow and enter to run it for real.
Here is a detail when using '!!:n' with ':p' when trying to iteratively construct a complex command. I want to emphasize the use of the up-arrow (which will show the interpolated arguments), as opposed to rewriting exactly what you wrote in the previous command, since the usage of ':p' will be interpreted by the shell as a command in itself:
$ echo a b c d
a b c d
$ echo !!:2:p
echo b
$ echo !!:2
-bash: :2: bad word specifier # there was only 1 argument in last command
However:
$ echo a b c d
a b c d
$ echo !!:2:p
echo b
$
<up-arrow pressed once will give the following prompt>
$ echo b
So you say cd M-. M-backspace M-backspace instead of cd M-.. Four keystrokes is still an improvement over shift-1 shift-4 shift-; h, which is five, plus you get to see where you're going to go before you get there.
You're incorrect. The number of M-backspace you appear to need would seem to depend on how many `little' words comprise the part that needs deleting, e.g. foo/2013-03-13 needed three M-backspace to rid me of the date and a further backspace to remove the trailing slash, which isn't insignificant to all commands, e.g. ls -ld bar/ when bar is a symlink.
In comparison. !$:h understands its task at a higher level. And thanks to key rollover, typing different characters, like !$:h, is quicker than tapping away at . until visual feedback, which may lag, tells me I've done enough.
Using ssh-agent also means you can practically put a very large pass phrase on your SSH key, because you'll only type it infrequently. Good luck brute forcing my passphrase.
I'm pretty sure ssh-agent doesn't store the password, but the private key. Also, the fact that it supports timed expire (and can be setup to drop keys upon events such as screen lock) make it a wiser choice than passwordless keys.
That's correct. And ssh-agent doesn't give access to the private key either, only to perform operations like signing. The only way to extract the key is to search it in the process memory, which I believe would requires root level access.
re: 4: not only is an ssh agent by far safer, but most agents now allow you to set a timeout on a key, so it's not indefinitely saved in memory.
A passwordless key gives anyone with acces to that file, access to the login associated with it. If that file is inadvertently exposed (oops, checked it into github...), any machine you have a login on must be considered compromised.
ssh-agent and 'ssh -A' is also useful if you have to login to one machine to access another, without having to copy your private key to the first machine.
For example if you login remotely to a machine, and want to access a git repository on another:
I use agent forwarding often, but you still need to be careful, especially if you forward your agent to a machine not under your control. From the ssh man page:
Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.
Consider using a dedicated key for each of those circumstances, set sane defaults in your ~/.ssh/config on all machines, and be very careful about what ends up in any of your ~/.ssh/known_hosts files, as they provide a road map to other destinations.
The ssh_config HashKnownHosts option hashes the contents of the known_hosts file, making it intractable to get a list of hosts. But of course your shell history will still provide it.
3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?
The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff.
I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled it easily
I'm not sure. I haven't closely studied the difference between each algorithm. My guess would be that sort -u would perform better as the data set gets larger with a good block size setting because it does do an external sort. Cardinality would also affect the performance. If the unique set handily fits in memory, an external sort on a large data set wouldn't be very efficient.
Yes it will, and a bit (read: a lot) more than 10GB as it needs to store the contents of variable x in a hash table (with the corresponding hash key and value of the counter). There's no other magic way it can 'know' whether a particular line has been seen before. You can't rely on hash keys alone as the hashes aren't guaranteed to be unique.
For files with relatively few duplicates it's going to be a lot slower than sort | uniq.
Trying it on a 128MB file (nowhere near enough time to test a 10GB file) filled with lines of 7 random upper case characters[1] (so hardly any duplicates):-
$ wc -l x.out
16777216 x.out
$ time ( sort x.out | uniq ) | wc -l
16759719
real 0m17.982s
user 0m42.575s
sys 0m0.876s
$ time ( sort -u x.out ) | wc -l
16759719
real 0m20.582s
user 0m43.775s
sys 0m0.688s
Not much difference between "sort | uniq" and "sort -u".
As for the awk method:-
$ time awk '!x[$0]++' x.out | wc -l
has been running for more than 20 minutes and still hasn't returned. For that 128MB file the awk process is also using 650MB of memory (according to ps). Will check up on it later (have to go out now).
This Linux machine has ~16GB of memory so the file was going to be completed cached in memory before the first test. All things considered equal the awk method will be roughly O(n) (e.g. linear against file size) and sort/uniq will be O(n log n). So, theoretically, the awk method will eventually surpass the sort method because it's having to do less work (it's only checking for a previously seen key rather than sorting the entire file) but I'm not sure the crossover will be anywhere useful if the file doesn't contain many duplicates.
Repeating it for a file containing lots of duplicates (same 128MB file size but contents are only the 7 letter words consisting of A or B, so only 128 possible entries):-
$ time awk '!x[$0]++' y.out | wc -l
128
real 0m1.207s
user 0m1.192s
sys 0m0.016s
$ time ( sort y.out | uniq ) | wc -l
128
real 0m14.320s
user 0m31.414s
sys 0m0.428s
$ time ( sort -u y.out | uniq ) | wc -l
128
real 0m12.638s
user 0m30.366s
sys 0m0.188s
Notice that "sort -u" doesn't do anything clever for files with lots of duplicates.
So awk is much faster for files with lots of duplicates. No great surprises. When I get a chance I'll repeat it for a 1GB file and a 10GB file (with lots of duplicates otherwise the awk version will take far too long).
Yes it will ... [need] to store the contents of variable x in a hash table [...] There's no other magic way it can 'know' whether a particular line has been seen before. You can't rely on hash keys alone as the hashes aren't guaranteed to be unique.
Technically that's true, but the result of a cryptographic hash like SHA-256 is (practically) guaranteed to be unique. Depending on the average length of an input line and how many of the lines are unique, storing only the SHA-256 hash value could take far less memory than storing the input lines along with a non-cryptographic 32-bit hash value.
You are describing a bad implementation of a bloom filter [1]. Anyway, people expect "uniq" to be correct in all cases (i.e., to never filter a unique line). A default implementation where it would possible (even with a minuscule chance) that this doesn't happen would be a recipe for disaster. It may be a cool option though ;)
No, this is not a bad implementation of a bloom filter, and the size of the minuscule chance matters; unless SHA-256 has a flaw in it that we don't know about, SHA-256 collisions are far less likely than undetected hardware errors in your computer. The universe contains roughly 2²⁶⁵ protons, 500 protons per distinct SHA-256 value, and has existed for roughly 2⁵⁸ seconds, which means there are roughly 2¹⁹⁸ SHA-256 values per second of the age of the universe.
Typical undetected bit error rates on hard disks are one error per 10¹⁴ bits, which is about 2⁴⁷. If your lines are about 64 characters long, you'll have an undetected bit error roughly every 2^(47-6-3) = 2³⁸ lines. SHA-256 will give you an undetected hash collision roughly every 2²⁵⁵ lines. That is, for every 2²¹⁷ disk errors, SHA-256 will introduce an additional error. If you're hashing a billion lines a second (2³⁰) then that will be 2^(217-30) = 2¹⁸⁷ seconds, while the disk is giving you an undetected bit error every minute or so. A year is about 2²⁵ seconds, so that's about 2¹⁶² years, about 10⁴⁹. By comparison, stars will cease to form in about 10¹⁴ years, all planets will be flung from their orbits around the burned-out remnants of stars by random gravitational perturbations in about 10¹⁵ years, the stellar remnants will cease to form cold, dark galaxies in about 10²⁰ years, and all protons will have decayed in about 10⁴⁰ years.
And if you somehow manage to keep running uniq on your very large file at a billion lines a second, in a mere 500 times the amount of time from the universe's birth to the time when nothing is left of matter but black holes, SHA-256 will have produced your first random false collision.
Another possibly relevant note: there are about 2¹⁴⁹ Planck times per second. None of the above takes into account the Landauer and related quantum-mechanical limits to computation, which may be more stringent.
1. Use zsh, not bash. AUTO_PUSHD, CORRECT_ALL and tons of other options make some tricks redundant. Also, the zle M-n and M-p are more useful than C-r imo.
4. Use cron, not at. Or even systemd timer units, if you're so inclined.
5. Use public-key authentication and keychain, not password-based SSH.
6. Don't send emails from command-line naively; you have no control over the headers. Use git-send-email or similar.
7. Consider using something slightly more sophisticated than Python's SimpleHTTPServer to share files/ folders. One example: woof (http://www.home.unix-ag.org/simon/woof.html)
I've given tmux two separate tries and both times it had screen corruption issues with curses apps. screen is tried (and tried, and tried) and true.
I've also invested enough time in learning bash over the years that zsh is not a net-win for me. I've switched to using it on some systems but I seldom use more than what's available in bash. I would switch back to bash on these systems for consistency, but I feel like I've already sunk too much time in to this experiment of using zsh.
don't be a zsh/tmux hipster; there's nothing wrong with "good enough". this lesson has played out several times as businesses / software with arguably better execution / implementation loses out to existing players that have been around a while.
Edit: though, if you're relatively new and haven't been using bash/screen/whatever for decades, I don't think anybody would call you a hipster for using zsh/tmux instead of bash/screen. The marginal utility of learning tmux or zsh is much higher for somebody who hasn't already used other stuff forever.
I take it you don't work on many disparate unix systems on a daily basis :) I find most of the time I'm lucky if there's even bash installed on the remote system, it's usually ksh. tmux? way too new. Python? Nope. Perl is the only scripting language I'd wage my balls on beyond awk if I hope to reuse the script again.
Sad, but I find this is generally the case in extremely large enterprises where there is a mix of AIX, HPUX, Linux and Solaris being used due to years of weird procurement decisions. Sigh.
So happy to see this on HN! Actually I took most of the tips from unix threads here and r/commandline, I highly recommend that subreddit if you like this kind of tricks!
An arguably better (and slightly more portable) way to accomplish this is the special variable $_ , which expands to the last argument of the previous command. Since $_ is a variable rather than a history substitution, it still works when there is no command history (e.g. in scripts) and allows all the usual variable expansion forms, for example ${_##*/} to extract the last path component.
That's good to know, although !$ is easier to type since you only need to depress the right shift key, whereas yours requires a quick shift on the opposite side. Makes a big difference when you're trying to quickly type "rm -rf !$" as root. :)
If you are on OS X and use Terminal.app “Alt-.” won’t work because Alt is used for alternate characters. You have two options: enable “use option as meta” in the app settings (but you lose the extra characters) or use “Esc-.” instead.
Tar can transfer more filetypes and attributes than scp can (even using -p option). `scp -p` only transfers mode, mtime and atime; you lose ownership, extended atributes, symlinks, and hardlinks.
You will also get better compression with tar (or rsync), as it is compressing the files directly, and not just the ssh stream (-C is just passed on to ssh).
In particular, scp is mindblowingly slow on lots of small files. I independently rediscovered the tar-pipe trick while sitting there watching scp laboriously copy thousands of 100-bytes so slowly I could count the files as they went by. That should not be possible, even at modem speeds. Fine for moving one file, OK for directories of very large files, not suitable for general usage where you might encounter a significant number of smaller files.
Absolutely. Connection latency hits you the hardest, since each file is sent serially, and requires 2 (or 3 with -p) round trips in the protocol, and this is on top of an ssh tunnel with it's own overhead. I can't remember what my tests showed, but I have this inkling feeling that tar over ssh was far faster than rsync for an initial load, since there's no round trips required, but you lose some of possible rsync benefits, like resume-ability and checksums.
If my first tar attempt fails for some reason, but it made a lot of progress, I switch to rsync. Best of both worlds. This hasn't come up often enough for me to script it.
Add pv (available in many standard repos these days, from http://www.ivarch.com/programs/pv.shtml if not) into the mix and you get a handy progress bar too.
and so forth will result in a throbber as it can't query the pipe for a length.
If you demoggify the first example to:
pv file | nc ...
you get a progress bar on the sending end without manually specifying a size.
Even without a proper % progress bar, the display can be useful as you can at least see the total sent so far (so if you know the approximate final size you can judge completeness in your head) and the current rate (so you can see it is progressing as expected (so you get some indication of a problem such as an unexpectedly slow network connection, other than it taking too long)).
Also, I find that if I'm going to copy the data once, I'm often going to copy it twice, or which to get a more up to date version of it at a later time. Rsync clearly wins in these cases.
Finally, from the compress flag on rsync:
Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection.
Unfortunately OSX uses a badly outdated openssh version just like many other unix tools and it doesn't support the -W option. You could try upgrading openssh using homebrew if you want.
Note that each job gets an identifier (which you can see by running `jobs`). Other commands like `kill` can work with the id number by using %[id]. For example:
$ some_command
^Z (hit control z)
$ some_command_2
^Z (hit control z)
$ jobs
[1]- Stopped some_command
[2]+ Stopped some_command_2
$ kill %1
$ jobs
[1]- Terminated: 15 some_command
[2]+ Stopped some_command_2
I have my terminal emulator configured to set the URGENT ICCCM hint when it sees a bell character. When I realize a command I just entered is going to take a while, and I want to go look at something else on another workspace, I do this:
$ alias b='echo -e "\a"'
$ long_running_thing
^Z
$ fg ; b
[long_running_thing resumes]
Then when the job completes, the terminal bell will ring and my window manager will get my attention.
You can also use "jobs" to see background and stopped tasks. If there are multiple jobs, you can use "fg 2" to foreground the second job on the list, etc.
Another nice job management tool is disown, which lets you log out of your session without killing the job (similar to starting the command with nohup)
And also, you can pass the -h flag to disown to leave the job in the job control table (i.e. you can still bring it to the foreground/suspend it/etc.), but still skip sending the SIGHUP on logout.
While we're at it, consider using weborf [1] as an alternative to Python's SimpleHTTPServer for simple file sharing. I found it able to saturate a gigabit ethernet connection when hosted on a Core 2 Duo ULV laptop with an SSD.
I will have to look at weborf. I have always wished debian packaged publicfile, similar to djbdns or even dbndns. As it is I am still looking for a "djbdns-like http server" that is apt-get installable and actively maintained.
I will never understand why gnome-user-share depends on apache...
Thanks for the suggestion. I used to have a very customized .bashrc with nice little things like that, but have decided to stick with standard stuff for things that work off of muscle memory.
I got tired of sshing into a new box and half the things I'd type wouldn't work properly until I remembered to copy my settings files over, which seemed more trouble than it was worth for a short-lived s3 instance.
That's why I was excited to discover ctrl-r. It's a built-in method of searching history that I can remember and it'll work everywhere.
I've got a public repo of my dotfiles, so the first thing I typically do is "git clone git@github.com:pavellishin/dotfiles.git && cd dotfiles && ./install.sh"
After that, I launch tmux, and it's all hunky dory.
Heh.. right after I posted that comment, my thought was: 'you know - the correct answer here would have been to make an Uber command that would suck all the configs in and install them'.
Thanks for giving me the push to do so. I think I'll take your suggestion but put the command on a site somewhere so I can simply 'curl https://foo.bar/configs | bash'
If you are using 'set -o vi', then you can just use 'ESC' -> '/<somePartOfCommand>' then use 'n' to iterate in reverse order through the commands that match the vi regex. really powerful and useful.
"Add "set -o vi" in your ~/.bashrc to make use the vi keybindings instead of the Emacs ones."
Better to do this kind of thing in .inputrc, as:
set editing-mode vi
(or set editing-mode emacs) because any application that uses readline gets to use those settings. So for example you get command line editing in various command line apps. bash uses readline, so you'll get that. The python repl will give command line editing with .inputrc set as above.
If you hit ^Z right now, it will tell the shell on otherhost to stop some_command and give you the shell prompt on otherhost back.
If instead you wanted to stop the ssh process and get the shell prompt on somehost, hit Tilde ^Z (you don't have to hit a new Return, but ssh only notices these escape sequences after a Return).
Also if you use ControlMaster and have a few xterms open all with sshs to otherhost, and then you exit the first ssh you happened to have open, it will seem to hang and not give you your prompt back. What's happening is that that ssh process is the "control master" and it's still open because you've got other sshs to the same host open. Hit Tilde & to background the ssh process and get your terminal back.
Yes, you could also hit Tilde ^Z and then 'bg' the ssh process.
Just to note, splits in tmux and screen behave differently. iirc, in screen you have a set of splits and fill them in with different ports (kind of how vim thinks of viewports). so technically you can have the same viewport open twice on your monitor and the other will mirror the workings of the one that your are working in right now. in tmux each 'tab'/window is a set of splits, which act more like sub-windows, and don't really share across multiple spaces.
If you happen to have a malicious directory named (without double quotes):
".;sudo rm -rf /"
You'd be stuffed.
It's better practice to use the '-print0' flag of find(1) and
pipe the result into xargs(1) with the '-0' flag set. For safety,
it's best to also use '-r #' for expected number of arguments,
'-n' to stop xargs from running without arguments, and "-J %" so
you can use quoting.
EDIT: The reason being the shell is never involved in this process, and the shell is what is responsible for splitting commands on semicolons/newlines.
You are assuming the exact versions of the shell and
find programs that you use are the only ones that exist. It
may not be a problem on your exact system, but it can
be a problem elsewhere.
Sorry, I didn't see your 'EDIT' caveat when I responded --now that will
teach me to reply too soon. ;-)
Also, it seems I failed to be clear; I'm probably too tired I suppose.
My point was there is plenty of ancient and buggy code out there.
It could be "most", or even "many" modern unix variants have fixed a
lot of the old bugs in find(1), but if you don't have the luxury of
working on a current system, and your not allowed to upgrade it, then
plenty bad things can happen due to invoking a shell, handling space,
quote, and delimiter characters, and so forth.
reference:
$ uname -a
OpenBSD alien.foo.test 5.1 GENERIC.MP#207 amd64
setup:
$ mkdir test
$ cd test
$ touch file1
$ touch file2
$ touch file3
$ mkdir ';ls'
bad:
$ find . -type d -exec sh -c {} \;
sh: ./: cannot execute - Is a directory
;ls file1 file2 file3 test.sh
better:
$ find . -type d -exec sh -ec {} \;
sh: ./: cannot execute - Is a directory
also bad:
$ find . -type d -print0 | xargs -0 -r -n 1 -J % sh -c "%"
sh: ./: cannot execute - Is a directory
;ls file1 file2 file3 test.sh
better:
$ find . -type d -print0 | xargs -0 -r -n 1 -x -J % sh -ec "%"
sh: ./: cannot execute - Is a directory
better;
$ find . -type d -print0 | xargs -0 -r -J % sh -c "%"
best:
$ find . -type d -print0 | xargs -0 -r -J % sh -ec "%"
POSIX is all great and wonderful in theory, but in practice it's no
different than the bogus Java "write once, run anywhere" claim. If a
system or utility claims to be POSIX compliant, then you're probably
close, but you'll still need to do testing and debugging.
At least some of the issues with find/xargs are mentioned in the
following wikipedia article. It's probably more clear than I am right
now.
The contrived examples you've shown aren't examples of POSIX-incompatibility, or bugs in `find` at all. You've explicitly involved the shell. Of course trying to run every directory name as a shell command string is going to result in executed code!
Your original argument was that given:
find . -type d -exec chmod g+x {} ';'
It is possible to force code execution of arbitrary commands given a carefully crafted directory name. The key difference in this case is that the shell is not involved _at all_. I challenge you to find an implementation of `find` that is broken in this way.
As a side note, it is even possible to involve the shell in the picture in a safe way with `find`, without the use of `xargs` (and thus avoid the overhead of setting up a pipeline):
find . -type d -exec sh -c 'chmod g+x "$1"' _ {} ';'
(my contrived example is quite poor, though, since it does nothing but introduce unnecessary shell overhead)
Modern (POSIX > 2001?) `find`'s support `-exec {} +` which further reduces the number of reasons to invoke `xargs`:
find . -type d -exec sh -c '
for x; do
do_foo "$x"
do_bar "$x"
do_baz "$x"
done
' _ {} +
(example above shows how to make proper use of this feature with an explicit shell invocation)
It's not a problem with GNU or OpenBSD `find`, and I'm pretty sure that's the case for FreeBSD too. What version of find uses system(3) instead of execv(3)?
NFS basically breaks the client computer until the host responds, while SMB detects the I/O error earlier and, at least, doesn't hang the terminal.
Other than that, SMB allows for permissions and a bunch of other features. It is, basically, a more modern and robust protocol. Does NFS shine for some use cases? Indeed. Would it be the first choice for most ones? Nope.
Disclaimer: my sysadmin skills are not so great, I'm talking as a user
Soft mounts report errors immediately, hard mounts hang.
And for permissions, NFS provides everything under the sun that you could possibly need via ACLs. NFSv4 is a very modern protocol, much as SMBv2 is (not SMB though, it's awful).
Linux has had NFSv4 for what, 6 years now, at least, and even v2 and v3 had some limited ACL support?
That's nice to know, it seems that we have the "hard" configuration at the lab and it's a real pain. Backwards compatibility, you know. For my mini-cluster we use SMB and couldn't be happier.
1) The caveats of NFS can be crippling if you are a rubbish sysadmin. NFS requires a more thorough understanding than what you can get from a tip sheet.
2) Samba is single threaded, performance will suffer serving SMB from a Linux machine. For this reason it would be better to serve SMB from a Windows machine.
3) Using a foreign protocol between homogenous computers when the native protocol will do is non-ideal. NFS is the right thing when sharing filesystems from Linux to Linux.
If you use any NAS device with a weak CPU and you have a choice of CIFS or NFS, the NFS transfer rates are normally 25% faster and the load is much less on the server.
I believe NFS is a less chatty protocol with fewer roundtrips. SMB is not a particularly well-designed protocol overall.
My only evidence is anecdotal. Back in ~1998 we had Samba running on our small Linux file server using Windows NT desktops via 10mbit Ethernet. It was dog slow, not just for browsing but on sequential things like file transfer. We installed NFS mounts on the same box, and it turned out to be lightning fast. I don't remember just how fast, but it made everyone go "wow".
Nowadays, with NFS over fast 100mbps or gigabit Ethernet the latency difference probably is not significant enough to make a difference. I prefer NFS just because its more Unixy.
I third this. I was attempting to set up my Raspberry Pi for sharing a big HDD that was plugged into it on my home network, and the NFS mounts appear to weigh lighter on my vanilla Raspbian setup.
And fwiw, if there's some sort of error with the mount, the terminal lets me know right away.
I freaked out when I found out about the "screen" command several years back. "screen" starts a virtual screen that you can detach from with "ctrl-a d" and you can log out, login from a different machine/session and reattach with "screen -r". it has history so you can run long running commands and reattach 3 days later to continue from where you left off as if you had been logged in the whole time.
Tmux is the new screen. tmux allows for emacs or vim key bindings to run around the buffer and search, splits horizontal and vertical and easier configuration: http://tmux.sourceforge.net/
Yup! You might want to try tmux, though. I used screen for many years and switched over to tmux about 2 years ago, and have been really happy with it. It (tmux) is also maintained more regularly now; screen development seems to have stagnated.
This was a nice very timely page sitting with my private repo in mercurial, and the other one at github…
I discovered help <builtin> some months ago, and that was a great boon really. Like help "test" so I didn't have to go the rather large bash man page.
Here is a little shellscript for displaying a man page on Mac Os X (gman). (If you then click on on of the links on the man-page, it may pop up in your default browser).
#!/bin/bash
if [ $# -lt 1 ] ; then
echo "gman takes a man page, if found and formats it into html."
echo "Usage: gman [manfile]"
exit 2
fi
a=`man -aw $* |head -1`
if test x$a = x ; then
echo "Can't find $1"
exit 1
fi
# Figures out if it is a normal man page or something else (gz).
b=`man -aw $* |head -1 |grep "gz"`
echo $b
if test x$b = x ; then
groff -man $a -Thtml >|/tmp/tmp.html
else
gzcat $b |groff -man -Thtml >|/tmp/tmp.html
fi
qlmanage -p /tmp/tmp.html >/dev/null 2&>1
When creating excessively long oneliners (you know, the kind that should actually be a script, because you know you're going to find a use for it in a few weeks), the following key combo is golden:
^x ^e
It opens up the exported EDITOR with a tmp file containing whatever is on the command line.
Using SSH, especially on campus/in a train/other places where a wifi connection doesn't long, mosh is really a godsend. In places where mosh isn't practical or available, the following combos are really good to know:
<RETURN> ~ . # end ssh connection
<RETURN> ~ ? # show available commands
Paired with autossh which will reconnect by itself, it really takes the pain out of traveling while doing remote work.
Oh, and when you need to know what the decimal value of 0x65433 is, it's good to know that bash can do stuff like that:
$ echo $((16#65433))
414771
Reading the bash man page is not a bad idea in itself...
'ssh -R 12345:localhost:22 server.com "sleep 1000; exit"' forwards server.com's port 12345 to your local ssh port, even if you machine is not externally visible on the net.
But the remote host can limit your ServerAliveInterval, however most hosts don't close your session if there is something running. On our clusters this is the only working solution, and I tried both, trust me.
If that's an issue (I've never met a server like that, not to mention one like that and I couldn't change the setting) then wouldn't you rather use this?
while [ true ]; do sleep 1000; done
Just using:
sleep 1000; exit
means you can't make new connections after 17 minutes.
Yes, in fact I use the first one (while true; sleep; ls) but I thought the second is more succint. Anyone interested can make a loop. Please notice that most of the snippets aren't meant to be copied & pasted, but rather analyzed and understood by the user.
Ah, sometimes that's a hard line to draw since often the people that know enough to understand not to take it literally don't really need the pointer in the first place. :)
Bash and zsh support a surprising amount of emacs editing functionality. Cursor navigation (M-b, M-f, C-a, C-e), text selection (C-space), copy/paste (C-w, M-w, C-y, M-y), and undo (C-_).
Learn to use your shell's globbing features instead of overusing find. In zsh, you can do 'print -l /*.(c|cc|h|hh)' for example (I'm sure bash has an equivalent).
I actually hit a limit quite a few times already - I don't remember if it was the shell that complained or the command (mv, cp, ...) itself, but I know I couldn't execute the command. find with xargs or -exec worked, however.
It's what happens if one uses * or similar globbing. The shell will try to pass all the files as arguments to the command and if it is too many, it will fail. That is why using find is often needed.
All extremely useful, my favourite being the .inputrc rebindings of up and down to search history. Takes a little getting used to but is great once you are (good luck to anyone else trying to use your terminal though ;))
Also I found the following settings to be very useful:
# List the possible completions when Tab is pressed
set show-all-if-ambiguous on
#tab complete without needing to get the case right
set completion-ignore-case on
One of my new favorites-- use ctrl-r autocompletion to grab a command close to what I want from history, tack on a character that breaks it if necessary and run that.
Then use fc to get that command back in a vi editing context, change what I want and :wq
The resulting command is immediately executed. This process can be really fast compared to manually constructing long piped commands.
No he did mean ssh-keygen which generates a public and private key pair for you. Run ssh-keygen on your local machine and copy the public key to ~/.ssh/authorized_keys and you'll be able to login to the server without a password using ssh -i /path/to/private/key user@host
That approach is insecure, however, because anyone with the private key now has access. When running ssh-keygen, you should add a passphrase to the key, then add the key to ssh-agent so you don't need a password for the account, nor do you need to type the key's passphrase constantly.
Anyone with private key access has control over my user account and has better access than what my private key + passphrase would provide.
I understand layers (probably moreso than most), but this is something that always bothers me a lot from a practicality perspective. My passwords are encrypted at rest via encrypted filesystems. If you are running things on my personal machine as my user account, I'm already being keylogged and/or am executing arbitrary code for you. If I'm logged into somewhere via ssh (hint: I am whenever I have a network connection), you can just scan my ssh config and use my ssh key anyway. From there, you can probably do a lot of other nasty stuff. ssh-agent won't really prevent this. It will prevent the malware from working again when I reboot until I log into another remote host (which I've established I do a lot) where the keylogger now gets me.
It's possible, but extremely unlikely that I have might have completely read-only media. I could be using my TPM device to protect from from booting and executing modified system states. Some of this might prevent you from easily persisting the keylogger threat across reboots. I might also have a module or something that calculate checksums on startup of critical things, have ridiculous anti-exfil outgoing connection policies, etc that prevent all but the most targeted attacks.
I don't have all of that in place. (In particular, to anyone generating a profile for me, I don't build detailed outgoing packet filter rules (you are welcome).) But what I do have in place will probably prevent me from getting my initial passphase keylogged if I used ssh-agent since it's likely (although this isn't strictly necessary) that I'm going to get attacked again after I start logging into remote hosts. So they can't steal my password, but they do have unrestricted access to my user account and the remote users I can log into. That complicates things, but is still a major security failure, to the point where them having the passphrase to my key isn't super important. I mean, in this scenario, they already have the absolute best input vector (a history of me logging in so they can execute attacks at the times I'm supposed to be logging in, as well as direct access to the systems from my ip addresses) to the point where using my ssh key from elsewhere is probably a worse a idea.
Here is an example from the man page:
>For example, the following command will copy the list of >files and directories which start with
> an uppercase letter in the current directory >to destdir:
If you put that in, for example, `~/.vim/ftplugin/markdown.vim`, it will be used on all markdown files. (`../ftdetect/markdown.vim` controls how Vim determines that a file is markdown.)
I can't believe this past has sort | uniq when GNU sort (important distinction, the BSD version and hence OSX can't) has a -u flag so sort -u == sort | uniq
I use variations on "find" so often that I've created several little commands "fij" (find in any java source), "fit" (find in any text / org file), "fix" (find in any XML file) etc. which I use all the time
Searches all the files in current directory and all subdirectories for files that contain "hello". Add -l option to grep to only display the filenames instead of filename and match.
Though that's not strictly a glob, but brace expansion (it will expand to all the numbers in that range regardless of whether or not files with those names exist). bash does have 'shopt -s extglob', which enables a number of useful globbing extensions, though I don't believe there are any numeric ones among them.
A sequence expression takes the form {x..y[..incr]}, where x and y are either integers or single characters, and incr, an optional increment, is an integer. When integers are supplied, the expression expands to each number between x and y, inclusive. Supplied integers may be pre‐fixed with 0 to force each term to have the same width. When either x or y begins with a zero, the shell attempts to force all generated terms to contain the same number of digits, zero-padding where necessary. When characters are supplied, the expression expands to each character lexicographically between x and y, inclusive. Note that both x and y must be of the same type. When the increment is supplied, it is used as the difference between each term. The default increment is 1 or -1 as appropriate.
I also prefer to do substitution with the :s modifier rather than ^ as suggested at the link, for consistency's sake:
Relevant Bash manual pages:http://www.gnu.org/software/bash/manual/html_node/Event-Desi...
http://www.gnu.org/software/bash/manual/html_node/Word-Desig...
http://www.gnu.org/software/bash/manual/html_node/Modifiers....