Expect is such a great tool for a specific class of problem. Decades ago I worked as a test engineer for a big hardware vendor. We had maybe a couple hundred raid controllers that we were testing and to address some issues we found, the supplier needed us to change a bunch of configuration variables that were only accessible via a text menu system over a serial connection to these devices. After painful manual reconfiguration of a handful of these things, I instead wrote a fairly complex expect script to make the appropriate changes and also validate all the settings on the rest of the raid controllers in a fraction of the time it would have taken to do it manually. It saved me so much hassle and got me some pretty good kudos and respect from my direct leads. It's one of those tools that I kind of wish I would need to use again!
Linux tool? Hate when everything Unix automatically gets categorized as Linux. Have used Expect for many years for many use cases from automating servers to networking use cases as well.
For example, expect(1) is described in the 1989 Don Libes book "Life with UNIX". Libes is the author of Expect. A title such as "Life with Linux" would probably have made little sense at the time.
Correction: I went back and looked at the book and it does not mention expect. According to Wikipedia expect was first released the following year, in 1990. Apologies for the inadvertence.
Nevethless I think it is safe to say that expect was not written using or for Linux as Linux was not released until 1991.
My go-to example for `expect` is password-based SSH. Since we're all enlightened beings these days who surely never use this anymore, there's no risk to it:
`interact` hands you back control to do whatever you want, but you don't actually need to do that. You can have `expect` run every command, fill out every field.
The tool you need when you're in environments you don't deserve.
You forgot all the other prompts that ssh might produce before the password prompt (e.g. add new host to known hosts yes/no). There are more possible prompts than most people realize.
This isn't a script that we'd send on an unmanned, inter-planetary, mission. This could be something that can save the tedium of typing password on a daily basis. If it fails, well, we still have a reason to be alive: press ^C.
out of curiosity, why do you call the expect script in string form from a bash shebang, rather than make the script itself an expect script directly with an explicit expect shebang?
I have no technical reason, I just wanted to wrap it into a familiar context for people. People are I find less scared of "useful CLI tool I can script around" than "scripting language in its own right".
For this particular use case, I'd suggest sshpass instead.
https://linux.die.net/man/1/sshpass
...
Operations folks today might not understand -- everything just works nowadays, which is slightly boring. And boring is good!
But there's something unique about the feeling of making things work even though they are not designed to work, and really have no right to work. Expect has been a lifesaver for some of the most critical systems at some of the biggest companies in the world.
Definitely, `sshpass` is the more better keyhole tool for this job. I picked it as the most common example I can imagine everyone going "ugh, I have to do this by hand again?" at.
It has all the pitfalls of putting your front door key under your doormat. Except you're doing it in a world where everyone can materialize a unique key out of thin air, and you can always instantly tell your door which of those keys should or shouldn't be allowed to open it.
sshpass is a curse, even calling it a crutch would be improper flattery. It's a dangerous cheat that accomplishes nothing except impeding people from otherwise spending the 20 minutes it takes to figure out SSH keys.
Yes this is the proper dogma, but you're missing the point.
Any automation around passwords is a crutch and a mistake. But sometimes it is necessary.
You don't always control the remote systems. The remote systems are not always capable of key-based auth. And sometimes the remote system is not of high concern so the "danger" is null.
sshpass makes a reasonable effort to do the best-possible thing under these less-than-ideal circumstances. The other options suck more.
My most recent use of sshpass is to collect reports from a vendor over sftp. I would have preferred to use https with BASIC auth, but in truth that has exactly the same problems as sshpass, and I have other hills to die on.
There's certain devices (e.g. firewalls) that don't allow you to save your public key for SSH authentication, so you're stuck with passwords. I've used "expect" for automating backups to a TFTP server.
I used expect a ton years ago, for AIX system janitoring. A lot of things required telnet’ing into proto-ncurses mainframe applications and going through menus, typing and selecting stuff etc.
Many years ago I used the Perl version of Expect (https://metacpan.org/pod/Expect) to automate several workflows that wanted manual inputs.
I thought that it was far better than the TCL Expect because I could also use other CPAN modules and have Perl's regular expressions to parse the system's output.
To be fair, Tcl has Henry Spencer’s regexp[0][1], and is awesome; Postgres also base their regex on Henry’s work for Tcl[2]. Perl is certainly famous for its regex and popularizing thinking of problems in terms of regexes (for better or for worse[3]), but it’s not the only game in town.
>> Perl is certainly famous for its regex and popularizing thinking of problems in terms of regexes (for better or for worse), but it’s not the only game in town.
Agreed.
In that old system we did not have any TCL scripts or third-party libraries--just C, Perl, and C shell (no Bash; it was Unix).
Arguably, Tcl can do GUIs much easily and can integrate C less awkwardly than Perl. Both have their use cases, at any rate, and more people should use Tcl, Perl and any version of Expect. Nowadays it falls into the category of tools that were once somewhat used, but now are obscure, like fold/fmt, nl, comm and pr.
Perl/Tk has always been a red headed stepchild since it is a fork attempting to implement a C API that doesn't require Tcl to be linked. Because of that, it has long missed out on significant improvements to Tk over the years. Python Tkinter avoids that by just wrapping around a Tcl interpreter.
I use pexpect frequently for automating serial-port communications. It’s very useful for implementing test/flashing automation that needs some kind of a serial-port step (such as asking u-boot to run a few commands).
I do something very similar, how do you deal with random serial port data loss (dropped bytes)? Maybe it is specific to my environment, but I get mangled commands about 1% of time, which really sucks for automated workflows. I mostly countered this by writing commands to a tmp file, verifying md5sum and then executing the file on the device side.
Serial is slow but typically very reliable. If you’re seeing random dropped bytes, I’d take a look at your wiring and PC interface UART which might be low quality. FTDI UARTs in particular are often counterfeit (the real ones are good).
expect is one of those true love/hate things. Probably because the situations in which you need to use it are likely cursed. My time spent dealing with expect scripts is always miserable, and yet I don't know what I'd do without it
Yeah, I remember reading some posts around 1993 where someone wanted to port some existing Unix program to Linux. Instead of modifying the program they added the missing functionality to Linux. I think it was about polling and signal handling or something like that.
I just used python pexpect [0] for a fun project. I believe pexpect was inspired by the original Unix expect command.
I used it to let aider suggest and run shell commands [1]. This is handy, so aider can offer to run the code it just wrote for you, run migrations, install dependencies, etc.
Using pexepect makes it easy for aider to run interactive shell commands and capture their output. This allows aider to fix bugs or suggest next steps based on the command’s output, error messages, etc.
The only problem we had with pexpect (maybe expect) was that we started to have prompts disappear or text not appear/be presented to our python script. Could have been an issue of buffers not flushing. It’s been several years but this one detail I remember because we had several workarounds for it.
1. aider makes some change
2. aider then tells itself to use pexpect to check what changes it made
3. pexpect runs, opens browser somehow captures error messages if any and feeds that back?
It's a rare thing nowadays to find a terminal command that is interactive and doesn't provide alternative flags for batch processing.
Expect would still be useful for someone scripting a debugger (for example), but Tcl falls under the close to isoteric language nowadays and it takes less time to just repeat the commands in a readline capable interactive program than find just the right way to script it.
Last time I've tried to use expect was when I attempted to build a Tk frontend to the Haskell GHCi debugger (similar to the DDD application), but I got tripped up, from vague memory, on terminal ansi codes, and random lock ups I couldn't pinpoint.
Some tools like ssh and gpg go out of their way to prevent easy use via command line flags. They often have a good reason for doing this, but there are also sometimes good reasons where this type of protection causes problems. Since the implementers won’t budge on their stance, you need workarounds like expect.
We used pexpect to automate deployment from various servers and it would prompt users for passwords then they could login to perform deployment tasks for our software. It powered our entire deployment arch at that time. But it would frequently drop off text or prompts if you started doing nested logins.
claude.ai just suggested this as a tool for automating a workflow that involves ssh'ing into another machine and running some commands as sudo. I'm surprised I've never heard of this tool before. It works great for this scenario and the script can be used a reference to go in manually.
What the hell. If the device you are logging into supports sudo then it probably supports bash or python too. In that case a better solution is to
create a script on the target device and then call that script over ssh, using key-based authentication. You do not need expect. Expect is mainly used on devices with a rigid terminal UI that do not support scripting and require interactive password authentication.
That's definitely do-able, but I didn't want to do that because it adds a level of indirection. When I put my relevant commands in a script, I have to start thinking about what directory it's in and how that affects its operations. I also have to think about how to handle errors for failures of the various commands, and I can only handle those errors on the target machine. If I'd doing expect and a command has an error, I can handle it on my local machine which might have more info (maybe I can set it up to switch to a backup machine or something). Are these solvable problems? Sure, for me it's a matter of taste.
Also, your comment would read almost identically and perhaps even better without the first three words.
Expect was the first tool that really hooked me at my first job at Sun. I had to write a tool to automate verification of the boot process during firmware dev, handling variety of odd/unexpected behaviors, paths, and prompts.
Many years later I heard they were still relying on that tool!
Expect helped launch my early career in programming.
Automating tedious tasks as a network engineer and sysadmin motivated me to learn proper software development. If gluing together TCL/tk scripts could make my life easier, imagine what larger programs would do.
Very handy tool. I use it to partially automate logins on a CLI tool that have 2FA. Expect hooks op (1Pass’ CLI) to send the password, then tells the program what method of 2FA I want to use (Okta push to my phone).
In the days of landline service, you have to pay for long distance calls; so it mattered which side made the call, as the originator of the call was charged. So you make a quick call to have the computer call you back and stay on for a long time; the number associated with the computer location is billed.
