After skimming the "How it Works" page, I'd be curious to know how they get around laggy interrupts, e.g. cat'ing a massive file and trying to Ctrl-C out of it. With Mosh, there's no giant delay because they don't care about history, they're just sending snapshots of the latest data. Hence why they use UDP as well.
ET however uses TCP, and some type of BackedReader and BackedWriter for TCP data - which _implies_ to me that everything is being sent/received in a massive queue. Also, it supports scrolling, which would be difficult _(but not impossible)_ if they only cared about recent frames like Mosh.
Finally, I'm curious how well it handles mobile connections. That's another selling point of Mosh. Admittedly I never use mobile internet, so that's not a concern of mine, personally.
I love Mosh and I'm always happy to see competing apps, hope they complement each other well :)
> ...I'd be curious to know how they get around laggy interrupts, e.g. cat'ing a massive file and trying to Ctrl-C out of it.
Presumably with the TCP URG flag. That said, I only read years ago that's how it's done, without ever checking myself. I've found references to in in telnet, but not ssh, so perhaps the latter uses some other mechanism.
TCP urgent data only indicates that the application should try really hard to read up to a certain point. It doesn't provide a way to abandon transmission of data that's already been sent with write(). It also generally does not convince routers on the way to prioritize the urgent packet before packets already in their buffers.
The way you'd want to use it is not to transmit the Ctrl-C urgently (that direction is clear), but to transmit the interruption of the reply urgently back to the client. And I think there's nothing that will save the server from having to actually send all the bytes that you managed to attempt to send before you received the Ctrl-C.
mosh works around this by rendering the terminal window server-side and sending diffs of the terminal window (think VNC or streaming video) to the client. So the in-flight data from server to client is at most a screenful. If the data isn't sent in the first place, you don't have to worry about unsending it. (Of course the tradeoff is loss of scrollback.)
You won't get native scrollback with server-side buffering. Precisely because it's scrollback that's off the screen, there's no way to send ANSI escape sequences to edit it after the fact; it has to actually be written to the screen and then scrolled off. So you can't really do anything where you transmit the current screen at higher priority and scrollback at lower priority. (Unless you want to do something where mosh will write out scrollback when it receives it and then write the actual current screen below it, but I bet that will cause a ton of flicker.)
You can probably do some things to heuristically improve scrollback, e.g., telling the terminal sync thing to sync 1000 lines of scrollback if the connection seems good and only the current screen if it seems poor. You won't get reliable scrollback but maybe it's good enough.
I feel like 99% of what I get from mosh is good performance over mobile networks, not IP roaming per se. In particular, TCP's congestion control algorithm is intended for networks that get congested, i.e., low-bandwidth shared-media physical networks where a dropped packet is a sign that you should lower your transmission rate. That's approximately zero of the networks I use; when my packets get dropped, it's usually because I'm in a train and tethered to my phone which went briefly out of range of cell service, or because someone turned on the microwave, or because I'm being handed off between wifi or cell towers, or whatever. So constantly reconnecting over TCP (especially if there's a significant application-layer handshake for the reconnection) doesn't seem like it solves the problems I have. mosh uses a custom, non-TCP reliable transport layer that is actually built for mobile networks.
Both scrollback and tmux -CC support are feasible in theory with mosh, though difficult to get right:
There's a fork of mosh linked in the first comment that switches the terminal-to-user end from a synchronized terminal screen to a bytestream (same as the user-to-terminal direction, i.e., keyboard input). It's not going to be robust to, say, catting a huge file and needing to Ctrl-C, but it should maintain mosh's robustness to radio (wifi/cell) network blips. It was designed for scrollback, but I'd sort of expect it to make tmux -CC work too.
Anyway, it is super cool to see more efforts in this space, and I bet for people who work primarily at desks with reliable internet connections, this is a huge improvement over, say, autossh. But for my own use case for mosh, this doesn't seem like it's got feature-parity yet. Maybe there are clever things you can do with aggressively restarting TCP connections that get you the same experience as mosh's UDP approach.
IIRC, Terminal.app sends a key code when the user scrolls in mosh (or any program that takes over the screen). Maybe it would be possible to handle that and display the scroll back buffer?
Oh, yes, if there's a way to signal a scroll attempt to the application then it's totally doable.
Is this the "Scroll alternate screen" checkbox? I believe that just controls the scroll wheel, and makes it not do native scrolling and instead just send a page-up or something. If so, then non-native scrolling is definitely possible ... but it's already possible if you run screen or tmux instead of mosh. (Which is what I do, and it works fine for me, in part because I learned how to use screen's scrollback even before mosh existed.) I can see the argument for exposing this with better keyboard shortcuts than screen's inscrutable ones, but my impression is that people mostly want native scrolling so they can, e.g., drag in the terminal UI to copy and paste a large amount of text. You can't do that with non-native scrollback, since the text isn't all contiguous in the terminal window.
FWIW, I've had a lot of personal success with just plain SSH managing the resumption of sessions. The trick is to not send keepalives from the client, and to not time out idle sessions on the server. When you can control both of these, you don't need extra tooling to resume sessions.
That said, it's not as friendly to IP changes, nor does it provide predictive typing, so there's definitely something to be said for the value of tools such as these.
I do have one question, given the tmux and scrolling support, how much mosh-like lag compensation can ET provide?
ET is primarily made for IP changes and also because the
kernel/router will sometimes kill stale connections, and it's not possible to send keep-alives if your laptop is asleep.
There is no lag compensation because ET is not a terminal emulator, it's only a remote shell. I cannot create predictive letters and undo them reliably (passing backspace commands to the terminal sometimes does not undo the last letter). Lag compensation would need to be implemented by a terminal emulator such as hyper.js.
Sure, if you have a reliable internet connection, and a desktop computer and a server that you both control, you probably don't need this or mosh.
But mosh (and presumably this program as well) really, really shine with a laptop that you might hibernate for days, and then boot back up, and it's just as you left it.
Resumption? None that I can find in sshd_config but there are several related to increasing tolerances to prevent it from terminating in the first place.
ClientAliveCountMax default is 3
ClientAliveInterval default is 0
Ergo by default ClientAlive messaging isn't used. If Interval were set to 60, the connection would be terminated after 3 minutes.
From man sshd_config:
TCPKeepAlive
"Specifies whether the system should send TCP keepalive messages to the other side. If they are sent, death of the connection or crash of one of the machines will be properly noticed. However, this means that connections will die if the route is down temporarily, and some people find it annoying. On the other hand, if TCP keepalives are not sent, sessions may hang indefinitely on the server, leaving "ghost" users and consuming server resources.
The default is yes (to send TCP keepalive messages), and the server will notice if the network goes down or the client host crashes. This avoids infinitely hanging sessions."
That's probably not that interesting because it's considered spoofable. Maybe more interesting are the client side ssh_config options which allow for opportunistic multiplexing:
It occurs automatically, so long as the client and server haven't timed the other out.
The client will time out the server by sending a packet with no response (such as via the 'ServerAliveInterval'), the server via unsetting 'ClientAliveInterval', 'ClientAliveCountMax', and 'TCPKeepAlive'.
All your saying is that the tcp connection doesn't break if you don't send anything whilst either host is not unreachable.
You can break the connection in between and resume as long as your ip hasn't changed.
So it's not really resuming ssh or anything, just a function of tcp sockets. Generally disabled on public since something could open up a bunch of listening sockets and then leave them and never close.
With your servers that you've disabled the keep alive and intervals, if you drop a connection and never reconnect, it's going to keep it open. A malicious could keep opening connections and exhausting your server.
The Linux kernel will close the TCP connection if it's idle after about 5 minutes (configurable), regardless of the application settings (with one exception noted in a bit). That's why you can even get "connection closed by peer" errors in the first place. TCP keepalive messages are used to circumvent the kernel (and every router between the two endpoints) from closing that TCP connection due to it being idle.
Plus, I can put my computer into hibernation for upwards of 12 hours, and still come in and have a responsive SSH session over a new TCP connection. So long as the key being used to encrypt the session is still in use, and the associated TTY has not been closed out, your client can create a new TCP connection and resume the previous SSH session.
> A malicious could keep opening connections and exhausting your server.
This could (and does) occur regardless of your application TCP settings.
The linux kernel does not automatically close a TCP connection if it's idle, it can stay idle for years on end (unless you globally enable TCP keepalives, or some NAT/stateful firewall devices inbetween times it out).
What's happening is not that your client is creating a new TCP connection and resumes an SSH session, you are just and continuing on your existing SSH session over an existing TCP connection - which you can do as long as neither the client ssh process or server sshd process have been restarted, and neither end has attempted and failed to send any data or otherwise timed/errored out the TCP connection.
(e.g. if you did a (sleep 60 ; echo test)& , then went hibernate on your client side, the server side would attempt to send some output in 60 seconds, and eventually time out the connection after a few minutes unless you wake up in the mean time)
I did some hands-on research, and I now agree that you are correct. The behavior is absolutely acting like a resumption of the TCP session, not extra connectivity magic within SSHD.
Thanks for the persistence!
That said, though, there is something which closes idle, established, TCP connections, though it's not the kernel, it's iptables which does it. The timeout is something along the lines of 5 days by default, however. That's what I get for working from memory.
No, there isn't, TCP is end-to-end, and can only be closed by the endpoints.
If you use iptables (or rather netfilter)'s stateful filter, then netfilter will forget the connection tracking state of the connection after a long-ish time without any traffic (may be 5 days, I don't remember). That doesn't close anything, it simply means that netfilter doesn't know about the connection anymore. No packets are being sent when that happens.
That in turn has two effects:
First, if you have a rule that only allows packets through for established connections, that rule will now not match packets from that connection anymore. But if the packet matches another rule that allows it to pass, then it will still make it through the firewall, and as a side effect that would then re-establish the connection tracking state, which in turn would also make the rule for established connections match again for any packets that follow.
So, in most cases, that's not in any way a problem: The next packet your client sends after a long idle time will just re-establish the connection tracking state and the connection will continue to work just fine.
Second, if the connection is NATted, the NAT state is part of the connection tracking state, and as such, the NAT mapping of the connection is also lost when the state is dropped. That means that the next packet (if it comes from the right direction) will once again consult the nat table, just as the very first packet of the connection, to establish a new NAT mapping. Now, in many cases, that will establish a new mapping that's exactly the same as the old mapping, in which case the connection also will continue to work just fine. Only if the new NAT mapping differs, that will cause the connection to fail. Yet another reason why you don't want to have NAT (and by extension, why you want IPv6).
I think you're referring to the OS sending its own TCP keepalive probes - as configured by sysctls like net.ipv4.tcp_keepalive_(intvl|probes|time)
TCP keepalive messages are what the kernel uses, and not what circumvents the kernel's own probes.
Applications can also implementation their own layers of 'are you alive?'/'yes I am' type messages on top. SSH has its own messages (see the ServerAliveInterval/ServerAliveCountMax config keys)
> Mosh is a popular alternative to ET. While mosh provides the same core funtionality as ET, it does not support native scrolling nor tmux control mode (tmux -CC).
I use byobu (screen) for all the things this is supposed to address. I think. Byobu doesn't keep your ssh connection from dropping but the session is still there once you reconnect with no loss of history. Plus you can connect from multiple clients (using the same user account).
According to their website (the third bullet point):
> mosh: Mosh is a popular alternative to ET. While mosh provides the same core funtionality as ET, it does not support native scrolling nor tmux control mode (tmux -CC).
How does a GPL license "strangle" a network connection tool? Are you really bothered that you can't incorporate it into your own BSD/closed source project, or is it a purely ideological "it's not BSD, so it doesn't respect me as a developer" stance?
It's a pain to get mosh support in closed-source iOS apps, but I think that's a good thing, and that's more about the closed-source requirement. (Though the iOS requirement means that you can't take the fork-and-exec approach that e.g. JuiceSSH on Android does; your whole app has to be free software.)
There's a free-software one that charges money on the App Store for a precompiled binary, which seems like the right plan: https://github.com/blinksh/blink
The initial password is only used for authentication, isn't it? Someone could (unless more details are provided) snoop on the connection or even inject packets.
A one-time shared passkey is sent over ssh and then this passkey is used to encrypt/decrypt the data in both directions. The server and the client have different initial nonces.
That was my concern too. So ssh is used for the initial handshake? BFHD. A lot of the interesting stuff happens afterward. The "how it works" page explicitly says this:
> ET does not implement any of the SSH protocol
That implies they rolled their own wire protocol. Maybe they did a good job (like mosh appears to have done). More likely not. I'd be OK using this on an internal network or over a VPN, but relying on it alone for security would be premature.
Well, NaCl is certainly an improvement over a completely NIH approach, but doesn't necessarily provide features like forward secrecy. It would certainly be nice if the ET authors would explain in their "how it works" page how they've solved problems like these. Forward secrecy over resumable channels is not a trivial problem for which the quality of solutions should be assumed.
It's in a different category but there's also Apache Guacamole... typically you'd host your web proxy+container where ever your servers are, so the buffering and scrolling happens on the remote side and is rendered to your browser. No idea what happens if you try to ^C out of a 100MB cat halfway through.
It could, but then it would have to deal with the SSH line protocol which is complicated, because it allows for a LOT of stuff. ET is basically pretty dumb, it basically just dumps a PTY onto the network, and is way simpler.
To me this seems like Mosh, but worse. I think that each project's approach has different strengths. For me, the best part of Mosh is the intelligent, asynchronous handling of type-ahead. Mosh feels really snappy, even over poor connections. I'm not bothered by needing to run Tmux on the remote for scrollback; I do that anyways, and I have no need for this fancy "control center" mode.
My big frustration with both Mosh and ET is that neither expose a data communication channel along side the PTY. I use SSH for port forwarding, Tmux paste buffer sync, file transfer, and SSH agent forwarding. Without those features, I'll always need to use SSH/Autossh along side these tools for my other SSH use cases.
It gives you native scrolling / tabs in iTerm2 which for me was a huge plus. I hated using the keyboard for copy and paste in TMux. That's just a personal preference though.
It is not clear from the error that you were missing libsodium. I don't remember what was my error anyway, I'll try again later. But I can't even find libsodium on my apt.
You may be running a distro that doesn't have libsodium (such as ubuntu 14). In this case you have to grab libsodium manually and build et from source.
ET however uses TCP, and some type of BackedReader and BackedWriter for TCP data - which _implies_ to me that everything is being sent/received in a massive queue. Also, it supports scrolling, which would be difficult _(but not impossible)_ if they only cared about recent frames like Mosh.
Finally, I'm curious how well it handles mobile connections. That's another selling point of Mosh. Admittedly I never use mobile internet, so that's not a concern of mine, personally.
I love Mosh and I'm always happy to see competing apps, hope they complement each other well :)