Mosh is a truly impressive feat of thought and, perhaps more
importantly, engineering. The thinking behind Mosh is substantial, the
paradigms are fresh. Reading about mosh on its excellent website,
I always leave with a profound feeling of enthusiasm. Mosh belongs a
school of software development ethics that I'm sorely missing in the
world.
However, I do concur that, perhaps, Mosh solves a deep problem on the
wrong level or, even, in the wrong domain. The feeling, already
expressed here, that "tmux oughta do it" never leaves me. Why is that?
Because I know that ssh, by itsself, can handle connection loss quite
well[0]. Rather, I suspect, ssh can't deal with IP changes or the
destruction of the underlying interface.
So, here's the rub: Why is a stable, somehow-abstract, network interface
with a local IP, sitting on top of the actual network interfaces, be
they wired or wireless, not the answer? Over the years, on and off,
I tried to get the Linux ethernet bonding stack (ifenslave) to provide
me with just that solution, but was never able to.
Hence, I keep revisiting mosh, then revisiting ifenslave...
[0] On a machine that doesn't dynamically switch network interfaces on
demand, just unplug an ethernet cable, wait a while and plug it back
in: your ssh session will resume.
You missed the point of Mosh. SSH transfers a bidirectional stream of bytes between server and client. It uses TCP as transport. You can combine SSH with tmux to preserve your sessions as tmux sessions. If you use SSH to access your tmux session you fixed a part of the problem. Your client can now roam by reattaching the tmux session from a new SSH connection.
The other half of the problem remains: SSH transfers unmodified byte streams between client and server over TCP. On networks with high latency or high packet loss this design doesn't work. Mosh fixes this part of the problem by redefining the problem. Instead of transferring every byte in both directions it runs a terminal emulator on both sides and the problem turn into: keep the terminal emulator states in sync. The server sends patches from the last confirmed client state to the current server state to the last known client address:port pair via UDP. The client sends the user input (and pings) to the server via UDP. This design allows Mosh to skip over lost messages.
Mosh implements a delta algorithm between the server's buffer and the client's buffer. This means if packet loss is occurring, to sync back up, the server only needs to send the delta of its state to the client.
SSH, on the other hand, is stream-oriented, and will replay everything that has occurred, even if the resulting changes to the display in the end are minimal.
It's similar to how something like VNC functions compared to the X11 protocol. The VNC protocol is more efficient and tolerant of lossy or low-latency connections.
Mobile IP etc. solve the problem of changing IP addresses, but they don't address the flakiness of mobile connections (esp. when actually mobile). TCP takes some time to come back; having a faster mechanisms is much appreciated when your link loses a few packets twice every minute.
Mosh is brilliant software. I use it every day. Being able to close my laptop, move to a different café, and open it back up with all of my sessions seamlessly reconnecting is invaluable.
But -- mosh still can't do proper SSH agent forwarding[0], which is a real drag. Many of my common workflows become impossible. I hope it gets fixed!
I feel like mosh is fixing problems at the wrong layer. Instead of making application protocols connectioness, we should make Mobile IP work so that connections don't need to be broken in the first place, or, failing that, separate the protocol management from the session persistence system.
Combining regular ssh with screen has worked fine for me for years. Why would I want to switch to something with a much shorter track record for security?
(By the way: if you want a line-buffered ssh for use over bad connections, run ssh inside Emacs with M-x shell.)
Unfortunately we have very little chance of having an impact on how ip is implemented in mobile network, but we do have the possibility to change our softwares to accommodate deficiencies. Much like HTTP over SPDY is highly similar to HTTP over SCTP, but we have much more chance in making SPDY a reality rather than SCTP.
> Combining regular ssh with screen has worked fine for me for years.
I guess the real novelty (and it's too bad it isn't presented in greater depths) is SSP, the synchronization protocol. It can be an inspiration for other network programs.
Thank you for the tip with "screen"! I was aware of MOSH but did not know about screen oO. Is there any other line-buffered ssh which does not require Emacs?
Well, this is probably as good a place as any to ask a tough TMUX question I've had for several years. I love TMUX, but whenever I'm working with protocols or anything binary, if I accidentally push any binary out of stdout, it corrupts the display, and the only way I've figured out how to fix this once it's happened is to totally restart TMUX.
Does anyone know how to fix this?
(don't usually ask SO questions on HN but this has been bugging me for years)
'reset' might work, but often when this happens to me, it's because the window's name has been changed to something ridiculous. Hit "<prefix>," and rename your window back to something sensible.
I have very litle experience with TMUX... But you can try runnng the "reset" command. It resets the TTY, if TMUX does not use any crazy magic it should work.
Yes. I was wondering, faced with an unreliable on-train mobile connection, is there any existing way to improve the content delivery by using a proxy host?
It seemed to me as if the mobile browser kept redoing a whole lot of work, ie. essentially re-requesting web sites from scratch. I'd like to make a single request to a web site, have my proxy host complete the request and deliver the bytes piecemeal to the browser as I go in and out of range.
I think it's possible to do this transparently to the browser, but I think you might need a proxy host with a reliable connection as well as a proxy software running on the mobile device that can deal with the situation by telling the reliable host: I'm back, here's a list of running requests and the last byte I received, please continue.
> I generally do not connect to web sites from my own machine, aside from a few sites I have some special relationship with. I fetch web pages from other sites by sending mail to a program (see git://git.gnu.org/womb/hacks.git) that fetches them, much like wget, and then mails them back to me. Then I look at them using a web browser, unless it is easy to see the text in the HTML page directly.
Alas, both of them are proprietary and monolithic - IOW, you can only use the infrastructure iff you're also using the one specific client. Not very useful for extending even for other browsers, or even different protocols - which is a pity :(
> we should make Mobile IP work so that connections don't need to be broken in the first place
A lot of the problems with mobile networking are the results of connections literally being cut off. On trains I often end up without coverage (tunnels, in the middle of nowhere) exactly for that reason; the connection broke. I don't think that hacking a system based on connections to pretend that this never happens is a particularly good idea.
The major problem with it is that existing protocols need to be reinvented once again to fit new way of communicating. Of course you could encapsulate TCP/IP over it, but then you're losing all nice properties of the protocol.
Combining regular ssh with screen has worked fine for me for years.
This doesn't work for me because I use emacs over X11 over a tunnel.
mosh doesn't work because I feel required to use ssh-agent. That is, I feel required to use a password on my ssh private key and not using ssh-agent is too high a burden.
I run ssh inside emacs, and I also leave emacs running in server mode on my targets. I drop my signal, no big deal; I reconnect and reattach to emacs. No fuss, no muss, no lost work. It then becomes immense that I can do almost everything in emacs :)
Let's assume for a moment that RFC6725 is implemented in pacemakers or artificial organs. How long after death would/should these devices remain active? could we use Mobile ipv6 to communicate with the recently-departed loved ones to send final well-wishes?
If your workflow doesn't include tmux or screen, then I feel like your problem's already been answered. If you really wanted to, you could just add a "/bin/screen -DR; exit" to your remote .bashrc and the screen integration would be completely invisible after the initial instantiation.
As far as mosh is concerned, it feels like it's an extremely overengineered solution to the "I wish I had started this process in a screen session" problem, except you need to be thinking even further ahead than you would normally (installing and connecting via a mosh service, as opposed to remembering to connect to a screen session that can be run with very little prior setup and no client modification).
While the situation you describe is one that some people end up using it for - it's not the real reason mosh is useful.
Where mosh really shines is when your IP is changing too frequently (a mobile problem when you are on the move) and when you are dealing with high latency connections.
When you have to ssh in somewhere over a shitty, barely working mobile data connection - mosh is a lifesaver. It makes an otherwise unusable situation comfortable and productive.
Sure, it's handy for jumping between connections, on/off wifi and so on - but that's easy to achieve many other ways with tools that have been around for a decade or two at this point.
> solution to the "I wish I had started this process in a screen session"
Not necessarily. When I'm thethering my phone's data connection to my laptop I use mosh to ssh into my server because it resists way better to bad connectivity. No matter if I used screen or not, I would still have to kill the ssh process and connect again and reopen screen if I didn't use mosh.
I have been using Screen for years. Will Screen attempt to reconnect to a remote SSH session? I think what `screen -DR` will do is - it will automatically connect me to a screen session `after` I login to the remote machine.
But if I lost connectivity - my SSH session won't be auto-resumed from local machine in first place, I think thats one of the things `mosh` solves.
It promises lower latency etc, but I am just perhaps too slow to notice such things (or both SSH/Mosh feel equally laggy if remote connection is slow enough).
>But if I lost connectivity - my SSH session won't be auto-resumed from local machine in first place, I think thats one of the things `mosh` solves.
While true, it just seems (to me) that if that's the only problem you're solving (which is to say, not needing to type the command "ssh host" and let your key log you in) and it's at the expense of not being able to properly ForwardAgent (which I require very heavily in my line of work), then it's a lot of work for negative return.
But, again, to each their own. As mentioned below, if your latency over a phone connection is so bad that this helps you, then by all means, use it. :) I'm just noting that for pretty much everyone I work with, this would be a step backwards.
> you could just add a "/bin/screen -DR; exit" to your
> remote .bashrc and the screen integration would be
> completely invisible after the initial instantiation.
The problem is making all of those initial instantiations every time my connection is interrupted.
Huh. I certainly don't remember step 4 from my time of using screen. Is it a new feature or an option I didn't know about or something?
And what happens if I want to have two terminal tabs on my end logged into the same remote machine? (Which is basically 100% of the time for me.) It seems like that would be a case where mosh would be much smoother...
(Disclaimer: Once upon a time I used screen, but I stopped ages ago. mosh is intriguing, but as long as I'm working from home just plain vanilla ssh is all I really need...)
The ability to detach and re-attach to screen has been there since day one... that's probably one of the largest reasons most people use it. Terminal software these days can handle multiple sessions just fine - the benefit is being able to detach and re-attach.
Multiple terminals in one session was more critical back when we were using dumb terminals and whatnot - where you couldn't multiplex locally.
Just so it's clear, I meant specifically the '.bashrc runs "screen -DR"' part, ie automatically going back into the screen rather than needing to explicitly run screen -DR like I used to. (I don't seem to be able to edit my original comment at the moment.)
That's not something screen-specific. You can put whatever you want into .bashrc and it will be automatically run each time you open a new shell. pseudonym just suggested you could use it to automatically run a screen command.
> what happens if I want to have two terminal tabs on my end logged into the same remote machine
Don't do that. Open a new window in your screen instead.
Then start giving windows descriptive names. Then start giving them automatic names. Marvel at most fast you've gotten using the window selector (with type-ahead-find).
> And what happens if I want to have two terminal tabs on my end logged into the same remote machine? (Which is basically 100% of the time for me.) It seems like that would be a case where mosh would be much smoother...
You can use 'screen -x' to attach multiple times to the same session.
"I wish I had started this process in a screen session" problem
Why is everyone here assuming all programs can be run in a screen session? I run emacs over X11 through a tunnel. This is far superior to emacs in a screen session.
I've solved this by buying FastX from StarNet. I run a server on Linux and display my X programs on my desktop at work. When I get home, where I run the client on my home desktop, I can connect to the same server and disconnect my work clients and reconnect them to my home machine. It's the perfect solution for me.
Mosh is really great for avoiding disconnections and local echo. Lack of SSH agent forwarding was a problem for me as well, but I have found a workaround - I open a separate plain 'ssh -A' connection at the same time when I need my keys. On target machines I have SSH_AUTH_SOCK pointing to the stable well known location, which is a symlink to the unix socket recreated always by the last 'ssh -A' connection on the server. This allows me to use ssh from tmux session connected using mosh.
Agent forwarding is an antipattern, and insecure usage of it is prevalent. Forwarding it allows anyone who has root access on the remote system to impersonate you, and it can therefore be used to escalate privileges after a single system compromise.
Mosh always seems oddly slow. I have a good connection to most of my servers (under 10ms to many, under 100ms to all)... and it just feels laggy, in vim sessions, in tmux scrollback, everywhere. I find just a vanilla ssh (with tmux always) to be far superior in terms of latency.
I love the idea of mosh, but I find the implementation to be lacking... or I am doing it wrong. :)
Yep, it's in my daily tools as well and very useful especially for 2G mobile connections, bad Wi-Fi and alike. It seems their development has pretty much stopped after initial spark of enthusiasm. Too bad, mosh-over-ssh and port forwarding would be pretty killer features for many scenarios.
Yeah mosh is amazing. Just using screen isn't nearly as good because you still have restart the ssh connection. I usually have 5+ open at once so restarting them all is annoying.
Are you aware that a forwarded agent can be used by anyone with access to your account (e.g. root) to authenticate as you? They can even re-forward your agent to their own computer.
Too bad mosh needs an udp port to run its own server.
One of the greatness of ssh is its lean network requirements (it can even be considered https if seen from intermediate proxies, hence whitelisted) and this extra udp port is quite a burden in some environments.
I still agree that mosh is a fine tool for mobile users. I'm just a tad worried about its security layer implementation being less audites than openssh for instance (as mosh isn't as widely used)...
Well, Mosh still uses OpenSSH for the initial authentication, and during the rest of the connection, all the security is handled by a single layer (AES-OCB, using the reference implementation), which is not specific to Mosh.
It's not a replacement at all - it doesn't support scroll back (screen buffer). Latency often changes the order of keystrokes. I've tried to used it so many times and gave up every single time after the enthusiasm ran out.
The mosh protocol is specifically designed such that reordering won't happen (since it does use UDP as a base protocol, it requires this protection). I think they'd be very interested to hear about any such bugs you have regarding that.
Also, as far as I was aware, ssh provided no scrollback either. Any such scrollback is a property of the (local) terminal you're using, not ssh itself.
> Any such scrollback is a property of the (local) terminal you're using, not ssh itself.
The Mosh client is also a terminal emulator (similar to screen); part of the reason it can be so efficient is that it isn't just another layer in the network stack over which standard lines of text travel, it is a remote console viewer (like VNC is for graphical desktops, only sending parts of the desktop that have changed; and if there is a massive wall of text spamming the screen, it will just show you the end result rather than sending it all)
Both ssh and mosh use a ptty on the remote device, but this doesn't provide scrollback itself, just a fixed size window where you can see some previous commands. Scrollback is a property purely of the local terminal.
Though after doing some research, it seems mosh skips some intermediate states when large amounts of lines are printed, causing them not to be sent to the client. This would cause the observed behaviour of no scrollback in the local terminal.
I've never had this experience! And I've used mosh for some time extensively on all sorts of connections ranging from abysmal to horrible. Maybe you used an older version?
Yes, it's been an issue in the past as I gave up on Mosh. To be fear, I will reinstall and try. Worst of all, I was having these issues on relatively fast connection, so, that's what turned me off as I could imagine when there are real latency issues in place.
one issue ive with mosh over time is that it doesnt timeout properly every time (it does most of the time after a long period). this keeps tmux/screen sessions attached "forever" which can be an issue with some programs that detect if you're detached to send notifications for example
Several commenters are asking what mosh provides that's better than ssh+screen/tmux.
A few years ago I spent a summer at a remote field site, sharing a flaky T1 with 100 other people over equally flaky site-wide wireless network. (Many of those people were undergraduates who had been politely asked not to use the research network to access facebook, but, well, you know.) Not only was packet loss common, but latencies to an outside host varied whimsically from 50ms to 2000ms. In those conditions ssh was unusable, tmux or not. I was able to do work on remote servers only because of mosh.
Let me clarify a little more. Mosh hides latency by doing display prediction. It guesses what the results of your keystrokes will be and displays them locally, highlighted as predicted with an underscore.
My main issue with mosh is having to open a massive UDP port range, which just goes against my instinct when it comes to security.
The other security issue is the whole 'seamless reconnect' - it's probably been thought of, but does anyone have any links to how they've mitigated the massive MITM risk this opens users to?
You don't actually need to open a UDP port range, a single UDP port is enough. But then you'll have to tell mosh with the -p parameter which port it has to use.
I wish half of this had been implemented in tmux instead. tmux is already client/server, add local typeahead (which is very cool) to the client and let me run it over whatever transport I want (spoiler: it will still be ssh, because I don't really benefit that much from their reinvention of TCP)
Both "client" and "server" in TMUX terminology are on the same machine. AFAIK, MOSH creates a terminal emulator in each connected machine and keeps them in sync.
My biggest complaint about Mosh is that how it handles the login sequence breaks common .bashrc/.bash_profile idioms like
test -z "$BASHPROFILEREAD" || return
(to avoid double-sourcing) and there's no good way to fix them. I haven't tested ksh93/zsh yet, but I suspect similar issues. I suspect the developers do little more with their shell initialization than set a few aliases and variables so it's not an issue for them, but fixing it for myself is far more work than worth it. It's too bad, everything else works well for me with mosh.
I don't think you've ever actually been on a bad connection. mosh has saved my ass more than once when I've been stuck with a satellite connection in the middle of nowhere. (Try fixing a failing loadbalancer while on a transatlantic cruise, using SSH)
Because even a small amount of packet loss makes SSH absolutely unusable (e.g making it completely freeze), whereas I've managed to function using mosh with around 30% packet loss.
You're ssh'd to a remote server and in a screen session. Actually, let's say you've got tmux going, and like four panes with different sessions to different hosts.
Now, you close your laptop, hop on your bike, enjoy a nice lunch somewhere, and head to a café for the afternoon. You open your laptop again.
Correct me if I'm wrong, but with your setup, you need to go to each pane and re-ssh to each host. But with mosh, I don't need to do anything except wait a few seconds. All of my connections are re-established, and (even without screen) my sessions are restored just as I left them.
That's almost as practical as keeping all your connections alive in a session, on a machine with a reliable network. Except that in my setup I can connect from any computer, while your is tied to only one.
I rarelly do anything like that anyway, because I don't like long standing connections. What are you doing with all those connections open anyway?
Configuring stuff? Why don't you clone your settings, and just push when done? (Or use puppy.)
Accessing local resources? Nope, if that was the case, losing the connection will disrupt your work anyway unless you script everything, in what case screen'll fit much better.
Waiting for a repply? Email and SMS are much better protocols for handling that.
Don't get me wrong. I see some value in automatically reconnecting to sessions (and when I care about a session, my computer does that). Also, handling bad connections (not intermitent) well is a feature. But those come at a cost - you are now tied to one client computer, and you can't script it as well as plain ssh, and some security concerns. I don't think it's worth that cost.
No it doesn't, in fact it doesn't have anything to do with network connections whatsoever. Yes it allows you to re-attach to a running session, but that has nothing to do with making a connection with, say, 40% packet loss usable.
are those links becoming invalidated after 3month and auto reposted or something ?
This URL must have shown up like 3 times on the front page in the past year alone
How it can be replacement if it uses ssh to start daemon on server to listen on UDP port? But yes, it's great. Still not perfect, still issues with non unicode chars sometime, still some issues with rendering from time to time.
What's a non-Unicode character, and where would you find one? Do you mean the crazy private use characters used to make fancy Vim status lines, or something else?
I actually find lack of local echo somewhat valuable. When my connection is spotty, that gives me immediate feedback about what is and is not getting through.
You get similar feedback when using mosh -- text that is echoed locally but has not yet reached the remote host is displayed with an underline. On a bad connection, you see underlined text promptly as you type, and the underline gradually disappears as the network catches up.
I have been using mosh for two years now and it has greatly improved my life. It means that rather than buying a brand-new laptop, I can just connect to a $5/mo linode with an SSD and program on that.
Q: Has your secure datagram protocol been audited by experts?
No. Mosh is actively used and has been read over by security-
minded crypto nerds who think its design is reasonable, but any
novel datagram protocol is going to have to prove itself, and
SSP is no exception. We use the reference implementations of
AES-128 and OCB, and we welcome your eyes on the code. We think
the radical simplicity of the design is an advantage, but of
course others have thought that and have been wrong. We don't
doubt it will (properly!) take time for the security community
to get comfortable with mosh.
Reposts are fine, gets the blood flowing again. I totally forgot about Mosh and decided to watch the awesome presentation on their website, also decided to enable on my tablet via JuiceSSH.
However, I do concur that, perhaps, Mosh solves a deep problem on the wrong level or, even, in the wrong domain. The feeling, already expressed here, that "tmux oughta do it" never leaves me. Why is that?
Because I know that ssh, by itsself, can handle connection loss quite well[0]. Rather, I suspect, ssh can't deal with IP changes or the destruction of the underlying interface.
So, here's the rub: Why is a stable, somehow-abstract, network interface with a local IP, sitting on top of the actual network interfaces, be they wired or wireless, not the answer? Over the years, on and off, I tried to get the Linux ethernet bonding stack (ifenslave) to provide me with just that solution, but was never able to.
Hence, I keep revisiting mosh, then revisiting ifenslave...
[0] On a machine that doesn't dynamically switch network interfaces on demand, just unplug an ethernet cable, wait a while and plug it back in: your ssh session will resume.