Another really neat solution to sending data between two machines that don't have a pre-existing channel is magic-wormhole[1]. It's much simpler because it's just a number and some easy-to-distinquish code words like 1-stethoscope-zulu.
I dove in to see how they solve the routing / firewall problem. Turns out you’re relying on the kindness of strangers and a public WebSocket relay. Hard to see how they could do any better, alas.
IMO one of the big holes in the architecture of the web is the lack of a standard peer-to-peer “content-addressable routing” protocol, perhaps based on any of a number of distributed hash table schemes. It’d be perfect for something like this.
Excellent! Yeah, I often daydream about a web where something very much like that had become foundational.
Unfortunately, the academic work on DHTs emerged at around the same time as both the stultifying dominance of Internet Explorer and the panic over Napster et al.
> A better solution for p2p E2EE file-transfer is croc [0] because it can compile to a statically-linked binary without runtime dependencies.
For some definition of "better", sure. I really do wonder why the author didn't make it compatible with magic-wormhole. Seems a bit silly to cause incompatibility between two implementations that are so similar that they are functionally identical.
> Custom code-phrases are supported.
magic-wormhole supports that too (you still need to have a channel number but you can pick that too).
The one feature of croc which magic-wormhole doesn't have is transfer resumption (though I do wonder how difficult it would've been to add this to magic-wormhole -- or to at least extend the magic-wormhole API to support it). As far as I can tell, the only other real benefit of croc is "it's written in Go" -- which is a real benefit for some people (most notably, Windows users without a working Python setup) but it's not really one that justifies having an incompatible version of basically the same idea.
Just a note, there is a Go port of magic-wormhole that does properly interoperate with the official python version and has the benefit of not requiring a python environment: https://github.com/psanford/wormhole-william
Oh neat, that's quite cool. I was hoping to write an Android app for magic-wormhole -- so you could share files with mobile devices. Packaging Go on Android is much nicer than Python, so I might revive that idea at some point.
I have a very (very) rough react-native frontend for this that runs on android. I've been waiting for the go-mobile module support to settle down a bit before cleaning it up and releasing it. Its probably about time for me to pick it back up again.
This is not to discourage others from building on top of wormhole-william. I built the public API[1] first and then added the CLI tool with the hope that others would be able to easily pick it up and use it in other applications.
I mostly use Firefox Send to transmit data between my own devices and devices I do not trust.
I wish the web client could generate QR codes to make that easier.
Probably I could combine Fsend and another CLI tool to enable that use case. Right now I'm manually typing links into different devices.
I like that this removes some of the need to trust the service. When you open a link in a browser it's tedious to verify that the webpage doesn't send the secret to anyone, but this way you only have to trust the command line tool, which can't be replaced unpredictably.
Somebody that knows zilch about Rust with a question: I like to write CLI tools with Go, the built-in libs and the simpleness of the language make it really easy for writing good tools quickly. What advantages does Rust have over Go that one would use it for this purpose?
I don't know the current state of go package management but I do know the equivalent Go project cited Rust's package manager Cargo as a good reference for developing their own. You forget Rust's standard library is quite minimalist and you get to sample competing libraries which often use interchangeable interfaces. In particular there are several crates (libraries) which make handling CLI arguments convenient.
I'm not certain, just learning it. You might find this blog post that does a code review of ripgrep interesting, though. It's a deep dive into a serious-scale cli tool.
It's 2020. Is it finally possible to send a file over the Internet directly from one machine to another?
Only if you are not behind symmetric NAT I suppose.
Tangent: why does symmetric NAT exist? It seems like the kind of thing firewall fetishists do "for security" without considering the problems it causes and without any clue of what sort of attack it thwarts that mere stateful firewalling does not thwart.
Realize that without symmetric NAT, TURN would be largely unnecessary.
I am pretty sure there are many people on here who have not forgotten (it is not even that old! :P) and still use any variations of netcat, or even rsync. Nmap's Ncat is cross-platform (I use MSYS2 on Windows so it does not matter much to me), supports SSL, proxies, connection brokering, and a lot of other stuff[1][2].
> A basic file transfer is very simple: Start Ncat in listen mode on one end, start Ncat in connect mode on the other end, and pipe the file over the connection. There are two ways to do this that differ only in which end listens, the sender or the receiver. Sometimes you can't create a listening socket on one end of the transfer because of a lack or permissions, NAT, or filtering. As long as you can listen on at least one end, though, you can use this technique.[1]
This works with most if not all netcat implementations.
> The basic file transmission technique described at the beginning of this section fails if neither participating host is capable of listening, or the two hosts can't communicate directly. This situation has become common with the prevalence of network address translation. A way to work around it is to use a third host as an intermediary. The intermediate host listens in connection brokering mode and the other two hosts connect to it. Recall from the section called “Connection Brokering” that in connection brokering mode any input received on one socket is copied and sent out to all other sockets. With just two hosts connected this is especially simple: anything coming from one host gets forwarded to the other. This example shows host1 sending inputfile to outputfile on host2, using host3 as an intermediary.[1]
magic-wormhole solves this problem (while being encrypted and incredibly easy to use). If the two hosts can directly reach each other, it will directly do the transfer, otherwise it uses a public relay server which both hosts can reach.
I'll go ahead and plug https://patchbay.pub here. If you're somewhat less concerned about having encryption, you can do CLI transfers of files of arbitrary size without installing anything (assuming you have an HTTP client like curl already), and without needing an account:
Your service sends everything through the instance so we rely on the bandwidth you're willing to share. It already is amazing, but the same thing through WebRTC would be really great. I know it means work would mostly be on the client side though, and curl wouldn't be enough anymore
These are all very interesting, I am currently facing an issue getting BitTorrent file transfer to pierce an "unfriendly" firewall that I can't change permissions on. I've tried a VPN out with no luck. I wish I knew if one of these would allow me to get out.
Firefox Send is a web page, so if you can access the web you can use it. Also, a firewall that you can access the web from will have ports 80 (http) and 443 (https) open, so configure your tools to use one of those (preferably 443). If both machines are standard internet-facing Unix-likes that you have accounts on, you'll already have scp preinstalled for easy file transfer.
As long as your traffic is encrypted (which ssh/scp is) and as long as your machine isn't compromised then a firewall can tell so little about your traffic that it would have to be pretty paranoid to attempt to block any of it. It'll work at a Starbucks, but perhaps not in China.
Deep packet inspection will almost invariably be able to recognize ssh/scp from the packet sizes, the patterns, the ports and so on. You really do need some kind of obfuscation. Starbucks won't catch it, but I know for a fact my former high school does, for example.
Looks cool!
Personally, however, I’ve been using croc (https://github.com/schollz/croc) and can’t recommend it enough. I use it almost on a daily basis and is a super easy file transfer tool.
Conveys the ecosystem. When I see “CLI tool written in Python” vs “CLI tool written in Rust/Go” I pick the latter because Python’s packaging situation for command line tools is beyond awful.
By packaging situation do you mean the user-side of things, i.e. how to install stuff separately of each other? For that I've had good experience using https://pypi.org/project/pipx to install several cli tools without conflicts. Uninstalling them has also been a breeze.
I wrote a tool called pipsi which I believe has influenced pipx but despite the existence of such tools I have no idea these days how to install python packages any more. Ir has become only more confusing and complex. Both for cli tools as well as libraries.
Pipx is the replacement for pipsi. It's good for installing cli tools.
For managing project dependencies, Poetry is the current best. It means I dont have to interact with virtualenv or pip or setuptools or pip-tools or manifest.in.
This is very surprising to read. When I read something like this from others, I was always arguing that "it's not anymore", but I'm absolutely sure you know better than me. Would you mind elaborating actual problems or writing a blog post about it? I am VERY curious about your experience and opinion in this topic!
Personally it's more about familiarity, I dev Rust so I know that I have all the prerequisites and setting this up should be trivial, meanwhile I don't do a lot of Python work so getting that to work might be a bigger commitment.
I don't do a lot of Go either but at least Go generates freestanding binaries so if I install it in precompiled form I know that it will be fairly frictionless and won't rely on half a billion dependencies (which may or may not conflict with something else). I use a bunch of Go programs regularly (like docker or Arch's yay package manager) and it never gets on in the way.
On the other hand if it's an npm package I don't even think about it.
Yes, but every time I tried it, it always fails me with some obscure error that I run away from it with bitter memory(why did I do this in Python!?!).
If it is not a priority for the language, I’ll rather choose a language which cares about it.
I've never really used Python, but as a consumer of tools, I don't recall ever coming across a Python tool packaged as a binary - so it might be possible, but dose anyone actually do it?
GCs have a nonzero overhead wrt manual management. This is an undeniable fact. The only time they can compare is for memory allocation profiles that already "emulate" a sort of a tracing/refcounted GC themselves.
Aha. I have a web server written in rust that literally uses about 5Mb memory on RPi. And that one actually has a proper web framework, not something barebones.
On top of that, that server doesn't actually use any sorts of downgraded language version that Java EE is.
You literally get that for free with Rust.
I still don't understand why people argue against this. Rust is a better language in so many aspects (tooling, language capabilities) it's really hard to argue against it.
I went with Rocket. Basically I wanted to control the switching of devices on my sound amplifier using USB infrared remote. So I built the UI using yew and the web server using Rocket - but I wanted to do it fast and only in Rust.
I am not serving static content, so this reply is absolutely irrelevant.
On top of this, you can do the same with docker run nginx -v.:/var/www (which btw is a similar number of symbols). Also python’s http module barely supports anything but the most basic content-types (so good luck with eg SVGs).
Sure, if you want to be pedantic, but the context was limited memory use, and ref counting does not increase the memory used for unreferenced objects (except in cycles) unlike most other garbage collection algorithms.
No GC forces the programmer to think about memory, this is a good thing because you need to to that in all garbabe collected languages anyway - or you will get bloat.
For most use cases I consider garbage collection to be a mistake. Scripting languages or small one-off applications exempted.
Unnecessary sarcasm aside, I'm Keen to look at it simple because it's written in Rust. I like Rust so this is interesting for that reason alone.
Besides that, Mozilla and the Firefox community have a lot of Rust devs so writing this in Rust may make this more likely to get PRs from people involved in the FF Send project
Thanks for your extremely useful sarcastic comment. Maybe the author just wanted to play with the language and share their knowledge through a cool open-source tool. Maybe it's just the best language for the job.
It does actually a lot - I know that when I contribute to this project I have a much lower probability of shooting myself in the foot compared to most of the other languages
Well I can't use this on Windows nor can I guarantee that this cryptic shell script will run on my friends computer who still uses Windows. Even so, he wouldn't know how to run it.
Also it can't be a "secure sharing service" if you're still using 'sha1sum' and it not being end-to-end encrypted...
Also it's quite secure since file content is clearly mapped to the URL you're sharing with the other party via independent channel and the receiving party can verify the content of the file.
It may not be completely private if VPS does not run on your own HW, but nothing prevents you from running it on your own HW in your secured location.
You can use this on windows the same way you'd use it on Linux. You just install a bunch of programs and run commands from command line. (basically just like you'd need to do with fsend) It's less hassle on Linux, since the commands are pre-installed. Fsend is more hassle on Linux since it's not preinstalled and requires rust, compilation, etc.
The other people just open a https links and download files in any browser or via curl, no tools required at all, not even JS.
The VPS doesn't really belong to you. Google has exactly the same ability to view your secret file shared unencrypted over Drive as they do to view your file stored on your f1-micro instance.
That's just your assumption. VPS != Google. It's just a virtual machine running on some HW. I agree that if that HW is not owned by you, it can be a problem. If that's a problem you can just add another layer of encryption to the mix, like a GPG command.
I can and do own that HW. At least as much as one can own anything - it's located in my appartment. You can just as well run the web server from your own workstation/laptop.
The only reason to put shared files on other computer is that the other computer is more available.
That's not the same thing, Firefox Send stores the files on the cloud, it's accessible from anywhere without requiring your computer to remain accessible.
[1]: https://github.com/warner/magic-wormhole