Show HN: Tunnelling TCP through a file

[fiddyschmitt]

https://github.com/fiddyschmitt/File-Tunnel

This program can be used to tunnel TCP connections through a file.

People have used it for interesting things:

- Bridging connections which would otherwise be blocked by a firewall

- Tunneling through RDP (similar to an SSH tunnel)

- Exposing a localhost web server to others

Key features I put effort into:

1. The shared file is restarted every 10 MB, so it doesn't grow indefinitely.

2. Optimisations for latency & bandwidth. (800 Mbps on a Gigabit LAN. 108 Mbps if file tunneling through RDP)

3. Synchronisation between two sides (each side can be started and restarted in any order)

I'd love to hear about any weird and wonderful uses you might have for it.

Thanks, Fidel

[ycombiredd]

That’s cool. A question that immediately comes to my mind is this: how is this different than using the Windows SMB native support for named pipes over the IPC$ share? From your example it would seem the users on both sides of the share would require the same AAA for your implementation as would be required to use IPC$ via SMB.

If that is indeed the case, (I am not a Windows pr SMB expert, so perhaps you will tell me it is not the case and to be clear, I am not asserting that it is or trying to say anything negative; I am just curious and you seem like a pretty likely source to be able to answer the question, since something motivated you to write this tool)

One difference would seem at first brush is that a named pipe used for this would not need to be periodically truncated in 10 minute intervals to prevent runaway file growth. Again, not knocking what you’ve made at all, I’m genuinely curious for the answer.

[mr_mitm → ycombiredd]

There are many other protocols besides SMB that can be used to share a file. Webdav, ftp, nfs, two systems accessing a third file share, shared folders in a VM, ... Or a file share provided by RDP like in the example, which does not include the IPC share as far as I know.

[ycombiredd → mr_mitm]

Hah! You answered my question re how is it different. It’s strange… I re-read your comment twice a with a furrowed brow: “provided by RDP like in the example” and kept thinking to myself “what? There was a screenshot clearly showing…” yada yada SMB. So I went back to the look at the image in the readme. /smh. You know how they say eyewitness testimony is often among the least accurate, even immediately after the incident in question, yet the eyewitness is always so sure, “but I just saw it and…” Wow. Mind playing tricks on me.

Anyway, you’re right, there was no SMB involved in that readme. :-) So strange when you can’t trust your own eyes or short-term memory. Crazy.

[nullindividual → ycombiredd]

If you want an overview of how redirection works in MS-RDP, Microsoft has a good deep dive at https://learn.microsoft.com/en-us/openspecs/windows_protocol....

[adolph → mr_mitm]

A classic is dd over nc

https://www.ndchost.com/wiki/server-administration/netcat-ov...

The main trick is to do it over UDP to get speed like Aspera:

https://www.ibm.com/products/aspera

[_flux → ycombiredd]

I wish unix domain sockets worked over NFS. It would be such a nice way to assign user-level permissions on networked services. ssh /services/gw etc. Maybe even file:/// -urls for the web browser, though the cross domain logic would need updates.

It would presume trusting the permission system over NFS, though.

[rakoo]

There's a more complete version of this with nncp: https://www.complete.org/nncp/

[aeonik → rakoo]

This is utterly mind blowing.

[fiddyschmitt → rakoo]

Woah!

[rnmmrnm]

reminds me how the other day I circumvented disabled SSH port forwarding by using https://github.com/nwtgck/yamux-cli + ncat already on the machine to do something like: yamux -l lport | ssh server -- <script: ncat --proxy-type=http -klp proxyport & and yamux localhost proxyport>.sh

good thing I needed HTTP access or else i had to find that socks5 server that's actually working again.

[aeonik → rnmmrnm]

I've been trying to read up on TCP/UDP multiplexing and I'm having a hard time understanding what the use cases are, or advantages over just using regular connections, sockets, and application logic.

Do you have a good resource recommendation to learn this more?

[rnmmrnm → aeonik]

Well here I tunneled multiple TCP connections traffic over a single input/output streams (the SSH shell IO stream).

Basically when you got a single communication channel and it's expensive to create more channels to tunnel multiple TCP streams where each connection get's its on channel, this is when you reach for the muxer.

i.e. here I probably could make it without yamux if instead I used SSH control port(socket? master?) and created many shell IO streams. but using yamux makes it cleaner. Also the server I'm working on is very buggy and you really get like 50% success rate logging in.

I suggest you download this `yamux` utility and play around with it really :)

[fiddyschmitt → rnmmrnm]

Nice! It's so good when you can cobble together various tools to achieve something like that.

[dheera]

Feature request: TCP over Facebook Messenger, TCP over Whatsapp

For use on flights where they give you "messenger only" internet access.

Feature request: TCP over JPEG cat pictures on HTTP port 80

It would likely do a better job at circumventing the China firewall than existing VPNs that can be flagged as VPN protocols.

[mike_d → dheera]

> For use on flights where they give you "messenger only" internet access.

They still rate limit these flows to a very low speed.

[ranger_danger → dheera]

> on flights

You could just sniff the traffic for an already-logged-in MAC and clone it.

[simpsond → ranger_danger]

But that will negatively impact the person who paid. That’s not cool.

[DonHopkins]

Will this work with a pigeon and a USB stick? (Given a pigeon strong and fast enough to fly back and forth thousands of times.)

[imhoguy → DonHopkins]

Ping-eon would be very high.

[HocusLocus → imhoguy]

> Will this work with a pigeon and a USB stick? (Given a pigeon strong and fast enough to fly back and forth thousands of times.)

> Ping-eon would be very high.

The last 10 years of the Internet has been one long setup for this joke. Resetting the clock for 10 years.

[fiddyschmitt → DonHopkins]

Haha!

[jclarkcom]

Neat idea. For better performance, you might want to use native filesystem APIs, for example on windows use CreateFile with the FILE_WRITE_THROUGH flag to prevent caching and and extra flushing.

[fiddyschmitt → jclarkcom]

Thanks J :) Absolutely! I did try various flags to optimise performance. For CreateFile (which in .NET is wrapped by the FileStream class) I tried FILE_WRITE_THROUGH (which is FileOptions.WriteThrough in .NET), and found it impacted performance quite a lot.

The key to high performance as you rightly pointed out was preventing flushing. In the end, what worked best was reducing the number of writes to disk (which is the bottleneck). I did that by buffering 10-50 ms worth of TCP data, coupled with only flushing explicitly (using a large buffer so that neither BinaryWriter or FileStream flush automatically).

[jclarkcom → jclarkcom]

I think you can use named pipes if you are using SMB. Here is a example code in python (not tested).

import socket import win32pipe import win32file import sys

BUFFER_SIZE = 4096

def create_pipe(pipe_name): return win32pipe.CreateNamedPipe(pipe_name, win32pipe.PIPE_ACCESS_DUPLEX, win32pipe.PIPE_TYPE_BYTE | win32pipe.PIPE_READMODE_BYTE | win32pipe.PIPE_WAIT, 1, BUFFER_SIZE, BUFFER_SIZE, 0, None)

def open_pipe(pipe_name): return win32file.CreateFile(pipe_name, win32file.GENERIC_READ | win32file.GENERIC_WRITE, 0, None, win32file.OPEN_EXISTING, 0, None)

def proxy_server(machine_name): server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server_socket.bind(('0.0.0.0', 8080)) server_socket.listen(1)

    pipe_name_1 = r'\\' + machine_name + r'\pipe\proxy_pipe_1'
    pipe_name_2 = r'\\' + machine_name + r'\pipe\proxy_pipe_2'

    while True:
        client_socket, _ = server_socket.accept()
        pipe_1, pipe_2 = create_pipe(pipe_name_1), create_pipe(pipe_name_2)
        win32pipe.ConnectNamedPipe(pipe_1, None)
        win32pipe.ConnectNamedPipe(pipe_2, None)

        while True:
            data = client_socket.recv(BUFFER_SIZE)
            if not data:
                break
            win32file.WriteFile(pipe_1, data)
            response = win32file.ReadFile(pipe_2, BUFFER_SIZE)[1]
            client_socket.send(response)

        client_socket.close()
        win32file.CloseHandle(pipe_1)
        win32file.CloseHandle(pipe_2)

def proxy_client(server_name, machine_name): target_host, target_port = 'cnn.com', 80

    pipe_name_1 = r'\\' + server_name + r'\pipe\proxy_pipe_1_' + machine_name
    pipe_name_2 = r'\\' + server_name + r'\pipe\proxy_pipe_2_' + machine_name

    pipe_1, pipe_2 = open_pipe(pipe_name_1), open_pipe(pipe_name_2)
    target_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    target_socket.connect((target_host, target_port))

    while True:
        data = win32file.ReadFile(pipe_1, BUFFER_SIZE)[1]
        if not data:
            break
        target_socket.send(data)
        response = target_socket.recv(BUFFER_SIZE)
        win32file.WriteFile(pipe_2, response)

    target_socket.close()
    win32file.CloseHandle(pipe_1)
    win32file.CloseHandle(pipe_2)

if __name__ == '__main__': if len(sys.argv) < 3: print("Usage: python script.py [server|client] [machine_name] [server_name (for client only)]") sys.exit(1)

    mode, machine_name = sys.argv[1], sys.argv[2]

    if mode == 'server':
        proxy_server(machine_name)
    elif mode == 'client':
        if len(sys.argv) < 4:
            print("Please provide the server name or IP address for the client mode.")
            sys.exit(1)
        server_name = sys.argv[3]
        proxy_client(server_name, machine_name)
    else:
        print("Invalid mode. Please use 'server' or 'client'.")
        sys.exit(1)

[naikrovek → jclarkcom]

You wrote all that without testing anything? Oh you used ChatGPT probably

[fiddyschmitt → jclarkcom]

That's awesome!

[bawolff]

What's the benefit of this over netcat + mkfifo?

[fiddyschmitt → bawolff]

Hi Bawolff! Yes that's a great combo. I guess File Tunnel provides a couple of things that makes the tunnel maintenance a bit easier.

1. It gracefully supports each side of the tunnel turning on and off.

2. It accepts any number of clients, and forwards them through the tunnel.

3. It recycles the shared file.

[lambdaxyzw → bawolff]

netcat+mkfifo does the actual transfer with TCP - so there's no tunneling involved. This solution tunnels tcp through file abstraction, so it works significantly differently.

As for the uses, see the three author listed. For example, by using a file share as transport you may evade firewall.

[zamadatix → lambdaxyzw]

  mkfifo /mnt/shared/connection
  nc localhost 12345 > /mnt/shared/connection < /mnt/shared/connection

Seems the same?

[Thorrez → bawolff]

That only works on Linux, but it looks like this program works on Windows too.

[andrewstuart]

So there’s multiple writers to the file? How is this arbitrated?

Also how does either side know when the file has been updated?

[jandrese → andrewstuart]

It looks like one file per direction, and this is a point to point connection. It looks like each side just reads to the end of the file and then tries to read more to see if any new data is available.

[fiddyschmitt → jandrese]

Yes exactly. One file in each direction made arbitration easier.

In the future I will implement a single file to handle both directions.

[fiddyschmitt → andrewstuart]

Hi Andrew, there's just one writer (which can tunnel multiple TCP connections).

Arbitration was indeed one of the trickiest bits. Originally I pre-reallocated the full file size (10 MB). Then used an integer at the beginning of the file to signal to the other side that a block was ready. The other side repeatedly read that int, and read the corresponding part of the file. But writing twice (once for the data, once for the int) had a significant performance impact.

In the end, what worked best was not pre-allocating the file. Rather letting the file grow whenever the writer writes to it. The reader knows when data is available by doing a PeekChar() in a tight loop. It's surprisingly fast, and accurately reflects the state of the file.

[resoluteteeth]

For RDP you can use virtual channels so you don't need to use a file over drive redirection

[fiddyschmitt → resoluteteeth]

Cool. I think rdp2tcp uses virtual channels for tunneling:

https://rdp2tcp.sourceforge.net/