from Hacker News

Ask HN: Best modern file transfer/synchronization protocol?

by daveidol on 12/22/23, 5:46 PM with 58 comments

Hi HN,

I'm looking to build something to transfer files between two devices on a network (one client and one server, both of which are under my control).

Obviously I could write something bespoke with raw TCP sockets, but rather than reinventing the wheel I'm curious about what existing options people recommend. I assume there are some better technologies than FTP nowadays?

Ideally some kind of built-in fault tolerance would be great, because my plan is to use this on a phone/in an environment where the connection could be interrupted.

Edit: just to clarify - this is something I want to build into an application I am writing, ideally with support across iOS (client), Windows, and mac (server).

One way transfer is all I need, and I mostly plan on photos/videos (so multiple files ~3-20MB in size).

Thanks!

  • by orbz on 12/22/23, 6:17 PM

    No need to get fancy, scp or rsync are the tried and true options here.

  • by ryukoposting on 12/22/23, 6:16 PM

    I use syncthing for this. It's a little fiddly to set up, and transfer speeds aren't great. However, it's very reliable once configured, and it barely uses any resources after an initial scan of the folder you want to sync.

  • by 2color on 12/22/23, 11:06 PM

    https://iroh.computer/sendme

    https://iroh.computer/

    Iroh is what you’re looking for. You can embed into your code, and it handled wire encryption, verification, and NAT hole punching. All over QUIC.

  • by pvtmert on 12/22/23, 8:28 PM

    I highly recommend Unison (https://github.com/bcpierce00/unison)

    It allows you to sync between 2 machines (bi-directional) over TCP or SSH.

    Note that TCP way is not encrypted, you may use wireguard as transport layer encryption for that purpose...

    You can use an external application to copy if file size is larger than an arbitrary number. (Eg: use rsync for files > 1gb)

  • by whalesalad on 12/22/23, 5:55 PM

    seconding rsync and syncthing.

    the server could expose an smb or nfs share, the client could mount it, and then sync to that mount.

    rsync over ssh also works, if you do not want to run smb/nfs.

    this is also a cool tool https://rclone.org/

  • by rmorey on 12/22/23, 6:38 PM

    I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well

  • by arun-mani-j on 12/23/23, 3:38 AM

    On a similar note, can someone tell me what's the fastest (wireless) way to transfer files between two laptops on same network (i.e. hotspot)?

    scp, rsync, wormhole give me only 2-3 mb/s.

    For the context, I'm trying to transfer about 50-70 GB files.

    What's causing the bottleneck here and what am I missing? Thanks in advance!

    https://github.com/magic-wormhole/magic-wormhole

  • by oschrenk on 12/22/23, 6:38 PM

    If it’s one way (that wasn’t quite clear from the requirements to me).

    take a look at https://tus.io/

  • by bhaney on 12/22/23, 5:55 PM

    Yeah I usually just use rsync for this. In a loop if the network is unreliable.

  • by rrix2 on 12/22/23, 6:42 PM

    I built something on top of the Syncthing API this week after using it on its own for years.

    A local instance of Syncthing can behave as a robust sync tool + inotify API for applications consuming the files: https://docs.syncthing.net/rest/events-get.html#get-rest-eve...

    i believe there's an embeddable golang library, but if you want something easy to use on android check in on syncthing-fork which lets you define more granular sync conditions including "just turn on 5 minutes every hour" https://github.com/Catfriend1/syncthing-android

  • by mynegation on 12/22/23, 6:29 PM

    What are your latency and bandwidth requirements? How big are the files? If you are already looking past obvious TCP-based choices like HTTP and FTP, you might be interested in FASP/Aspera https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol

    Edit: I’ll leave it here just in case it is useful for others but it may or may not be embeddable into your app, especially on the phone.

  • by jayknight on 12/22/23, 5:53 PM

    I would probably start with rsync.

  • by sneak on 12/22/23, 5:53 PM

    rsync over ssh for one-shots.

    syncthing for continuous use.

  • by smackeyacky on 12/22/23, 10:45 PM

    Grab an S3 bucket on amazon.

    Do a 3 way sync with the s3 command line tool.

    That way, you have a neat cloud backup as well. Wouldn't take any more than 20 minutes total to set up.

  • by eternityforest on 12/22/23, 11:47 PM

    What about Jami? It runs almost everywhere and having an embeddable Jami library would be absolutely amazing, although a fair amount of work.

  • by tripleo1 on 12/24/23, 8:47 PM

    1. warpinator, syncthing 2. rclone??

    --

    3. self hosted ipfs in tailscale or something? (that would be cool)

  • by pluto_modadic on 12/22/23, 6:48 PM

    continuous sync - mutagen.io (maybe you could extract some of the libraries)

    depends on if it's large or small files.

  • by Helmut10001 on 12/23/23, 6:28 AM

    rsync. If you are looking for a long term solution: zfs on both sides with zfs send.

  • by yetanother12345 on 12/22/23, 8:41 PM

    eh... wget or curl ? or not modern enough for you?

  • by toomim on 12/22/23, 6:21 PM

    Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...