from Hacker News

Interview with CEO of rsync.net: “no firewalls and no routers”

by dmytton on 3/18/21, 4:20 PM with 334 comments

• by korethr on 3/18/21, 6:22 PM

  I wonder if they have any sales to large enterprises or similar institutions.

  In my experience, the larger organizations will have a "security" questionnaire required of their vendors, and the person administering it is a droid, incapable of evaluating whether the questions, originally written in the mid-00s and only updated for buzzword compliance since, are applicable to modern security practice today, or to the particular product/service/vendor in question. And no firewalls or routers would be massive, disqualifying red flags on such a questionnaire.

  Never mind that a KISS setup tends to bring security because of its minimized attack surface. In the minds that write and administer those questionnaires, security only comes from sufficient amounts of the right kinds of complexity.

  I'm sure it can be done. IIRC, Cloudflare doesn't use any firewalls, and they do some big business. It just isn't easy to get past the droids programmed to ensure that all pegs shall be properly square, IME.

• by travisgriggs on 3/19/21, 2:23 AM

  > I start the day with a short walk outdoors. I don’t want the first thing my eyes see to be print, and I don’t want the first thing my body does to be sitting. So I walk a bit.

  I like that. I like that a lot. That's a very enviable practice. I think I know what I'll be experimenting with this next week.

  Did not expect to read that article and have the most stand out thing be a routine change I'd want to copy. You never know.

• by robotmay on 3/18/21, 5:43 PM

  Nice article. rsync.net is one part of my personal computing setup that I never even think twice about. It's simple and it works, and that clearly applies to the infrastructure too. I use ZFS locally and it has made managing my own data strangely pleasing, and it's nice to have the same system on my off-site storage too.

  On the laptop-front, I find myself drifting towards a similar setup to John. I have a hefty workstation laptop but the battery life is dire and it weighs a ton, so I pretty much just run it as a headless machine next to my server now. I'm planning on picking up a Pinebook Pro as an "outdoors" machine to just remote in. I also find myself extremely unwilling to arse about swapping multiple machines on my monitors so being able to keep my work machine separate and secure but operate it from my desktop is a nice compromise.

• by rsync on 3/18/21, 4:28 PM

  Unrelated, as an aside ...

  I really am enjoying the developer Q&A interviews that console.dev is putting out.

  They're very much like the "usesthis"[1] profiles but more in-depth and with more interesting details ...

  [1] https://usesthis.com/

• by Crontab on 3/18/21, 9:17 PM

  John's usage reminded me of something I read in Rob Rike's "Uses This" interview[1]:

  "I want no local storage anywhere near me other than maybe caches. No disks, no state, my world entirely in the network. Storage needs to be backed up and maintained, which should be someone else's problem, one I'm happy to pay to have them solve."

  [1]https://usesthis.com/interviews/rob.pike/

• by anderiv on 3/18/21, 7:30 PM

  This was a pleasure to read. I've been an rsync.net customer for ~6 months now, and am using Borg to send de-duped, encrypted backups to rsync.net from a few on-premise linux systems. As compared to other similar backup systems I've used, it's been a pure pleasure to implement and maintain.

  Thank you for your great product and support, John!

• by tfsh on 3/18/21, 9:59 PM

  Meta: I really dislike the style of console.dev, the article is shunted to the left and leaves the rest of the screen real estate to be taken up by an - albeit pretty - but unnecessary piece of digital artwork. This - https://ibb.co/nzbFxjW - is what the article looks like on my ultrawide which made for very uncomfortable viewing

• by Aeolos on 3/18/21, 5:25 PM

  > "I have a early-2009 “octo” Mac Pro [...]" > > OS: macOS

  Does this make anyone else a bit uncomfortable?

  I don't think MacOS is still receiving security updates on that hardware. I'm all for using old hardware for as long as it keeps working, but I would never browse the internet with a vulnerable OS on a vulnerable processor (spectre etc...)

  Or am I missing something?

• by AdamJacobMuller on 3/19/21, 12:27 AM

  I really appreciate the information and would love more information on your architecture in particular. Also definitely love your information on your personal process and flow. Some of it seems interesting to adopt and some of it seems bad for me but I could see why you do it, some I do in similar ways but a bit differently. e.g. I use git and online services to make sure that any of my computers are completely replaceable and that I can pick up work at any moment from any one.

  I can't agree more with the "no firewalls" approach to things, though I prefer to call it "host based" firewalls as it scares people less! I'm glad you've had no compliance/audit pushback on that, I architect things similarly and have had success pushing back on the requirement as well.

  I'm very surprised by the l2 switches and actually choosing to run completely unmanaged switches. I assume you're running all 10G or more? Maybe i'm overthinking the complexity of your network but I would be lost without snmp counters on my switches and running switches+networking in fully l3 mode has some great isolation benefits, especially if you want full switch-level redundancy.

  Do you have some more details on your data architecture? I'm very curious how do you do data direction/redundancy/sharding and balancing customer data across servers. I'm not trying to pry for things you consider secret but I think you have a very similar architectural mindset and I'm curious how you solve these things.

• by booi on 3/18/21, 5:39 PM

  A simple layer 2 network topology only works in very narrow use cases (like this one). But a "dumb switch" means you also lose a lot of observability and it's very difficult to apply consistent network acls.

• by efxhoy on 3/18/21, 7:06 PM

  That was a nice read! Good to read about something simple after a day working with AWS and their managed magic.

  Scrolling through the cert pages 2015 seems to be in the future though?

  > We personally toured every single major datacenter in Hong Kong and Zurich to choose the facilities that best met our old-fashioned standards for datacenter and telco infrastructure. The same will be true of our upcoming Montreal location in Q4, 2015. https://www.rsync.net/resources/regulatory/sas70.html

• by api on 3/18/21, 7:04 PM

  I go by the rule that if something is not secure enough to plug directly into the Internet, it is not secure. That doesn't mean I'll necessarily do that, but that should be the bar.

  The only exception is special purpose backplane networks that are designed explicitly to be isolated. These are basically data busses for clusters, not user-facing networks.

• by bflesch on 3/18/21, 6:36 PM

  Big fan of rsync.net but the firewall comment caught me a bit off-guard. The benefit of a firewall is that it's an isolated system which - apart from port blocking - guarantees a certain level of traffic logging and known-good state.

  If you have everything on one host I'd say your overall setup on that host becomes much more complex because you only need to get hit by one successful exploit chain and all logs on that host cannot be trusted any more.

• by gautamcgoel on 3/19/21, 2:23 AM

  The pricing model doesn't make sense to me. Their prices start at $0.025/GB/month, so renting 1TB of storage for a year would cost $300 - at that price, I could just buy my own disks and run ZFS myself. I kinda hoped they could offer lower prices using economies of scale. I checked the prices for Tarsnap, expecting it to be cheaper - it's actually 10x more expensive! Maybe someone can explain what I'm missing.

• by frammie on 3/18/21, 5:55 PM

  Really well done interview, some real interesting bits in there.

  One part concerned me though, in the interview, it mentions "we own (and have built) all of our own platform." and it fails to mention a few critically important key parts of a storage platform, first being encryption. How are personal files being handled? Is encryption being used? Are you able to access this data using a shared key?

  As well as contingency, what happens if critically important data is stored on your platform. On your website you mention:

  "We have a world class, IPV6-capable network with locations in three US cities as well as Zurich and Hong Kong"

  however fails to mention if replication is done across these locations. If technology (drives) is stolen from your datacenter, or mechanical failures beyond your control happen, how will you be able to recover from physical failure if you only appear to be serving from a single location?

  Excuse me if I'm wrong but I couldn't find anything concrete in either the interview or your website. The premise of the platform seems quite well aligned with keeping alive the the UNIX philosophy, and reminds me of Tarsnap.

  Either way, well made interview and interesting approach to a storage platform.

  As a sidenote, what keyboard are you using? It seems really interesting and you failed to mention it in the interview :)

  EDIT: It appears that you offer Geo-Redundant Filesystem as as separate product, maybe you would want to make this a bit more visible on your website except for only the FAQ and order pages. Either way, it seems like a sufficient move, that does still leave the topic of encryption though. As mentioned traffic is encrypted using SSH ofcourse, but is the data itself encrypted on your platform?

• by RaitoBezarius on 3/18/21, 9:25 PM

  You write down that you have no router, though your primary US location is connected to a "quintuple-homed network" and all global locations are at least triple-homed.

  What does that mean exactly? Is your IP provider quintuple-homed? Or are you running a bit more complicated setup than you explain but the gist is that you have no particular routing mechanisms?

  What does that say regarding your high availability? If one of your location is down, then it's definitely down until being fixed?

  Anyway, that was interesting, just curious about the fact of having no router at all. Thanks!

• by ttsiodras on 3/18/21, 4:59 PM

  Interesting interview - thanks John! Didn't know there was a UFS2 "phase" before ZFS... I wonder how much time those fscks took! :-)

• by tiffanyh on 3/18/21, 8:34 PM

  @rsync

  If you had to do it all over again, what would you do different (if anything)?

  E.g. product/positioning/tech-stack/employees/business-decisions

• by ynx on 3/18/21, 11:07 PM

  The replacement for Spectacle, FYI, is Rectangle, which is almost identical, but still maintained

• by ciil on 3/18/21, 5:26 PM

  Jealous of how well you seem to be able to keep to KISS as a principle.

• by poisonborz on 3/18/21, 8:49 PM

  This was maybe the first service I see that was somewhat complex, but the 4 line main page header text clearly explained what the tool does - the subpages are also great, low-key, great reads. Kudos to whoever copywrote the site.

• by limaoscarjuliet on 3/19/21, 9:45 AM

  Question for rsync:

  You said: This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security.

  You seem to suggest the big firewalls do not bring any value to the table. I always thought they had more "intelligence" - dropping sessions based on some bad patterns, guarding against DDoS (to some extent), etc.

  Are you saying BSD is as good as these expensive boxes? Does it apply to SSH only or HTTP(s) and some other traffic as well?

• by hannofcart on 3/19/21, 5:40 AM

  I wish all SaaS services were like rsync

  - No nonsense description of what they do

  - Clear and simple pricing

  - Simplicity as a core feature

  Big fan. Look forward to using your services in the future.

• by mfincham on 3/18/21, 10:45 PM

  My first experience with rsync.net was very disappointing. To this day they still advertise “append-only mode” support for restic at https://www.rsync.net/products/restic.html.

  Their support people confirmed it doesn’t work (though they didn’t seem to understand why it would be fine for them to support it as advertised...) yet 6 months later they still advertise that they support it, even when I have e-mailed to remind them (and it still doesn’t work either) :(

• by ChrisArchitect on 3/18/21, 7:41 PM

  I don't care for newsletters on tooling, but these Q&A interview posts are good -- immediately went in search of a twitter, couldn't find due to difficult naming, but want to follow to keep up from time to time

  https://twitter.com/consoledotdev

• by kplex on 3/18/21, 6:01 PM

  Is rsync.net related to rsync the project?

• by mattbillenstein on 3/19/21, 12:20 AM

  So how is this sharded? Or how do you load balance a customer to the correct server if there's no router?

• by aDfbrtVt on 3/18/21, 4:42 PM

  Thanks for the interview, I was pleasantly surprised to see how simple the network architecture is at rsync.

• by lokl on 3/18/21, 10:09 PM

  I wish I had a personal use case where the pricing of rsync.net made sense. It looks like a great service. For now, I use Backblaze Unlimited. I realize they are not the same service, but Backblaze works for my personal stuff and the price is great.

• by oilbagz on 3/19/21, 10:00 AM

  I bought an rsync.net account a few years back when John made it known on HN, and have used it solidly as a backup for my .. wristwatch!

  I have a LILYGO that I coded up a time-tracking app, which basically creates an event log whenever I tap it, wherever I go - and when Internet is available, it squirts the log over to some text files that live on rsync.net ..

  Pretty neat to be able to do this without much of a desktop or mobile phone in the way, I have to say. I wonder if there are more opportunities for this kind of IoT service out there .. it sure was fun to get this working without REST ..

• by jeffbee on 3/18/21, 7:39 PM

  I always liked this set of marketing materials. But I also see where they conflict with my experience. "You may visit our datacenters any time you like for a personal tour and inspection to satis[f]y whatever due diligence requirements you may have" probably appeals to many customers, but for my dollar I would prefer a datacenter that nobody may enter.

• by canoebuilder on 3/18/21, 6:28 PM

  With regard to the iOS import/export mentioned, does anyone have any more recommendations? (I'm not familiar with the mentioned option, nothing against it, just seeking out all options)

  Simple file system interface to all devices first, then any further software interfaces on top only if desired.

  Thanks for making the option available for remote storage John!

• by hertzrat on 3/18/21, 7:59 PM

  I used to run Linux for everything but I’m having to use Windows these days. What would it take to get rsync.net playing nicely with windows? I’m imagining Windows subsystem for Linux (ubuntu) with duplicity installed to it? Are there any major hiccups to that sort of setup?

• by shydwoo on 3/19/21, 5:23 AM

  Why would i buy 1TB for $20 per month here instead of getting 6TB for $8/month from Microsoft?

• by antongribok on 3/18/21, 5:56 PM

  Reading this takes me back to when I started playing with storage professionally.

  For me it was in 2004, also using 3Ware controllers. I was running on RedHat (before RHEL) and XFS before it was common on Linux, and similarly had memory issues when trying to repair filesystems.

• by simonebrunozzi on 3/20/21, 4:16 PM

  I like the interview a lot, but oddly this one here is the part that I liked the most:

  > I start the day with a short walk outdoors. I don’t want the first thing my eyes see to be print, and I don’t want the first thing my body does to be sitting. So I walk a bit.

• by ElectricMind on 3/19/21, 1:07 PM

  //I try to maintain an “Hedonic Fast” Monday through Wednesday so, on those days, I am only looking for truly actionable headlines and comment threads that are relevant to my businesse//

  This is smart move!

• by zarkov99 on 3/19/21, 2:25 AM

  For people who use rsync.net, is this something that can replace Dropbox for multi-machine synching? For all its flaws, Dropbox does allows me a semi-seamless transition between my laptop and my workstation.

• by peppermint_tea on 3/19/21, 12:45 AM

  happy customer here.

  I do a simple rsync of my precious but not too sensitive data, daily.

  and for the more sensitive stuff, gpg before sending daily as well, the copies will add up but I prefer it that way.

  10/10 great business

• by tyingq on 3/18/21, 6:07 PM

  I do get the "no separate firewall" reasoning, but I'm paranoid enough that I'd at least want some PF rules just in case some daemon gets started by accident.

• by crazypython on 3/19/21, 12:55 AM

  7 daily snapshots: So I could sync my hard drive over, wait for a snapshot, delete everything, and keep the space, and use the snapshot as backup?

• by KingOfCoders on 3/19/21, 5:19 AM

  Used them in a startup for a long time, was very happy, excellent support, good pricing. Would always use them again. (used the Swiss location).

• by poorman on 3/18/21, 10:38 PM

  "I initiate my work in the terminal by port-knocking".

  Guess you don't need a firewall when you have no open ports?

  Haha yes! Guess I'm not the only one...

• by yyyk on 3/19/21, 6:24 PM

  Question to rsync: Which HDs does rsync use? Is there a preferred brand? Which brands have been found to be most reliable?

• by vzaliva on 3/18/21, 11:41 PM

  Firefox Reade mode makes this page more readable. Otherwis 50% of your screen space is taken by non-informative graphics.

• by sideshowmel on 3/18/21, 6:14 PM

  Don't know if running a dumb switch connected to your ISP is the best infosec policy:

  https://blogs.cisco.com/manufacturing/the-top-5-reasons-to-a...

• by 1vuio0pswjnm7 on 3/18/21, 8:31 PM

  Would be interesting to see those shell scripts for sending SMS via Twilio.

• by nix23 on 3/19/21, 10:54 AM

  Clean, simple architecture...a sysadmin's dream.

• by bacbilla on 3/18/21, 8:52 PM

  +1 on having your laptop as an ephemeral device

• by pjs_ on 3/18/21, 9:39 PM

  rsync.net rules

• by sparkling on 3/18/21, 5:59 PM

  Hetzner has a similar product at better pricing that i have been using a minimalist dropbox alternative

  https://www.hetzner.com/en/storage/storage-box

  Access via rsync/sftp/scp

• by richardfey on 3/18/21, 9:06 PM

  I think they need to hire someone that is strong on the security side of the business, for two reasons:

  * he appears not aware of the role of hardware firewalls in mitigating DDoS by handling efficiently a lot of active TCP sessions (they have specialised hardware for this purpose)

  * he is describing in great detail a lot of information that a phisher or other type of hacker can treasure to target him