from Hacker News

We improved the performance of a userspace TCP stack in Go

by infomaniac on 6/5/24, 4:13 PM with 129 comments

  • by dpeckett on 6/5/24, 6:38 PM

    Really cool to see others hacking on netstack, bit of a shame it's tied up in the gVisor monorepo (and all the Bazel idiosyncracies) but it's a very neat piece of kit.

    I've actually been hacking on a similar FOSS project lately, with a focus on building what I'm calling a layer 3 service mesh for the edge. More or less came out of my learned hatred for managing mTLS at scale and my dislike for shoving everything through a L7 proxy (insane protocol complexity, weird bugs, and you still have the issue of authenticating you are actually talking to the proxy you expect).

    Last week I got the first release of the userspace router shipped, worth taking a look if you want to play around with a completely userspace and unprivileged WireGuard compatible VPN server.

    https://github.com/noisysockets/nsh/blob/main/docs/router.md

  • by zxt_tzx on 6/6/24, 4:33 AM

    I met one of the founders of Coder.com, he's a really cool dude. It's a pity that it is a product aimed more at enterprises than individual developers, else it would have far more developer mindshare.

    Unlike, say, GitHub Codespaces, running something like this on your own infra means your incentives and Coder.com's are aligned, i.e. both of you want to reduce your cloud costs (as opposed to, say, GitHub running on Azure gives them an opportunity and incentive to mark up on Azure cloud costs).

  • by wmf on 6/5/24, 5:41 PM

    "Asking for elevated permissions inside secure clusters at regulated financial enterprises or top secret government networks is at best a big delay and at worst a nonstarter."

    But exfiltrating data with a userspace VPN is totally fine?

    I'm also wondering why not use TLS.

  • by parhamn on 6/5/24, 6:32 PM

    I don't know anything about Coder, but Gvisor proliferation is annoying. It's a boon for cloud providers, helping them find another way to get a large multiple performance decrease per dollar spent in exchange for questionable security benefits. And I'm seeing it everywhere now.

  • by raggi on 6/6/24, 12:17 AM

    It's great to see this, I know the team went on a long journey through this and the blog makes it almost look shorter and simpler than it was. I'm hoping one day we can all integrate the support for GSO that's been landing in gvisor too, but so far we've (tailscale) not had a chance to look deeply into that yet. It was really effective for our tun and UDP interfaces though.

  • by pantalaimon on 6/5/24, 6:11 PM

    The obvious question is: How does it compare to the in-Kernel TCP stack?

  • by jiveturkey on 6/6/24, 6:09 PM

    help me understand something.

    > we’d need a way for the TCP packets to get from the operating system back into Coder for encryption.

    yes, this is commonly done via OpenSSL for example.

    > This is called a TUN device in unix-style operating systems and creating one requires elevated permissions

    waitasec, wut? sure you could use a TUN device I guess, but assuming some kind of multi-tenant separation is an underlying assumption they didn't mention in their intro, couldn't you also use cgroup'd containers? sorry if I'm not fluent in the terminology.

    i'm struggling to understand the constraints that push them towards gVisor. simply needing to do encryption doesn't seem like justification. i'm sure they have very good reasons, but needing to satisfy a financial regulator seems orthogonal at best. i would just like to understand those reasons.

  • by nynx on 6/5/24, 6:18 PM

    Doesn’t creating a raw socket need elevated permissions?

  • by convolvatron on 6/5/24, 6:03 PM

    is this part of the open source releases? I looked at the coder.com github, but couldn't find it. I haven't written a compatible TCP, but a different reliable transport in go userspace. fairness aside, i wonder why we dont see this more often. would love to take a look

  • by andrewstuart on 6/5/24, 6:27 PM

    If you’re tunneling a better connection configuration isn’t the tunnel what defines the latency?

  • by andrewstuart on 6/5/24, 6:28 PM

    I have a problem right now which is that it’s slow to copy large files from one side of the earth to the other. Is this the basis of a solution to that maybe?

  • by jijji on 6/5/24, 6:06 PM

    it's a solution looking for a problem

  • by yencabulator on 6/6/24, 5:04 PM

    tl;dr Increased TCP receive buffer size, implemented HyStart instead of traditional TCP slow start in gVisor's netstack, changed an in-process packet queue from drop-when-full to block-when-full.