from Hacker News

Static IPs for Serverless Containers

by ekzhang on 12/2/24, 8:04 PM with 66 comments

  • by ekzhang on 12/2/24, 9:06 PM

    Hi! This is a blog post sharing some low-level Linux networking we're doing at Modal with WireGuard.

    As a serverless platform we hit a bit of a tricky tradeoff: we run multi-tenant user workloads on machines around the world, and each serverless function is an autoscaling container pool. How do you let users give their functions static IPs, but also decouple them from compute resource flexibility?

    We needed a high-availability VPN proxy for containers and didn't find one, so we built our own on top of WireGuard and open-sourced it at https://github.com/modal-labs/vprox

    Let us know if you have thoughts! I'm relatively new to low-level container networking, and we (me + my coworkers Luis and Jeffrey + others) have enjoyed working on this.

  • by jimmyl02 on 12/2/24, 9:41 PM

    this is a really neat writeup! the design choice to make each "exit node" control the local wireguard connections instead of a global control plane is pretty neat.

    an unfinished project I worked on (https://github.com/redpwn/rvpn) was a bit more ambitious with a global control plane and I quickly learned supporting multiple clients especially anything networking related is a tarpit. the focus on linux / aws specifically here and the results achievable from it are nice to see.

    networking is challenging and this was a nice deep dive into some networking internals, thanks for sharing the details :)

  • by qianli_cs on 12/3/24, 12:02 AM

    Thanks for sharing. This new feature is neat! It might sound a bit out there, but here's a thought: could you enable assigning unique IP addresses to different serverless instances? For certain use cases, like web scraping, it's helpful to simulate requests coming from multiple locations instead of just one. I think allowing requests to originate from a pool of IP addresses would be doable given this proxy model.

  • by heinternets on 12/3/24, 12:09 AM

    So much work seems to go into working around the limitations of IPv4 instead of towards a fully IPv6 capable world.

  • by cactacea on 12/2/24, 9:53 PM

    Static IPs for allowlists need to die already. Its 2024, come on, surely we can do better than this

  • by stuckkeys on 12/3/24, 1:44 AM

    This is just what I needed. Chefs kiss.

  • by ATechGuy on 12/2/24, 10:05 PM

    > Modal has an isolated container runtime that lets us share each host’s CPU and memory between workloads.

    Looks like Modal hosts workloads in Containers, not VMs. How do you enforce secure isolation with this design? A single kernel vulnerability could lead to remote execution on the host, impacting all workloads . Am I missing anything?

  • by klysm on 12/3/24, 1:26 AM

    Why is it important to have a static outbound ip address?

  • by handfuloflight on 12/3/24, 12:08 AM

    Do you block certain ports?

  • by fusjdffddddddds on 12/2/24, 11:25 PM

    It's going to take years for orgs to adopt IPv6 and mTLS+JWT/OIDC.

    Even longer for QUIC/H3.

  • by eqvinox on 12/2/24, 10:05 PM

    I guess my first question is, why is this built on IPv4 rather than IPv6...

  • by techn00 on 12/3/24, 8:30 AM

    side question: what do you use to make the diagrams?

  • by nodesocket on 12/2/24, 11:53 PM

    Couldn't a NAT instance in-front of containers accomplish this as well (assuming only needed for outbound traffic)? The open source project fck-nat[1] looks amazing for this purpose.

    [1] https://fck-nat.dev/stable/