from Hacker News

Fearless SSH: Short-lived certificates bring Zero Trust to infrastructure

by mfrw on 10/23/24, 9:44 AM with 160 comments

  • by edelbitter on 10/23/24, 10:46 PM

    Why does the title say "Zero Trust", when the article explains that this only works as long as every involved component of the Cloudflare MitM keylogger and its CA can be trusted? If hosts keys are worthless because you do not know in advance what key the proxy will have.. than this scheme is back to trusting servers merely because they are in Cloudflare address space, no?
  • by tptacek on 10/23/24, 11:24 PM

    I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA. When I'm selling people on SSH CA's, the first thing I usually have to convince them of is that I'm not saying they should trust some third party. You know where all your servers are. External CAs exist to solve the counterparty introduction problem, which is a problem SSH servers do not have.
  • by mdaniel on 10/23/24, 9:39 PM

    I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For context, I'm also one of those "immutable OS or GTFO" chaps because in my experience the next thing that happens after some rando ssh-es into a machine is they launch vi or apt-get or whatever and now it's a snowflake with zero auditing of the actions taken to it

    I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again

  • by TechnicalVault on 10/24/24, 10:29 AM

    The whole MITM just makes me deeply uncomfortable, it's introducing a single point of trust with the keys to the kingdom. If I want to log what someone is doing, I do it server side e.g. some kind of rsyslog. That way I can leverage existing log anomaly detection systems to pick up and isolate the server if we detect any bad behaviour.
  • by antoniomika on 10/23/24, 10:23 PM

    I wrote a system that did this >5 years ago (luckily was able to open source it before the startup went under[0]). The bastion would record ssh sessions in asciicast v2 format and store those for later playback directly from a control panel. The main issue that still isn't solved by a solution like this is user management on the remote (ssh server) side. In a more recent implementation, integration with LDAP made the most sense and allows for separation of user and login credentials. A single integrated solution is likely the holy grail in this space.

    [0] https://github.com/notion/bastion

  • by shermantanktop on 10/23/24, 11:27 PM

    I didn’t understand the marketing term “zero trust” and I still don’t.

    In practice, I get it - a network zone shouldn’t require a lower authn/z bar on the implicit assumption that admission to that zone must have required a higher bar.

    But all these systems are built on trust, and if it isn’t based on network zoning, it’s based on something else. Maybe that other thing is better, maybe not. But it exists and it needs to be understood.

    An actual zero trust system is the proverbial unpowered computer in a bunker.

  • by blueflow on 10/24/24, 11:16 AM

    Instead of stealing your password/keypair, the baddies will now have to spoof your authentication with cloudflare. If thats just a password, you gained nothing. If you have 2FA set up for that, you could equally use that for SSH directly, using a ssh key on a physical FIDO stick. OpenSSH already has native support for that (ecdsa-sk and ed25519-sk key formats).

    The gain here is minimal.

  • by keepamovin on 10/24/24, 4:56 AM

    Does this give CloudFlare a backdoor to all your servers? That would not strictly be ZT, as some identify in the comments here.
  • by johnklos on 10/23/24, 11:11 PM

    So... don't trust long lived ssh keys, but trust Cloudflare's CA. Why? What has Cloudflare done to earn trust?

    If that alone weren't reason enough to dismiss this, the article has marketing BS throughout. For instance, "SSH access to a server often comes with elevated privileges". Ummm... Every authentication system ever has whatever privileges that come with that authentication system. This is the kind of bull you say / write when you want to snow someone who doesn't know any better. To those of us who do understand this, this is almost AI level bullshit.

    The same is true of their supposed selling points:

    > Author fine-grained policy to govern who can SSH to your servers and through which SSH user(s) they can log in as.

    That's exactly what ssh does. You set up precisely which authentication methods you accept, you set up keys for exactly that purpose, and you set up individual accounts. Do Cloudflare really think we're setting up a single user account and giving access to lots of different people, and we need them to save us? (now that I think about it, I bet some people do this, but this is still a ridiculous selling point)

    > Monitor infrastructure access with Access and SSH command logs

    So they're MITM all of our connections? We're supposed to trust them, even though they have a long history of not only working with scammers and malicious actors, but protecting them?

    I suppose there's a sucker born every minute, so Cloudflare will undoubtedly sell some people on this silliness, but to me it just looks like yet another way that Cloudflare wants to recentralize the Internet around them. If they had their way, then in a few years, were they to go down, a majority of the Internet would literally stop working. That should scare everyone.

  • by EthanHeilman on 10/23/24, 9:38 PM

    I'm a member of the team that worked on this happy to answer any questions.

    We (BastionZero) recently got bought by Cloudflare and it is exciting bringing our SSH ideas to Cloudflare.

  • by WesolyKubeczek on 11/4/24, 7:52 AM

    I can’t wait for a bug to happen when you authenticate correctly but unexpectedly slide into someone else’s network.
  • by nonameiguess on 10/24/24, 1:19 PM

    Basic summary seems to be:

    * This has nothing to do with zero-trust. If you already require pubkey auth to every connection made to a server regardless of origin, that's already meeting the definition of zero trust.

    * What this actually gives you is a solution to the problem of centrally revoking long-lived keys by not having any and instead using certificate auth. Now the CA is the only long-lived key.

    * This is a reasonable thing large orgs should probably do. There is no reason the CA should be an external third-party like Cloudflare, however.

    * This also integrates with existing SSO providers so human users can be granted short-lived session certs based on whatever you use to authenticate them to the SSO provider. Also reasonable, also no reason this should be offered as a service from Cloudflare as opposed to something you can self-host like Kerberos.

    * This also provides ssh command logging by proxying the session and capturing all commands as they get relayed. Arguably not a bad idea in principle, but a log collector like rsyslogd sending to an aggregator accomplishes the same thing in practice, and again, I would think you'd want to self-host a proxy if you choose to go that route, not rent it from Cloudflare.

    All in all, good things a lot of orgs should do, but they should probably actually do. I get the "well, it's hard" angle, but you're usually looking at large, well-funded orgs when you're talking things like SOC and FedRamp compliance. If you want to be a bank or whatever, yeah, that's hard. It's supposed to be. As I understand it, at least part of the spirit of SOC and FedRamp and the like is your organization has processes, plans, procedures, and personnel in place with the expertise and care to take security seriously, not "we have no idea what any of this means, why it matters, and don't have the time, but we pay a subscription fee to Cloudflare and they say they take care of it."

  • by andriosr on 10/24/24, 4:12 PM

    hoopdev here. Zero trust for SSH is just table stakes these days. Real challenge is getting devs to actually adopt better practices without the tooling getting in their way.

    Found in practice that certs > keys but you need to think beyond just SSH. Most teams have a mix of SSH, K8s, DBs etc. Using separate tools for each just creates more headache.

    Haven't tried Boundary but Teleport/hoop/Tailscale all handle the mixed protocol issue decently. Main difference is hoop focuses more on protocol-level DLP and automated reviews vs pure network access. Horses for courses though, they're all valid approaches.

    Key is picking something devs will actually use vs work around. Nothing worse than a "secure" solution that drives people to create workarounds.

  • by curben on 10/24/24, 2:49 AM

    Cloudflare has been offering SSH CA-based authentication for more than 2 years [1], I wrote a guide back in feb '23 [2]. The announcement is more about offering new features, such as more granular user control.

    [1]:https://web.archive.org/web/20210418143636/https://developer...

    [2]: https://mdleom.com/blog/2023/02/13/ssh-certificate-cloudflar...

  • by singhrac on 10/24/24, 2:32 PM

    I get that HN does not like Cloudflare and does not like the term “Zero Trust”, but geez these comments are repetitive. Can anyone compare to Tailscale SSH? Are they basically offering an (even more) enterprise version of Tailscale’s product line?
  • by cyberax on 10/23/24, 11:29 PM

    Hah. I did pretty much all the same stuff in my previous company.

    One thing that we did a bit better: we used AWS SSM to provision our SSH-CA certificates onto the running AWS EC2 instances during the first connection.

    It would be even better if AWS allowed to use SSH CA certs as keys, but alas...

  • by dmuth on 10/24/24, 4:41 PM

    Using CAs and signed certificates in SSH is definitely the way.

    If anyone wants to play around with that, without the risk of locking themselves out of a server, I built a little "playground" awhile back whihc is a series of Docker containers that can SSH to each other. Give it a try at https://github.com/dmuth/ssh-principal-and-ca-playground

    (I haven't touched the project in awhile, so if there are any issues, please open an Issue and I'll gladly look at it!)

  • by arianvanp on 10/24/24, 12:38 AM

    Zero trust. But they don't solve the more interesting problem: host key authentication.

    Would be nice if they can replace TOFU access with SSH CA as well. Ideally based on device posture of the server (e.g. TPM2 attestation)

  • by amar0c on 10/24/24, 9:30 AM

    Is there anything similar ("central point of SSH access/keys management" ) that is not Cloudflare ? I know about Tailscale and it's SSH but recently it introduced so much latency (even tho they say it's P2P between A and B) that is unusable.

    Ideally something self hosted but not hard requirement

  • by nanis on 10/24/24, 10:58 AM

    > the SSH certificates issued by the Cloudflare CA include a field called ValidPrinciples

    Having implemented similar systems before, I was interested to read this post. Then I see this. Now I have to find out if that really is the field, if this was ChatGPT spellcheck, or something else entirely.

  • by udev4096 on 10/24/24, 10:42 AM

    > You no longer need to manage long-lived SSH keys

    Well, now you are managing CAs. Sure, it's short lived but it's not different than having a policy for rotating your SSH keys

  • by INTPenis on 10/24/24, 7:19 AM

    Properly setup IaC, that treats Linux as an appliance instead, could get rid of SSH altogether.

    I'm only saying this because after 20+ years as a sysadmin I feel like there have been no decent solutions presented. On the other hand, to protect my IaC and Gitops I have seen very decent and mature solutions.

  • by anilakar on 10/24/24, 7:08 AM

    Every now and then a new SSH key management solution emerges and every time it is yet another connection-terminating proxy and not a real PKI solution.
  • by koutsie on 10/24/24, 11:51 AM

    How is trusting Cloudflare "zero-trust" ?
  • by advael on 10/24/24, 1:51 AM

    You know you can just do this with keyauth and a cron job, right?
  • by c-linkage on 10/24/24, 12:06 PM

    Welcome to Kerberos[0] over HTTP.

    [0] https://www.geeksforgeeks.org/kerberos/

  • by xyst on 10/23/24, 11:27 PM

    Underlying tech is “Openpubkey”.

    https://github.com/openpubkey/openpubkey

    BastionZero just builds on top of that to provide a “seamless” UX for ssh sessions and some auditing/fedramp certification.

    Personally, not a fan of relying on CF. Need less centralization/consolidation into a few companies. It’s bad enough with MS dominating the OS (consumer) space. AWS dominating cloud computing. And CF filling the gaps between the stack.

  • by rdtsc on 10/24/24, 8:36 AM

    By “ValidPrinciples” did they mean “ValidPrincipals”?

    And by ZeroTrust they really mean OneTrust: trust CF. A classic off-by-one error :-)