from Hacker News

Removing PGP from PyPI

by dlor on 5/23/23, 2:09 PM with 187 comments

• by tzs on 5/23/23, 3:32 PM

  > In the last 3 years, about 50k signatures had been uploaded to PyPI by 1069 unique keys. Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19)

  Why not include the public key in the package?

  99% of the time what I want out of package signing is to know that the new version of the package I'm downloading is from the same people as the old version. I don't actually need to know who those people are...just that they are the same people as before.

• by Arnavion on 5/23/23, 2:39 PM

  >Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures.

  I don't know if it applies to any of those 1069 keys, but note that there is a way of hosting PGP keys that does not depend on key servers: WKD https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-s... . You host the key at a .well-known URI under the domain of the email address. It's a draft as you can see, but I've seen a few people using it (including myself), and GnuPG supports it.

• by usr1106 on 5/23/23, 3:07 PM

  Isn't that throwing out the baby with the bathwater? There seem to be non-neglible risks of installing malware from PyPI according to various headlines recently. But instead of improving security measures that don't work well they just remove them?

• by eduction on 5/23/23, 6:04 PM

  So they examined everything uploaded to PyPi with a signature over three years, including old versions, and classify those packages whose signing key is expired today, possibly years later, as "impossible to meaningfully verify." Never mind that the package may have been verifiable with a valid key for a full year or two before the key expired, and in the meantime may have been superseded by a newer version.

  They also say they can't "meaningfully verify" packages if the key does not have "binding identify information," by which they presumably mean automatically verifiable binding identity information, which usually means someone verified an email from keys.openpgp.org. This is a really narrow way to establish "binding identity information." For example someone who is a PyPi author and publicly links their PGP key from a (https) website on the same domain as the email on the key would not count. A well known longtime PyPi author with a well known key would not count.

  The ad hoc, out of band nature of how PGP keys are trusted is not remotely new - PyPi would have known from the very start of adopting PGP that many keys would not be automatically verifiable. It makes little sense to turn around now and act like this is some surprising thing.

  This has the smell of "we didn't want to bother supporting PGP any more because it's hard so we came up with an excuse."

  No need for an excuse, though: Just be honest about it and let the chips fall where they may, if you really don't want to support PGP. God knows there are valid reasons for not having the energy to deal with PGP. (FWIW I think it's a good solution for packages, for those who can navigate the tooling, but on the other hand I'm not volunteering my time to run PyPi.)

  P.S. There is a link in their post saying PGP has "documented issues." The specific issue described in the linked document is "packaging signing is not the holy grail" and a list of known things about PGP, like that verification of keys is ad hoc. It also concludes that there is no known better alternative.

• by rwmj on 5/23/23, 3:10 PM

  So they're removing PGP signatures, which certainly have some issues, and replacing them with ... nothing?

• by zzzeek on 5/23/23, 5:57 PM

  > Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19) 2.

  so...*reject those packages*. if you use a PGP key that isn't properly available or verifiable, reject it. That way every package with a PGP key will have 100% "key is properly discoverable" rate.

  it's not really reasonable to just drop this feature because most packages don't use it. Packages with tens of millions of downloads (like mine) make up a small percentage of total packages, but this small number of packages makes up a huge proportion of actual downloads, and package signing is most useful for these kinds of packages.

  if the adoption of "proper PGP keys" were ranked by packages/ downloads rather than "packages" alone, these rates would be much different.

• by jpgvm on 5/23/23, 6:54 PM

  I don't understand how Java can get this right with Maven Central and co but newer languages can't.

  Having a slight barrier to entry which is essentially "you must learn why signing is important for users of your library and this is how to do it", a) really isn't that bad and b) doesn't result in less quality packages being uploaded c) if it acts like any sort of filter that seems to be a good thing.

  Maven Central isn't short of high quality packages and no high quality OSS Java libraries are missing so the filter aspect isn't culling anything important.

  Java, Apt, RPM, etc all have this and have absolutely gigantic numbers of packages so the argument that it's too hard really just doesn't hold water.

  Doing so requires reading/understanding these ~3 pages of docs: https://central.sonatype.org/publish/requirements/gpg/

• by westurner on 5/23/23, 2:32 PM

  Now that you have removed GPG ASC signature upload support, is there any way for publishers to add cryptographic signatures to packages that they upload to pypi? FWIU only "the server signs uploads" part of TUF was ever implemented?

  Why do we use GPG ASC signatures instead of just a checksum over the same channel?

• by jwilk on 5/23/23, 3:39 PM

  Two days ago: https://news.ycombinator.com/item?id=36021172 ("PGP signatures on PyPI: worse than useless", >150 comments)

• by WhyNotHugo on 5/23/23, 4:52 PM

  When many developers didn't use 2FA they pushed for them to enable 2FA within a deadline. It sounds like the same approach could have been used for PyPI. E.g.: an attempt to make the feature useful before declaring it dead forever.

• by bryanlyon on 5/23/23, 4:59 PM

  I came here thinking they were removing the PGP package from PyPi, but they're just removing a barely-used signature system? I don't know why they have to remove it though. I doubt it requires much maintenance now that it's already in place.

  Even if only 37% of keys are verifiable, that's infinitely more than will be verifiable if they remove the PGP support.

• by zokier on 5/23/23, 4:30 PM

  At least you can't blame pypi for ignoring the report, and tbh I find this response time remarkably quick. It wouldn't have been far fetched to imagine someone in their position just trying to ignore/downplay/dispute this sort of reports.

• by jxy on 5/23/23, 7:32 PM

  I don't understand the argument. Isn't the whole point of PGP establishing some kind of chain of trust? If pypi.org has it's public key, it could sign a few major distributors's keys, and for smaller/individual packages I could either choose to always trust the same public key or don't use the package. It's not a centralized system to begin with. It's not pypi.org's responsibility to identify and verify all the keys belong to who say they belong. Pypi.org's unable to verify individual identities shouldn't impact the overall usefulness of the PGP for package distribution and verification.

• by NotYourLawyer on 5/23/23, 5:21 PM

  Interesting timeline. The Yossarian article that TFA cites and that I assumed was the impetus here was published two days ago on 5/21. But the audit was two days earlier on 5/19.

• by reidrac on 5/23/23, 4:00 PM

  I have been thinking about this in the context of Java libraries (really using Scala, but bear with me).

  If the repo requires a GPG signature, they could also ask for the public key of the developer making the releases (e.g. when they make the account), and they could sign it with their key at that point.

  Then make available the package, the signature, and the signed public key. Then I only need to trust the repo's key (in this case PyPi).

  Does this make any sense?

• by KRAKRISMOTT on 5/23/23, 2:50 PM

  What are we switching to? Does Pypi support ECDSA?

• by sacnoradhq on 5/24/23, 5:08 AM

  So how are Python packages signed? Are they just shipping rando code without any sort of E2E assurance?

  FWIW, Ruby also did a piss-poor job of handling gem signing by making it both difficult and optional.

  How fucking hard is it to get to the level of code release assurance as Debian or Fedora? Manage GPG keys, signfest them, and enforce a policy.

• by rvz on 5/23/23, 5:11 PM

  PGP is a solution in search of a problem. We have given it decades for it to be useful and it turns out that it is an enormous security failure. It needed to go.

  Sigstore [0] on the other hand makes more sense to use instead of problem.

  [0] https://www.sigstore.dev

• by 7to2 on 5/24/23, 6:44 AM

  Trust on first use is absolutely a valid use of PGP signatures that is being used in many real world systems (ask me how I know). You finding that PGP isn't being used they way you think it should does not justify removing it without providing a replacement.

  Why on earth wasn't the community asked before you implemented this change?

  > Given all of this, the continued support of uploading PGP signatures to PyPI is no longer defensible. While it doesn't represent a massive operational burden to continue to support it, it does require any new features that touch the storage of files to be made aware of and capable of handling these PGP signatures, which is a non zero cost on the maintainers and contributors of PyPI.

  This uninformed reasoning is what's indefensible.

• by forgotmypw17 on 5/23/23, 7:28 PM

  What an amazing opportunity for someone to add a new way of integrating PGP authentication by writing two short scripts:

  One to compile a list of file hashes and PGP-sign them.

  One to validate these hashes against the provided signatures.

• by lgxz on 5/23/23, 8:14 PM

  Replace a 31% effective solution with no solution? very impressive