from Hacker News

Researchers tested AI watermarks and broke all of them

by adg29 on 10/4/23, 4:32 PM with 87 comments

  • by jacobr1 on 10/6/23, 9:33 PM

    We need to focus on the other direction. How can we have chains of trust for content creation, such as for real video. Content can be faked, but not necessarily easily faked from the same sources that make use of cryptographic signing. The attacks can sign the own work, so you'd need ways to distinguish those cases, but device level keys, organizational keys, distribution keys all can provide provenance chains that can be used by downstream systems to _better_ detect fraud, though not eliminate it.

  • by obblekk on 10/6/23, 9:22 PM

    For written text, the problem may be even harder. Identifying the human author of text is a field called "stylometry" but this result shows that some simple transformations reduce the success to random chance [1].

    Similarly, I suspect watermarking LLM output is probably unworkable. The output of a smart model could be de-watermarked by fine tuning a dumb open source model on the initial output, and then regenerating the original output token by token, selecting alternate words whenever multiple completions have close probabilities and semantically equivalent. It would be a bit tedious to perfectly dial in, but I suspect it could be done.

    And then ultimately, short text selections can have a lot of meaning with very little entropy to uniquely tag (e.g., covfefe).

    [1] https://dl.acm.org/doi/abs/10.1145/2382448.2382450

    Curious if Scott Aaronson solved this challenge...

  • by great_psy on 10/6/23, 9:22 PM

    It seems it would be much easier to watermark non-ai images instead. Aka crypto signature.

    That will be much harder to evade, but also pretty hard to implement.

    I guess we will end up in the middle ground, where any non-signed image could be ai generate, but for most day to day use it’s ok.

    If you want something to be deemed legit (gov press release, newspaper photo, etc) then just sign it. Very similar to what we do for web traffic (https)

  • by brap on 10/6/23, 9:04 PM

    People have been trying to watermark digital media for decades, when there was (still is) a very strong financial incentive to get it working. It never worked. I don’t think it ever will work.

  • by epivosism on 10/6/23, 9:46 PM

    Wasn't this obvious from the get go that this can't work?

    If AI will eventually generate say 10k by 10k images, I can resize to 2.001k by 1.999k or similar, and I just don't get how any subtle signal in the pixels can persist through that.

    Maybe you could do something at the compositional level, but that seems restrictive to the output. Maybe something about like larger regions average color balance or something? But you wouldn't be able to fit many bits in there, especially when you need to avoid triggering accidentally.

    Also: here are some play money markets for whether this will work:

    https://manifold.markets/Ernie/midjourney-images-can-be-effe...

    https://manifold.markets/Ernie/openai-images-have-a-useful-a...

  • by KaiserPro on 10/6/23, 9:37 PM

    We already have well established systems to prove the provenance of images and other sources.

    At the moment the internet is a wash with bullshit images. Its imperative that news outlets are at a high enough standard to actually prove the provenance of them.

    You don't trust some bloke off facebook asserting that something is true, its the same for images.

  • by 998244353 on 10/7/23, 12:16 AM

    The actual paper seems to be https://arxiv.org/abs/2310.00076.

  • by skilled on 10/6/23, 8:34 PM

  • by rakkhi on 10/6/23, 9:20 PM

    It’s like captcha, highly annoying to users and authors, but if you don’t want to pay it works against low spend bots

  • by whywhywhywhy on 10/7/23, 11:01 AM

    I’ll never get over the “invisible_watermark” Python package being entirely visible to the naked eye, obviously degrades the image in an way that’s unacceptable and even easily spottable on any image once you know what it looks like.

  • by natch on 10/6/23, 11:26 PM

    Who was it, Eric Schmidt, who said we need to get over it, there is no privacy? I feel like we have the same energy here for authenticating human origin of content.

  • by TestingTest5 on 10/6/23, 10:39 PM

    Was only a matter of time anyways...

  • by bulla on 10/7/23, 12:50 AM

    What happened to C2PA?