from Hacker News

How ML Model Data Poisoning Works in 5 Minutes

by R41 on 3/24/24, 3:15 PM with 15 comments

  • by fxtentacle on 3/24/24, 5:23 PM

    "How ML Model Data Poisoning Works"

    It doesn't. The mentioned Nightshade tool is useless. Does anyone have any example of successful model data poisoning?

  • by Eisenstein on 3/24/24, 5:30 PM

    None of the cases of data poisoning it presented seemed effective in doing very much, except the MS case, and that was so flawed in implementation that it was a example of how not to deploy something.

    > Developers need to limit the public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production.

    Haven't we learned that more eyes to find flaws is better than locking things down?

  • by bee_rider on 3/24/24, 6:29 PM

    > In 2016, Microsoft released their chatbot named Tay on Twitter to learn from human interactions by posting comments. But after the release, it started to act crazy.

    > It started using vulgar language and making hateful comments. This was one of the first incidents of data poisoning.

    Is this true? I remember when this happened but I thought the story was that 4chan basically found an “echo” type debug command or something like that. The ML mode wasn’t being trained to say bad things, it was just being sent some kind of repeat-after-me command and then the things it was told to repeat were bad.

    It seems odd that somebody would write a whole blog post without bothering to check that, though, so maybe I’m mis-remembering?

  • by thesz on 3/24/24, 11:13 PM

    I recently made a comment that neural models cannot provide chain of reason, while symbolic methods can: https://news.ycombinator.com/item?id=39759033

    The vulnerability in the post is directly linked to that inability, in my opinion.

  • by stanleykm on 3/24/24, 6:54 PM

    When these articles pop up on HN at least there seems to be a lot of focus on training poisoning. While intellectually interesting, it seems less useful or practical than defeating inference.

  • by sonorous_sub on 3/24/24, 7:36 PM

    how to train self-smashing looms