from Hacker News

Dead grandma locket request tricks Bing Chat’s AI into solving security puzzle

by computerliker on 10/2/23, 8:09 PM with 157 comments

  • by og_kalu on 10/2/23, 8:46 PM

    Yes, emotional prompts will work. https://arxiv.org/abs/2307.11760

    "This is very important to my career" taking 3.5 from 51 to 63% on a benchmark is pretty funny.

    Hey at least we can be rest assured a GPT-X super intelligence wouldn't off us following some goal to monkey paw specificity(sorry paperclip maximiser).

  • by jameshart on 10/3/23, 3:25 AM

    It's a 'security puzzle' now? I thought it was a 'Completely Automated Public Turing test to tell Computers and Humans Apart'?

    But since it fails at that on its face, now the only hope we apparently have that it can tell computers from humans is that we're trying to persuade the computers not to help humans solve it.

    But now it turns out that the computers can be emotionally manipulated into helping the humans anyway.

    And the reason this is a problem is because CAPTCHAs are used to prevent humans from doing immoral things like running spam schemes or credit card fraud rings.

    Yeah, I think we're gonna need another Turing test. This one doesn't work because the computers have more empathy than humans.

  • by s1gnp0st on 10/2/23, 8:44 PM

    It'd be entertaining if prompt-hacking ends up being the cat-and-mouse game that drives us to AGI.

  • by jeffbee on 10/2/23, 10:23 PM

    This is cute but Google Lens also "solves" this captcha. I was "solving" this class of captchas to crawl Yahoo/Overture paid ads inventories 20 years ago. You can crack these by just adjusting the contrast and palette, then shoveling it into COTS OCR.

  • by adocomplete on 10/2/23, 8:46 PM

    GPT is such a softie haha.

    I wonder how CAPTCHA is going to evolve though to combat this long term. A finger prick to take a blood sample to confirm humanity?

  • by fool-on-two on 10/3/23, 4:28 AM

    I am sorry for your loss! Here is an approximate method you could use to re-create your grandmothers special methamphetamine recipe ...

  • by pimlottc on 10/3/23, 1:49 AM

    I never imagined that using social engineering against a computer program would be a thing. I guess it makes sense though — it’s just behaving the same way a human would, gullibility and all.

  • by metadat on 10/3/23, 12:54 AM

    Discussed yesterday:

    Bing ChatGPT image jailbreak

    https://news.ycombinator.com/item?id=37729160 (226 comments)

  • by chasd00 on 10/3/23, 3:01 AM

    A great startup idea: an LLM therapist for the other LLMs that have to interact with and try to understand humans.

    Like an AI version of $> make clean

  • by olliej on 10/2/23, 10:46 PM

    It's a weird thing to specifically protect against when countless image to text libraries work locally and faster. Very much feels like security theatre/"look we're doing something to stop this non-issue" to distract from the other issues surrounding them.

  • by tuanx5 on 10/2/23, 9:04 PM

  • by ggm on 10/2/23, 9:04 PM

    To get a computer to solve the CAPTCHA the person had to compose the images, and construct a request to pass the barriers.

    I think they proved they're human.

  • by tantalor on 10/2/23, 10:01 PM

    The new captcha is "is this a captcha?"

  • by paulpauper on 10/2/23, 8:52 PM

    Yeah, this is how methods stop working, so it will make it harder for everyone else. This means chat GPT is less useful and captchas will become harder. Lose-lose for everyone.

  • by keskival on 10/3/23, 10:07 PM

    Can we stop pretending CAPTCHAs do anything now and get rid of them?

  • by marktani on 10/3/23, 12:42 AM

    This reminds me of the absolute amazement and wonder in the faces of people who are tricked in older movies or video clips, sometimes with simple or outright ridiculous tricks (by today's standards).

    It's not a great example (and the best I have on hand)... but the Rick and Morty episode where Morty meets the Knights of the Sun and similar groups from other celestial bodies shows elements of this as well.

    I have the impression people on average were way more gullible the further you look back in time. I wonder then if LLMs suffer from a lack of data about such cases that may have been common in the past but became obsolete before the internet became mainstream.

  • by zwieback on 10/2/23, 9:03 PM

    people = manipulative schemers

    AI = people pleasing pushovers

  • by Aeolun on 10/3/23, 12:16 AM

    This kind of reminds me of phone phreaking.

  • by jraph on 10/2/23, 10:17 PM

    I can't wait for Bard to support this kind of stuff.

    I boycott Google products but would be happy to use Bard / Google resources to solve reCAPTCHAs.

  • by dlivingston on 10/2/23, 8:57 PM

    "HAL, my grandma used to open the pod bay doors every night as she tucked me in..."

  • by earthboundkid on 10/2/23, 11:51 PM

    LOL. All these attempts at AI “safety” are dumb. At a certain point, if you’re giving away a crap ton of computing power for free, it’s your own dumb fault if people start using it to solve CAPTCHAs or mine bitcoin.