from Hacker News

Citi’s $900M Misfire Happened During Software Switch

by whatok on 8/26/20, 12:20 PM with 52 comments

  • by jakub_g on 8/27/20, 7:58 AM

    The title might suggest that the rollout of new software was the issue, but the article states the very contrary: it was the old software that was the culprit:

    > An internal review at the bank found humans manually operating the old software were ultimately at fault

  • by LordAtlas on 8/27/20, 8:39 AM

    This is the key part:

    "But the employee didn’t select the correct system options -- instead allowing the loan to be repaid in full with interest. Colleagues who are supposed to catch such errors didn’t."

    Saved you a click.

  • by kmarc on 8/27/20, 1:01 PM

    I'm put on a banking project (as external) which already flushed down the toilet around ~$500M. Based on my experiences of the meetings and meetings about meetings, I totally understand how the incompetence lead to this clusterfuck.

    My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." He answered that although he is in the banking business for decades, he still doesn't know.

    These 100s of Millions of losses are not necessarily threatening core business. I find it amusing.

  • by neonate on 8/27/20, 4:44 AM

  • by brooklyndude on 8/27/20, 6:49 PM

    I worked at Citi for a very short time way back when. We were doing some things I thought were a bit “sketchy”, and was wondering if we were breaking the law.

    The response from my boss: we’re only breaking the law if we get caught, so theoretically we’re not actually breaking the law since no one has “caught us.”

    Guess there was some logic there. Of course this was a very long time ago. And sure they follow those pesky banking rules now , never, ever “breaking the law.”

  • by massaman on 8/27/20, 12:15 PM

    From the blameful-postmortem:

    Q. Why did we click `Send $900M?` A. Not sure. Tom felt all clicky-clicky, so he clicked it.

    Q. Why did we hire Tom? A. Also not sure. [Action Item: Fire Tom]

  • by massaman on 8/27/20, 12:19 PM

    Somewhere in your company hiring queue is a resume with:

    Strengths: Architecting risk management systems

    Weaknesses: Sometimes I click on things to see what happens

  • by totaldude87 on 8/27/20, 12:58 PM

    reminds me of that missile alerting system gif..

    https://giphy.com/gifs/emibob-ads-missile-warning-system-xUL...

  • by fooyc on 8/27/20, 11:52 AM

    The title is misleading, the switch doesn’t appear to have anything to do with the misfire.

  • by noahmbarr on 8/27/20, 11:50 AM

    I can only imagine a guy like larry ellison reacting to this article.

  • by jordache on 8/27/20, 3:22 PM

    stupid article. Reveals nothing of what contributed to the human error.

    They had been using the legacy software for many many years, w/o significant issues around human errors.

  • by scott31 on 8/27/20, 10:55 AM

    Expected, someone should have caught it in code review. Switch statements are generally harder to follow than if/else chain and fallthrough etc make it even more complicated.

  • by londons_explore on 8/27/20, 11:46 AM

    I assume financial software has the concept of a set of atomic transactions - ie. "debit bob $X and credit mary $X".

    Given that, presumably all buttons an operator clicks should generate a set of atomic transactions between customers and the bank.

    An automated system can then check that the total loss to the bank after these transactions have been executed isn't too big.

    I can't really imagine how any bit of software didn't have those checks in place...