from Hacker News

Google Cloud accidentally deletes customer's account

by frays on 5/9/24, 9:32 PM with 32 comments

  • by j45 on 5/9/24, 9:56 PM

    The cloud is someone else's computer. Convenience isn't always reliability and security.

    For anyone who is semi-technical, or not as technical as they wished when it came to file storage wondering what they can do...

    - Whether it's been corporate clients, small business, or individuals: I universally recommend everyone owning a small QNAP or Synology that is storage as a zero maintenance appliance, running software to maintain a 2 way sync of your cloud drives.

    - Even if you're using Google Cloud, MSFT, etc, continue to use it as you please, just siphon off a local data backup in case the internet or the cloud is down. It also can make some kinds of disaster recovery much quicker. For example if you back up your computers locally to a NAS, and then from there to the cloud, it can be a lot more manageable.

    - Throwing something like Tailscale on it makes it invisible and hyper secure to have on all your devices too. I typically never use the NAS' network connection tools, as it's likely a juicy target to break.

    - Last but not least, setup a different format of backup, automatically. Backup your file appliance to something like sync.com, tarsnap.com, backblaze, etc to back up elsewhere that you can access.

    I'd be happy to learn what anyone else is doing. As someone who lost a ton of data on a Microsoft account once, never twice.

    *I am perfectly capable of building a NAS myself running an open source package, but storage should be an appliance at home to focus on other things.

  • by auspiv on 5/9/24, 9:53 PM

    The customer isn't exactly small either - "UniSuper is an Australian superannuation fund that provides superannuation services to employees of Australia's higher education and research sector. The fund has over 620,000 members and $120 billion in assets (funds under management and total member accounts at 7 July 2021)."[1]

    https://en.wikipedia.org/wiki/UniSuper

  • by doctor_eval on 5/9/24, 10:02 PM

    > In the meantime, UniSuper's woes remain a lesson for companies leaping cloudwards. Someone clicking the wrong button, a previously unknown bug, an unforeseen series of events…

    Here’s the thing though. Back in the Dark Ages before cloud services, everyone had to self host. We had a Data General AViiON server (DG/UX FTW!) in a dedicated room, and one of the first RAID arrays in Australia (predecessor of CLARiiON).

    The cover was off the front of the array for some reason, and I had to squeeze past a coworker to get out of the room.

    Sitting down at my Wyse60 terminal to do some work, a bunch of errors started appearing on my screen. Turns out, I had also “squeezed past” the power button on the RAID array, which was normally recessed - but not when the cover was off. I’d inadvertently shut down the whole system. Fortunately we were in preproduction so nobody really noticed. But it scared the crap out of me.

    I knew someone at a small bank who told me they were also susceptible to similar problems. Just one big server, let’s hope it doesn’t go down.

    Cloud services - especially IaaS - _enable_ diversification, and it sounds like UniSuper’s IT team should be congratulated for understanding what this really means in the context of networked services. Diverse networks, diverse suppliers, diverse geography.

    Without cloud services, none of this is feasible for most SMEs.

    There are plenty of things we can complain about with the cloud but “someone clicking the wrong button” is even more of a risk if you run your gear in house.

  • by clvx on 5/9/24, 9:54 PM

    Whoever decided to put the data in a different cloud provider needs a raise and all the praise.
  • by thih9 on 5/9/24, 9:51 PM

    This is the website of the affected customer and their incident page:

    "We're progressively restoring UniSuper services", https://www.unisuper.com.au/contact-us/outage-update

    First entry is from May 2nd, indicating total outage. Today they enabled user sign ins and still write that "services will continue to be progressively brought online".

  • by hobotime on 5/9/24, 10:24 PM

    We provide IT support for small businesses, and it's almost impossible for them to understand that their 'cloud' data isn't backed up.

    It takes a lot of education combined with making the backups not cost too much.

  • by rsync on 5/9/24, 9:51 PM

    "Fortunately, UniSuper had backups at another cloud provider. Otherwise, a bad situation could have been oh so much worse."

    Years ago we ran ad campaigns on reddit that said something like:

    "Your data is stored on AWS and your backups are stored on AWS ... you're doing it wrong."

    ... and they got almost zero traction.

    In fact, many people were angered by the suggestion that data at a major cloud provider could be at risk in any way.

  • by belter on 5/9/24, 10:00 PM

    Sounds like Google moved to the next level, from deprecating services to deleting customer accounts. Next....Chrome will accidentally use your PC for running distributed generative AI workloads.
  • by enjaydee on 5/9/24, 11:28 PM

    I'm not sure how this isn't being news here!
  • by xyst on 5/9/24, 9:54 PM

    Cascade delete gone wrong
  • by blibble on 5/9/24, 10:09 PM

    your account still has 99.99999% durability, stop complaining
  • by esoterae on 5/9/24, 9:49 PM

    Enterprise