by jo-m on 5/13/24, 7:27 AM
I had something similar once happen at my previous job. The company was using Google Workspace and GCP. The person who had set up GCP initially left the company. 1 month later, HR deleted his Google acc. Little we knew, the payment profile was attached to he GCP projects through his account, so deleting his account effectively removed the CC. All our stuff was shut down immediately (within minutes).
We first had no idea what was going on. GCP support just told us "it seems you deleted your CC". Eventually, we figured out what happened.
Set up a new payment profile and started migrating our GCP projects to it. Eventually had to create multiple of them, because there is an arbitrary quota of how many projects you can have per payment profile (~4), and support told us it would take days to increase it.
Fortunately, all our data was still there. However, support had initially told us that it's "all gone".
by dualscyther on 5/9/24, 5:08 AM
The title I'm seeing is
> Google Cloud accidentally deletes UniSuper’s online account due to ‘unprecedented misconfiguration’
which is a lot more alarming.
I've heard of sudden and unwarranted bans before, but never an accidental deletion of a customer who they only just convinced to migrate to Google Cloud last year!
by taspeotis on 5/9/24, 4:11 AM
> UniSuper was able to eventually restore services because the fund had backups in place with another provider.
Lesson learned: back your GCP data up to a real cloud like Azure or AWS.
by sidcool on 5/11/24, 3:43 AM
Google Cloud accidentally deleted a company's entire cloud environment (Unisuper, an investment company, which manages $80B). The company had backups in another region, but GCP deleted those too. Luckily, they had yet more backups on another provider.
by siquick on 5/9/24, 11:59 AM
by steve_taylor on 5/19/24, 7:08 AM
Whoever made the decision to use GCP should be fired. Google’s random deletion of accounts and their resources is well known. Somehow there wasn’t anyone in the whole organisation who knew about this risk or Google had convinced them it doesn’t happen to big players.
This article doesn’t challenge the assertion by Google that this is a once-off, which is really sloppy journalism.
by bananapub on 5/13/24, 10:06 PM
I really would like to hear the actual-actual story here, since it is basically impossible it actually was "Google let customer data be completely lost in ~hours/days". This is compounded by the bizarre announcements - UniSuper putting up TK quotes on their website, which Google doesn't publish and also doesn't dispute.
if a massive client came and said "hey our thing is completely broken", then there would have been a war room of SREs and SWEs running 24/7 over two continents until it wasn't.
by octodog on 5/9/24, 6:53 AM
Hopefully there is a full RCA published once the services are fully restored. This is really concerning for any GCP customer.
by dankotanko1599 on 5/21/24, 2:00 AM
It's just mind-boggling that their architecture allows this to happen so quickly IMO. There are so many resources and dependencies, that completely nuking a cloud account cannot and should not be easy or fast... and should not actually be possible by the cloud vendor.
I suppose they need to guard against anyone setting up costly infrastructure and doing a "runner" (allowing a credit card to lapse) - in that scenario - deleting all the customers data should be the absolute last resort - after it's been reasonably determined they are being malicious.
How does AWS manage these scenarios? I'm sure they follow-up multiple times before hitting the nuke button. In-fact - they know and treat their "larger accounts" with special privileges and assurances. Unisuper is not a small fish.
by ecliptik on 5/9/24, 6:06 PM
Did they think it was leftover Google Reader infra?
by rswail on 5/9/24, 4:53 AM
The choice of cloud operators is down to two. How is it that google can totally screw up so badly?
by dehugger on 5/17/24, 9:16 PM
My previous employer has this happen with OCI (Oracle's cloud). Hundreds of servers suddenly deleted do to a internal billing problem on Oracles side.
Luckily they were able to restore from backups, but it took a full day and there was still significant data loss (the delta since last backup).
Since those servers were used to host the companies own cloud managed offering it ended up affecting all of their own cloud customers as well.
by delduca on 5/14/24, 8:44 AM
I bet it was some stupid AI running in background.
by ein0p on 5/11/24, 6:42 PM
Pichai be like: I know how to fix this! Let’s fire a bunch of people and move their jobs to Bangalore. /s