Author | Li Dongmei
620,000 members can't access their fund accounts, UniSuper complains 'it's all to blame on Google Cloud'
Last week, Australian superannuation giant UniSuper experienced a server outage, leaving members unable to access the server online or log in to the mobile app due to "a rare issue with Google Cloud that resulted in an inadvertent configuration error during the configuration of the UniSuper private cloud, affecting the superannuation fund's secondary system". To put it bluntly, it was the fund's Google Cloud accounts that were massively deleted, and Google Cloud also deleted UniSuper's backup data elsewhere.
If it weren't for the backups on other clouds, the entire cloud infrastructure of UniSuper would have been ruined.
The downtime lasted for about a week, until last Thursday, when service was finally restored to UniSuper customers. Investment account balances will be able to reflect the lost week's data, and UniSuper says it will update the period as soon as possible.
Unisuper is a superannuation fund that manages retirement savings for members of Australia's higher education and research sectors. With 620,000 members and $125 billion in assets, the fund ranks 13th in Australia by number of members and 7th by assets under management.
Since Unisuper members were unable to access their accounts, the fund had to restore data from backups from different providers, as the deletion affected two redundant instances stored on Google services.
In the aftermath of the incident, the head of the UniSuper Fund and the global CEO of Google Cloud issued a joint statement in which they apologized to members for the outage and said it was "very frustrating and disappointing."
They mentioned that the outage was due to a misconfiguration caused by the accidental deletion of a UniSuper cloud account, which had never happened before on Google Cloud.
"Google Cloud CEO Thomas Kurian has confirmed that the outage was triggered by a series of unprecedented circumstances, starting with an accidental misconfiguration during the delivery of the UniSuper private cloud service, which ultimately led to the deletion of the UniSuper private cloud subscription." ”
This is an isolated and "unprecedented incident" that has never happened to any Google Cloud customer in the world before. Of course, none of this should have happened. Google Cloud has identified the source of the outage and is taking steps to ensure that the same situation never happens again. ”
UniSuper CEO Peter Chun sent a letter to the fund's 620,000 members on Wednesday evening explaining that the outage was not caused by a cyberattack and that no personal data had been compromised as a result of the outage. Chun emphasized that Google's cloud services were the source of the problem.
While UniSuper tends to deploy duplicate services across two geographic regions, it ensures that if one service fails or data is lost, the other can be leveraged for easy recovery. However, in this case, the fund's cloud subscription was deleted, and the backups in both regions were also deleted.
UniSuper was eventually able to resume service because the fund had a backup at another cloud provider. "Google Cloud isn't the only cloud service provider used by UniSuper, and this plan ensures our ability to restore service and minimize data loss," says UniSuper. ”
"It took a lot of attention, effort, and collaboration between the teams to recover the UniSuper private cloud instance, and the results resulted in extensive recovery of all core systems."
"UniSuper and Google Cloud have worked together to restore our private cloud extensively, including hundreds of VMs, databases, and applications."
As of May 10, in an email sent to members by Danielle Mair, UniSuper's Chief Member and Advisory Officer,
UniSuper members can already log in to their accounts online, check their balances and see the latest investment performance of their accounts. However, it is important to note that the account balances shown may not reflect transactions that have not been processed due to the interruption.
Danielle Mair said their team is processing transactions as quickly as possible to minimize delays for members.
It's unclear exactly what caused the incident, but Google Cloud has repeatedly stressed that the outage was not caused by malicious behavior or cyberattacks. But it has to be noted that Google has only apologized for this outage so far, and has not revealed any other more details.
It took 6 months to migrate to Google Cloud, and all accounts were deleted in less than a year
In fact, Unisuper hasn't been migrating to Google Cloud for a long time.
In June last year, according to foreign media reports, with the help of cloud consulting company Kasna, Unisuper had transferred all non-production workloads (including about 1,900 virtual machines) from the Australian data center to Google Cloud, and completed the migration in September last year, with a total migration time of 6 months.
According to UniSuper, the company's IT setup consists of two data centres, one in the Port of Melbourne and the other in Mitchum, Victoria. Specific details about the two data centers have not yet been announced.
The company also has cloud services in Microsoft Azure. In 2017, the company experienced an outage following a fire at its Port Melbourne facility. Eventually, UniSuper's plan was to exit its enterprise data centers altogether.
Unisuper has migrated most of its business to the Google Cloud platform in 2023, having previously distributed these workloads to Azure and its own two datacenters. The migration included moving all non-production workloads, including 1,900 virtual machines, to Google Cloud.
At the time, Sam Cooper, head of architecture at UniSuper, told ITNews: "The key attraction for us to move to Google Cloud was the team's familiarity with Google's technology, and our underlying was Google Engine. ”
The company hopes this move to the cloud will enable UniSuper to scale quickly and meet potential business growth opportunities. UniSuper also drove the move last year with the acquisition of Australian Catholic Superannuation and its desire to increase organic and inorganic growth.
"We need to be able to leverage cloud providers to do this quickly, and to be able to do that in a way that gives us the right amount of risk mitigation," Cooper said.
UniSuper put out a tender with AWS, Microsoft, and Google when selecting a cloud provider, and finally chose Google as the best match. The company says it won't split and rebuild applications to be cloud-native, which is an important choice because some applications aren't ready for cloud-native, which means UniSuper can quickly and securely migrate to the cloud to gain the necessary scalability.
In other words, less than a year after Unisuper moved over, it encountered an "unprecedented" outage of Google Cloud.
No cloud vendor is immune to downtime
In the wake of the outage, on May 11, Kenneth Dredd, a Twitter executive who claimed to be Google Cloud's director of cloud services in Australia, said he had been outright fired by Google.
I was made redundant by Google. I was previously the Head of Cloud Services Australia at Google, where I was responsible for our customers' data displays.
I accidentally deleted data while uploading a report to my UniSuper account that showed that our algorithm detected overly optimistic forward-looking statements in their environmental, social, and governance (ESG) disclosures.
I have accepted Google's generous severance compensation and, with the CEO's blessing, I will continue to pursue green money laundering directly with the Australian Securities and Investments Commission (ASIC).
PS: Green money laundering, also known as green laundering, refers to illegal investment or financing activities formed by disguising actual interests and transferring interests to the private sector through partners, minority shareholders, social funds, etc. The term originated in the 1998 "Marquez Affair," in which Marquez's company executive Scottin took advantage of the company's acquisition of the Minnesota Railroad to greenwash for his own gain. The term greenwashing soon became popular in financial, political and social circles to represent the concept of illicit profiteering.
Will you continue to use Google Cloud in the future? Danielle Mair, UniSuper's Chief Member and Advisory Officer, said in an email that UniSuper takes its responsibility to provide reliable service to its members very seriously. The team's focus is on getting the system back online quickly, safely, and reliably.
At the same time, they are working closely with Google Cloud to complete a complete root cause analysis. Google Cloud has confirmed that this is an unprecedented isolated incident and has taken steps to ensure that this issue does not happen again. UniSuper will evaluate the incident and ensure that the company is in the best position to serve its members.
In response to Danielle Mair's reply, netizens ridiculed, "He seems to have said everything, and it seems like he didn't say anything."
In fact, cloud and other network outages occur from time to time, and some of the world's major cloud vendors, including AWS, Microsoft Azure, and others, have experienced these situations. For instance, in June 2023, an incident occurred on AWS that lasted more than two hours and affected several services on the East Coast of the United States. Last September, Microsoft Azure also experienced an outage in its data center in Australia, leaving users unable to access Azure, Microsoft 365 and Power Platform services for more than 24 hours.
Parekh Jain, CEO of EIRTrend and Parekh Consulting, points out that these issues are usually resolved quickly, but the exception to UniSuper's outage that lasted so long. From a reputational standpoint, this can damage Google and lead to a lack of trust in the company as a CSP. "The current UniSuper cloud outage on Google Cloud in Australia takes a very long time to resolve, which has negatively impacted Google Cloud's reputation in the region," he noted.
Jain added that such disruptions can also lead to disruption to customer business and data loss, which is why many people are leaning towards a multi-cloud strategy for risk management.
Original link: Google Cloud deleted all data and backups of 100 billion fund customers, the technical person in charge was laid off on the spot, and Google only said sorry in the end? _Google_ Li Dongmei_InfoQ featured article