UniSuper’s production Google Cloud VMware Engine (GCVE) private cloud was automatically deleted one year after it’s creation due to a misconfiguration in how it was created. When it was created, there was a bug in the creation script which passed a null value. This caused the private cloud to be created with a one year subscription, rather than a perpetual one. After one year passed, Google Cloud dutifully deleted the private cloud.

The rest of this post is still relevant for more context, but it is now clear that this was indeed a Google Cloud bug. I remain puzzled why the communication over this incident was so bad. By their silence, Google Cloud let millions of people think that they had deleted a company’s entire Google Cloud environment. I’m also confused why this information came out via a non-employee on Twitter rather than through official channels.

Update 2: Google Cloud has published a blog post explaining exactly what happened. The details do bear out that this was entirely a Google Cloud issue from start to finish, with no fault on the part of UniSuper.

Over the past two weeks, UniSuper, an Australian superannuation fund, faced every cloud user’s nightmare: their cloud provider deleted their data. From May 2nd to 13th, UniSuper experienced a major outage and in several updates blamed Google Cloud for the incident.

The disruption of UniSuper services was caused by a combination of rare issues at Google Cloud that resulted in an inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud, which triggered a previously unknown software bug that impacted UniSuper’s systems. This was an unprecedented occurrence, and measures have been taken to ensure this issue does not happen again.

This culminated in a press release, jointly signed by Google Cloud CEO Thomas Kurian.

Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.

This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.

UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.

Restoring UniSuper’s Private Cloud instance has called for an incredible amount of focus, effort, and partnership between our teams to enable an extensive recovery of all the core systems. The dedication and collaboration between UniSuper and Google Cloud has led to an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.

The press release uses vague language, obscuring the technical details of what happened. Given the lack of precision, I assumed this was written solely by UniSuper. Gergely Orosz checked with Google Cloud, and they confirmed that this was an official joint statement.

Given the few details provided, I did more research to try to understand what exactly happened.

What was deleted?

Google Cloud doesn’t have a resource called a “Subscription”; the closest concept would probably be a “Billing Account.” While Google has a Virtual Private Cloud (VPC), you wouldn’t describe it as an instance, and deleting a VPC doesn’t seem like it would have the drastic effects described.

However, if you research UniSuper’s Google Cloud migration, you will see that they use Google Cloud VMWare Engine (GCVE). From June 2023:

The superannuation fund has shifted almost all non-production workloads out of its data centres and into Google Cloud so far.

UniSuper used the Google VMware Engine (GCVE) managed service and engaged partner Kasna to assist with the migration.

GCVE has a resource called a private cloud. A private cloud contains the hosts, management server, storage, and networking for a VMware stack.

Let’s look at the API method for deleting a private cloud:

A PrivateCloud resource scheduled for deletion has PrivateCloud.state set to DELETED and expireTime set to the time when deletion is final and can no longer be reversed. The delete operation is marked as done as soon as the PrivateCloud is successfully scheduled for deletion (this also applies when delayHours is set to zero), and the operation is not kept in pending state until PrivateCloud is purged. PrivateCloud can be restored using privateClouds.undelete method before the expireTime elapses. When expireTime is reached, deletion is final and all private cloud resources are irreversibly removed and billing stops.

All private cloud resources being irreversibly removed sounds a lot like the outage that UniSuper had. Some people have said that Google deleted their entire Google Cloud account. This seems less likely to me because Google has a number of safeguards and delays when deleting projects:

Warning: You can recover most resources if you restore a project within the 30-day period. Some services have delays in restoring and you might need to wait some time for services to be restored. Some resources, such as Cloud Storage or Pub/Sub resources, are deleted much sooner. These resources might not be fully recoverable even if you restore the project within the 30-day period.

Update: User smcwhtdtmc on HN pointed out the Terraform resource for vmwareengine_private_cloud hard-codes the delayHours parameter to 0, i.e. immediate deletion. All private cloud resources being immediately irreversibly deleted sounds a lot like the outage that UniSuper had.

How did everything get deleted?

UniSuper’s press release says:

UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.

Again, the language here is frustratingly vague and passive but implies that Google Cloud deleted multiple independent private clouds in separate regions. Google Cloud doesn’t have a “geography”; it has zones and regions. At first read, it sounds like they are describing a multi-region setup. Google Cloud has two Australian regions, Sydney and Melbourne, which would make sense.

Looking closer at the docs, though, GCVE offers two kinds of private clouds: a standard private cloud hosted in a single zone or a “stretched private cloud”. A stretched private cloud runs in a single region across two zones, with a third zone as a witness zone for failover. A close reading of the press release doesn’t rule out UniSuper having a single stretched private cloud running in a single region.

So we either have a single stretched private cloud being deleted or two separate private clouds being deleted. A single stretched private cloud seems more likely to me, but I can’t be definitive.

How was it deleted?

So we come to the big question: How were the private cloud(s) deleted? The press release makes heroic use of the passive voice to obscure the actors: “an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.”

Update: 2024-05-21 Miles Ward has the details on what actually happened in this X thread. My guesses were close, but not quite right.

Based on my experiences with Google Cloud’s professional services team, they, and presumably their partners, recommend Terraform for defining infrastructure as code. This leads to several possible interpretations of this sentence:

1. UniSuper ran a terraform apply with Terraform code that was “misconfigured”. This triggered a bug in Google Cloud, and Google Cloud accidentally deleted the private cloud.

This is what UniSuper has implied or stated throughout the outage.

2. UniSuper ran a terraform apply with a bad configuration or perhaps a terraform destroy with the prod tfvar file. The Terraform plan showed “delete private cloud,” and the operator approved it.

Automation errors like this happen every day, although they aren’t usually this catastrophic. This seems more plausible to me than a rare one-in-a-million bug that only affected UniSuper.

3. UniSuper ran an automation script provided by Google Cloud’s professional services team with a bug. A misconfiguration caused the script to go off the rails. The operator was asked whether to delete the production private cloud, and they said yes.

I find this less plausible, but it is one way to interpret Google Cloud as being at fault for what sounds like a customer error in automation.

Maybe it was a Google Cloud bug?

There are a few holes in my theory that this was primarily UniSuper’s fault:

1. On Tuesday, 7 May, UniSuper attributed a statement to Google Cloud in which Google Cloud admitted fault for the outage:

The disruption of UniSuper services was caused by a combination of rare issues at Google Cloud that resulted in an inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud, which triggered a previously unknown software bug that impacted UniSuper’s systems. This was an unprecedented occurrence, and measures have been taken to ensure this issue does not happen again.

2. Thomas Kurian (apparently) signed off on a joint statement with UniSuper. This statement is less accusatory than UniSuper’s other statements but doesn’t clarify which party was at fault.

3. Google Cloud hasn’t released any pushback to the story, either directly or through proxies in the media.

Why would Google Cloud not respond to this story if it was UniSuper’s fault? It’s hard to say, but putting out a competing statement blaming or contradicting your customer is a bad look with that customer and with all future customers.

Conclusion

Given how little detail was communicated, it is difficult to make a conclusive statement about what happened, though I personally suspect UniSuper operator error was a large factor. Hopefully, APRA, Australia’s superannuation regulator, will investigate further and release a public report with more details.

1. Prices never go up^{1 2}.
2. We will absolutely soak you on data transfer charges.

Last week Google Cloud published a post explaining how they were ignoring the first rule in favour of adopting the second.

The post was called “Unlock more choice with updates to Google Cloud’s infrastructure capabilities and pricing”, though the <title> was more straightforward: “Updates to Google Cloud’s Infrastructure pricing”. The main announcements in the post were:

• Prices were being raised on Cloud Storage (some decreases on Storage too), Persistent Disk Snapshots, Cloud Load Balancing, Network Intelligence Center, and Cloud Ops Monitoring (Weirdly, the Monitoring price increase was only mentioned in customer emails, not in the blog post).
• A new, cheaper archive snapshot option for persistent disks is coming later this year.
• Storage Transfer Service will be free for the rest of the year to let customers move files into different buckets.

The common theme with the price changes is the introduction of data transfer fees, an increase in fees for active storage tiers, and a decrease in archive storage cost.

So, today, we are announcing we will adjust our infrastructure product and pricing structure … [These changes] are also designed to better align with how other leading cloud providers charge for similar products, so customers can more easily compare services between leading cloud providers.

The main alignment here is Google adding data transfer charges to match AWS and breaking architectural promises they’ve made to their customers in the process. This is an incredibly short-sighted move and will damage customer trust in Google Cloud for many years.

Cloud Cost and Cloud Architecture

As Corey Quinn so eloquently put it, “all cloud cost is fundamentally about architecture”. When designing for the cloud, pricing is one of the most important signals to take into account. Pricing and quotas indicate how the Cloud provider has designed the product, how they want you to think about the product, and how you should use it.

The pricing changes Google is making strike at the heart of their customer’s applications and will force many customers to rearchitect their applications, or pay a much larger amount to keep their existing architecture.

One of GCP’s most differentiated features has been their multi-region and global services like Cloud Load Balancing, Cloud Storage, BigQuery, Cloud Spanner, and Cloud KMS. These let you operate on the same resources in multiple regions, giving you more resilience to an outage in a single region. As a bonus, Google’s multi-region services often come with strong consistency across regions so customers don’t have to deal with consistency at the application layer.

In contrast, most AWS services are region oriented, and when AWS provides multi-region services like DynamoDB Global Tables and S3 Cross-Region Replication, you need to handle eventual consistency yourself.

Google’s message was that multi-region services would give you higher availability. You too can run like Google. From the announcement of dual-region buckets:

With this new option, you write to a single dual-regional bucket without having to manually copy data between primary and secondary locations. No replication tool is needed to do this and there are no network charges associated with replicating the data, which means less overhead for you storage administrators out there. In the event of a region failure, we transparently handle the failover and ensure continuity for your users and applications accessing data in Cloud Storage.

People can argue about whether Google’s global/multi-regional control-plane provides more availability than AWS’s region-first approach, but it was a distinctive set of features that only Google had.

Storage changes

The deepest architectural changes Google is making are to storage:

• Increasing free internet egress from 1GB to 100GB per month. This matches AWS’ recent increase, but is still a welcome change.
• A mix of increases and decreases in multi-region pricing depending on the storage class and region (+50%, +25%, -40%, -25%). Standard storage pricing is unchanged.
• A mix of increases and decreases in dual-region pricing, depending on the storage class and region. nam4 (US) and eur4 (Europe) are +22% on standard, +10% on nearline, -2% on coldline, and -44% on archive. asia1 (Japan) is +10% for all storage classes except for -13% on archive storage.
• Storage operation pricing changes have a lot of fine details, but at a high level the cost for Insert/Update/Delete operations in multi-region and dual-region buckets is doubling (100%).
• Change in egress pricing. If I’m reading it correctly and the existing exclusions stay in place, it looks like a pretty large decrease in bucket egress for most uses. Previously 10TB of bucket egress (direct to the internet) would be charged at $0.11/GB to most of the world. Now that will be $0.02/GB from US buckets to the US, $0.05/GB to Europe, and $0.08/GB to Asia. This is a pretty large decrease, and I’m surprised Google didn’t make more of it in the announcement.
• Changes to data replication pricing from $0.00 GB to $0.02/GB in us, nam4, eu, and eur4, and to $0.08/GB in asia and asia1. (These are the dual-region and multi-region bucket locations).
• Reading data from a multi-region bucket in the same continent will no longer be free, it will now be charged at the same rate as data replication. This will make many data-heavy workloads unviable with multi-region buckets.

The last two changes in particular make dual-region and multi-region buckets a much less attractive offer and invalidate a lot of architectural assumptions developers will have made. Businesses that have built a high-availability architecture around multi-region buckets are now faced with the unappealing options of:

1. Pay a lot more for replication + reading from their buckets.
2. Rearchitect their application and migrate to a single region, or perhaps a dual-region if they can make the data-replication pricing work.

Load balancing

Google is also increasing prices on Cloud Load Balancing, adding a $0.008-$0.012/GB charge for outbound data processing.

Starting October 1, 2022, we’ll apply an outbound data processing charge of $0.008 - $0.012 per GB (based on region) to all Cloud Load Balancing products in order to maintain consistency and alignment with the variable costs of the services across our Cloud Load Balancing portfolio.

Depending on your egress rates and network tier, this will amount to a 5-10% increase on internet egress, and an added cost for most kinds of internal load balancing.

Persistent disks

Persistent disk snapshot pricing is also changing. I suspect this won’t be as big a deal for most customers as the previous changes.

• Regional snapshots are increasing from $0.026 to $0.050 (92% increase).
• Multi-region snapshots are increasing from $0.026 to $0.065 (150% increase).

The price changes mention a “us-central1 baseline” and “United States multi-region baseline”. This makes me think that prices in all other regions will be going up by an equivalent percentage.

Google also plans to introduce an archive snapshot tier later this year which will be priced at $0.019/GB/month, but with a minimum billing period of 90 days.

As with Cloud Storage, previously free data transfer for creation and restoration of multi-region snapshots will now be charged at inter-region rates. Restoring a 100GB multi-region snapshot will now cost $1 in the US and Europe, and more in other regions. Depending on your architecture, these charges could add up!

Why now?

On March 7th, 2022 The Information published Why AWS Makes Money and Google Cloud Doesn’t

[..] In a positive sign, Google Cloud CEO Thomas Kurian last month told colleagues during an internal all-hands meeting that he expects the cloud unit to be profitable later this year, according to a person who viewed the event.

Another issue is that Google Cloud has fewer high-margin services—such as cloud database and analytics software—to sell to customers than AWS does. […] Former employees say AWS relies on these offerings for a large portion of its operating profit.

The Information doesn’t mention it here, but the highest margin product AWS sells is bandwidth. Charging for data transfer is as close as you can get to pure profit in the cloud. As Cloudflare has previously demonstrated, AWS likely marks up egress bandwidth by up to 8,000%, and it wouldn’t surprise me if inter-AZ and inter-region charges were also in that vicinity.

If Google Cloud wants to get to profitability by the end of the year, then introducing data transfer pricing makes a lot of sense. The extra revenue from data transfer will be nearly 100% profit and go straight to Google Cloud’s bottom line. It will be difficult for existing customers to rearchitect their applications, and most customers will probably just pay the extra charges.

However, if Google Cloud is trying to build the number two cloud business behind AWS, this move is an unforced error that will damage their credibility for years.

Killed By Google

Google has developed a reputation for killing consumer products, and even Google Cloud has made several price increases which have changed architecture invariants, along with numerous deprecations:

• 2011 - Increasing App Engine pricing by 2x-10x
• 2018 - Increasing Google Maps API pricing by up to 1400%
• 2020 - Reintroducing management fees for Kubernetes clusters
• 2020 - Google Hire shut down

These changes have damaged Google’s credibility for many customers and undermined their commitment to building GCP to be an equal to AWS and Azure. Google has been trying to build customer trust with their Enterprise API stability promise, and signing many 10-year deals with large enterprises like Deutsche Bank, Mayo Clinic, and Sabre.

However the last few months have seen several shortsighted decisions:

1. Charging for personal Google Workspace accounts (Google Workspace comes under Google Cloud). Many of the people affected by this will be decision makers for Google Cloud purchasing decisions, and not people you want to remind about Google’s reputation for shutting down products.
2. Laying off some Cloud technical support staff and outsourcing support to third-party vendors.
3. Last week’s Cloud price increases.

Put together, it paints a picture of an organisation myopically focused on short-term profitability over long-term strategy. These changes show they don’t understand customer perceptions of Google Cloud, and the “rules” of cloud pricing.

These changes are hard to even understand as a business strategy. Raising prices on locked-in customers feels like a move you’d see from Oracle, not from the third-ranked competitor in a rapidly expanding market.

My best guess as to why Google is doing this now is that Google Cloud set (or had set for them) an OKR to reach profitability. All of the teams are trying to increase profits, regardless of the long-term costs.

A different future

Google Cloud’s strength has always been its technology. It has fewer products than AWS, but those products are more flexible and cohesive. Their IAM model is far simpler than AWS’, and as previously outlined, Google’s global infrastructure has offered many unique, differentiated products.

At a time when it’s hard to even keep track of Azure’s many multi-tenant security vulnerabilities, and multiple AWS outages still in recent memory, Google Cloud should have been pressing their advantages.

Instead of raising prices, I think a better strategy would have been to cut prices on data transfer to nearby regions and lean into multi-region services. Google owns massive amounts of dark-fibre and under-sea cables connecting their data centres and could afford to lower their prices. Once inter-region transfer is cheaper, many more applications become feasible to run in high-availability, multi-region configurations. This would be a strong offering to sell, and one that is hard for Amazon and Microsoft to replicate. Customers running compute in multiple regions and using higher-margin services like Spanner would help make back some of the revenue lost from lowering inter-region data transfer.

This would have been a compelling, differentiated offering, playing to Google’s strengths. Instead, Google has altered the deal. Their customers will now be praying that they don’t alter it further.