Google blames London outage on PLC configuration error

Google Cloud traced the 20 October London outage to a PLC configuration error that read voltage beyond an isolated breaker, preventing generators from starting and draining UPS batteries.

Google Cloud has attributed the 20 October outage in London to a programmable logic controller (PLC) configuration error. The controller measured voltage on the utility side of an isolated circuit breaker, so backup generators did not start after the remaining power feed failed. The site’s UPS batteries then discharged.

The outage began at about 06:53 BST at the datacentre hosting europe-west2-b, one of three availability zones in Google’s London region. An underground cable fault cut the facility’s primary utility feed. The site was already operating with one of its two mains feeds isolated for repairs.

After a quarterly generator load test on 14 October, Google isolated feed B for corrective work. The facility was therefore in a temporary single-feed configuration instead of its usual dual-feed design.

The PLC was designed to detect a drop in incoming voltage and start diesel generators. Its sensors were connected upstream of the feed B breaker. Although the breaker was open, the cable on the utility side remained energised. When feed A failed, the PLC continued to detect voltage on the unused cable and did not issue a generator start command.

Google’s incident report described the problem as “a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position.” The static UPS batteries were designed to provide about five minutes of backup power. After they discharged, the data halls lost power.

More than 35 Google Cloud services were affected in europe-west2, including Compute Engine, Kubernetes Engine, Cloud Run, App Engine, BigQuery, Cloud SQL, Spanner, AlloyDB, Cloud Storage, Persistent Disk, Virtual Private Cloud, Cloud Load Balancing and Cloud NAT.

Customers reported disruption affecting banks, HM Courts & Tribunals Service video hearings, Royal Mail, healthcare operations, retailers and delivery companies. Technology providers including Stripe, Snowflake, MongoDB and JFrog also reported related problems.

Google redirected workloads to the two surviving availability zones. The additional compute demand, database failovers and cross-zone network traffic saturated available capacity. This caused queue backlogs, delayed failovers, higher latency and intermittent HTTP 500 and 503 errors. Some customers using the unaffected zones also experienced disruption.

Google reconfigured and tested the PLC code and undervoltage controls on 20 and 26 October. The revised system uses busbar power status and breaker position, rather than upstream voltage readings, to determine when generators should start.

The company has begun reviewing PLC configurations at other facilities with similar electrical designs. It has scheduled coordinated cable repairs, generator load-transfer tests and the restoration of the site’s dual-feed configuration through November.

London is Google Cloud’s only UK region, so customers have no separate UK region for automatic recovery. The incident also exposed capacity constraints when workloads from a failed availability zone shifted to the remaining zones.

Articles by this author