Four Lightning Strikes Hit Google’s Belgian Data Center, Minimal Data Lost
Google says the incident occurred on August 13, 2015, which is when lightning exacted its wrath on the local utilities grid that powers the company's European data center. There was a brief loss of power to the data center's storage system that's used for what Google dubs the europe-west1-b zone.
To keep users' data safe, Google destroys all failed drives on site
To be fair, the incident could have been much worse. Google has numerous safeguards in place for these types of things, including automatic auxiliary systems that were able to rapidly restore power. Battery backups are also in place. Nevertheless, some recently written data located on storage systems more susceptible to power failure because of extended or repeated power drain was lost forever.
"In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state. However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk," Google said.
Water cooling tanks sit underneath cooling towers at Google's data center in St. Ghislain, Belgium
"This outage is wholly Google's responsibility," Google added. "However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data."
While this incident was extremely isolated, it underscores the importance for multiple backup schemes for mission critical or otherwise important data that you can't afford to lose.