We’ve received the Reason for Outage (RFO) from our eastern data centre. The outage at the data centre caused the outage on May 19, 2022.

The following is the content from the RFO:

‌

Root Cause Analysis

The Master Electrician and Senior Facility Technicians determined that due to a physical misconfiguration, a failing power supply had caused a cascading power failure across multiple electrical circuits. An automated transfer switch (ATS) intended to mitigate issues of power failures across multiple circuits had been incorrectly connected to an electrical circuit for a cabinet. That contained the secondary network devices providing redundancy. When the power supply failed in the core network switch in the first cabinet, it caused the breaker on the “A” circuit to trip. The ATS then switched this load to the alternate “B” circuit, that too caused the breaker in the “B” circuit to trip.

‌

Corrective Measures and Mitigation

The Master Electrician and Senior Facility resources immediately identified the incorrect electrical circuit management during the troubleshooting process and rerouted the equipment connections to individual dedicated circuits. Subsequently, it was determined that the ATS devices are not a necessary component of this design as the switches already contain redundant power suppliers, and the introduction of the ATS simply added complexity and resulted in the condition which led to the tripped breaker. As a result, they will be removed in an upcoming maintenance window. The impact of this change is expected to be minimal as all connected devices have redundant power supplies. Further details will be available in forthcoming change communications.

To further mitigate such scenarios, we will be assessing the relocation of the core network equipment to separate MMRs. Details will be communicated once available.

Although we believe this incident to be an edge case, we have initiated an audit of our critical infrastructure to confirm proper configuration across multiple facilities. We expect to complete the audit over the next three months.