Canadian operator Rogers Communications over the weekend shed some light on the suspected cause of a prolonged network outage that took down fixed and mobile communications across the country on Friday.
In a notice posted Saturday, CEO Tony Staffieri said service had been restored and the operator had “narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction.” He added it was working to further home in on the root cause and identify steps to increase redundancy in its network to avoid a repeat of the blackout.
“We will take every step necessary, and continue to make significant investments in our networks to strengthen our technology systems, increase network stability for our customers, and enhance our testing,” Staffieri wrote.
Rogers also promised to proactively credit the accounts of all customers who were impacted.
The outage began early Friday morning, impacting both wireline and wireless services. It lasted well into the evening, with Rogers posting on Twitter at 9:52 p.m. on Friday evening that its wireless services were only just starting to recover. By 7 a.m. Saturday, Rogers said services had been restored for the vast majority of customers.
Internet monitoring company NetBlocks noted in a tweet that the incident knocked out a quarter of all observable connectivity in Canada, impacting banking, government services and emergency response capabilities.
Cloudflare, a content delivery network and DDoS mitigation company, said in its analysis of the event that the outage lasted a total of around 17 hours. It noted the incident began following a “clear spike” in Border Gateway Protocol (BGP) updates which began around 8:15 a.m. on Friday, lending credence to Staffieri’s statement about the outage’s cause.
The timing of the event is poor for Rogers, which is currently struggling to alleviate concerns about its proposed merger with Shaw Communications.
The outage prompted Canada’s Minister of Innovation, Science and Industry François-Philippe Champagne to speak with Staffieri directly. On Twitter, Champagne said he “shared with him the frustration of millions of Canadians.”
He concluded “This unacceptable situation is why quality, diversity & reliability are key to our telecom network.”