Sandy's new challenges reflect resiliency of traditional and next-generation services

Samantha Bookman, FierceTelecomAs the Northeast, and particularly the New York metro area, continues to recover from the effects of Hurricane Sandy, critics have been quick to point out how vulnerable the communications infrastructure (and the power infrastructure) is to giant, unprecedented superstorms. Getting lost in the storm of "what-ifs," however, are the examples and data that showed how resilient and adaptive that infrastructure, and the people responsible for it, could be.

A number of things went wrong, as was widely publicized. As water surged into lower Manhattan, power substations were damaged and the electrical grid had to be shut down. Generators at major carriers and at the city's numerous data center "hotels" kicked in, only to face the same flooding compounded by refueling issues. Wireline voice service went down in parts of Manhattan when a Verizon CO went under water and lost power. And cell towers across the region were damaged or gradually went offline as their backup power supplies drained away. But here are a few things that went right.

The Internet goes beyond the red—and survives

Internet monitoring service renesys rates the seriousness of an Internet outage on a green, yellow, red scale. Red status is reached when 5 percent of Internet networks in a measured area are down. New York went way past that.

renesys NYC network outages

Detail of network outages. (Screencap source: renesys)

"In fact, Manhattan's outage rates were much higher--on the order of 10%, which is impressively low given the fact that ConEd cut power to much of the island," renesys' James Cowie wrote in a blog post. "Silencing ten percent of the networks in the New York area is like taking out an entire country the size of Austria, in terms of impact on the global routing table."

Internet users on Tuesday morning, Oct. 30, were unable to access several websites, including visitor-heavy sites like Huffington Post. While it and other sites experienced several hours of downtime, HuffPo, whose servers were housed in a data center that lost power due to generator fueling problems, was eventually able to fail over to a backup site in New Jersey. The site utilized parent AOL's private cloud to get back up and running.

Other websites operated on a partial basis; for example, IEEE's home page was accessible, but search functions were limited or unavailable for a few days following the storm. As Data Center Dynamics pointed out in a post-storm summary, the distributed nature of the Internet meant most websites were able to get back up or keep operating despite the high outage rate.

Data center providers go above and beyond

One of the most compelling storm stories from a wireline perspective was the loss of power in major data center "hotels" in the New York metro area. A number of data center providers have space in lower Manhattan's tall buildings, most notably 60 Hudson St., 32 Avenue of the Americas, 75 Broad St., 111 8thAve. and 65 Halsey in Newark, N.J., among others.

Zayo data centers NYC

Map of Zayo's lower Manhattan network and data center facilities. (Image source: Zayo)

When ConEd shut down power, data centers' backup generators automatically kicked on. But the scale of the damage to the city and the surrounding region took its toll, and providers had to go into overdrive to stay operational.

For example, zColo, Zayo's colocation provider subsidiary, met with a range of unique situations during and after the storm, with generators being a top issue. "The generator infrastructure at those locations are not set up to run for extended times," said Chris Morley, president of zColo, in an interview with FierceTelecom. "So a weeklong runtime stresses what they're capable of doing. It's not a great situation."

Limited power caused a temporary problem with cooling in zColo's 111 8th Ave. suite on Nov. 1, with temperatures in some spaces climbing to over 100 degrees (Fahrenheit). zColo took measures to preserve the hardware in those areas, including shutting down backup servers, until the situation stabilized.

Fuel became another priority for the provider. "Most companies have committed refueling engagements. When you encounter something like last week those arrangements fall apart," Morley said. "We were able to refuel throughout the phase, but it was an interesting environment. Fueling trucks would pull up and everybody would flow out of the buildings to get fuel. All the preplanning--if you rested on that, you weren't going to be in very good shape."

Other data center providers that stayed up or were undamaged by the storm offered space and equipment on a temporary basis to companies trying to keep their websites or networks up. Telehouse America's sites at 25 Broadway and Chelsea Center, as well as its Staten Island site, stayed up, and by early Wednesday the carrier-neutral colo provider was offering space to those who needed it.

Throughout the crisis, Morley emphasized, planning prior to the storm combined with zColo's data center and network experience made a difference. "We've dealt with disasters before and have learned over time," he said.

The provider also made customer contact a priority. "I think this was the most proactive communication we've ever had, and it was critical given the time frame of this event. We got lots of positive feedback from customers on our transparency," said Morley. In addition to updates via email and on Zayo's website, a conference bridge was available every 30 minutes to 1 hour for customers to talk to the company's CTO and receive more detailed updates.

Wireless, wireline are equally critical

Most industry watchers already feel cellular technology has come into its own thanks to its reliability and affordability of wireless calling plans. Hurricane Sandy's impact on wireline voice networks as well as wireless cell towers, and the alternatives people resorted to in order to have service, reinforced the fact that both traditional and wireless services are critical parts of the U.S. communications infrastructure.

The experience of Zayo, which offers Fiber to the Tower (FTTT) services in the Northeast, was typical of the challenges faced by wireless service providers following the storm. Approximately 300 of its towers, from Philadelphia to New York City, were affected to some extent, as 65 were "hard down," according to Morley.

"We prepared ahead of time to make sure we had additional battery backups. Those run about 48 hours. We had to get out and change them, but some sites were inaccessible from a safety perspective," Morley said. Downed wires and impassable roads made it difficult to reach all of Zayo's towers; however, "by Friday [Nov. 2] we had 95 percent of all towers returned to commercial utility [power] or solved."

Because Zayo deals with a number of dark fiber customers, keeping the company's fiber network up and secure was extremely important. "As a standard, we reroute traffic on our network in every instance where we would be able to. One specific regional long haul route had a fiber cut [as a result of the storm] and we were able to move that traffic to another protected route," Morley said.

While it's impossible to predict when a storm like Sandy will hit again, it is possible for communications networks to absorb major impacts thanks to preplanning, network and data center redundancy, and the efforts of providers to keep the lines of communication open.--Sam