Comcast unravels monolith of spaghetti code with virtualization

When it comes to virtualization, Comcast has been at the forefront of the cable industry by dipping its toes into distributed access architecture (DAA) about five years ago.

Those early efforts paid off last year for Comcast as upstream and downstream traffic increased significantly during the Covid-19 pandemic due to virtual learning and work-from-home scenarios. Comcast's access network covers about 59 million households throughout the U.S, and it is comprised of about 750,000 miles of fiber and coax cable, according to Elad Nafshi, senior vice president of next-generation access technologies at Comcast Cable.

With cable upstream typically running on a smaller slice of 5 MHz to 40 MHz, Comcast engineers worked to go beyond the usual node splits to provision more upstream capacity during the panedimic. With the recent releases of games such as "Call of Duty," as well as new Xbox Series X and PlayStation 5 consoles, Comcast saw an increase in upstream usage last month.

DAA basically seeks to digitize the access network and virtualize it so that Comcast could rely on web-based speed, and have Moore's Law drive its capacity capabilities, rather than rely on proprietary appliances that have longer lead times, according to Nafshi.

"So what we're able to do with our DAA is number one, we virtualized the CMTS so we could bring it up on the network connected to digital fiber connectivity, which is not only a lot more efficient, but could be carried much further so that we could centralize the vCMTs in far fewer locations," said Nafshi, who outlined Comcast's access network journey during Tuesday's FierceTelecom Blitz Week.  "Those locations would be hardened data centers. We're going to be able to increase the reliability and the scale, and scale it much, much easier, and we'll be able to follow Moore's Law where effectively every 18 months we'll be able to double the capacity without doing much of anything.

"Then we rely on a new digital generation of nodes in fiber that will provide us with real time visibility and real time telemetry into everything that goes on in a plant. And that is one of the game changers that we'll be able to introduce with this architecture."

Using photos from gear deployed in Atlanta, Nafshi showed what was needed to aggregate 192 service groups in one location. The virtualized equivalent of those 192 service groups brought in a 22-to-1 rack savings. In addition to the cost savings, Comcast also gained real time detection capabilities down to the second level of its access network by operating at web speed and web scale.

"When you look at the operations of a virtualized web based architecture, you could continue to rely on CLI and SNMP for running the platforms," Nafshi said. "Or you can run on modern-based KAFKA, AWS-driven platforms that are able to provide you with down to a 5 second view of what goes on the CMTS. What goes on the switching layer. What goes on the nodes, and what goes on the cable modems, and tie it all together into a complete end-to-end real time view of what the customer experience looks like.

"When you look at the current appliances and the legacy CCAP gear, typically with those we would deploy our code once a year because 'Oh my god, what happens if something breaks?' And it's such a monolith of spaghetti code, that it's very difficult to add additional features. Or you could rely on a fully automated CI/CD deployment pipe that deploys the virtual platform out of a common Git Repository with a push of a button where operations go through with real time validations."

"We deploy literally 10s of 1,000s of changes on an annualized basis relying on this platform with being able to do all of that hitlessly and seamlessly without any impact to the customers as we're going through it."

Nafshi said the next level for Comcast would be to fully automate the decision-making of the deployment code.

"Even that it will be machine and AI driven," he said. "If the machine detects any type of failures, the machine will be able to automatically rollback the code, analyze where the failure is, and then present the options to the operator going forward. So this next generation operational model is really driving the ability to truly drive visibility into the access network like never before."

RELATED: During COVID-19, Comcast ups its virtualization game from core to cable modems with AI and ML

During the coronavirus pandemic, Comcast was able to increase its upload capacity by 36% by using its internally developed Octave platform, which it started to develop two years ago. Octave can detect network anomalies, such as external LTE noise, across the cable modems Comcast has deployed.

Modulation varies across networks and cable modems based on plant conditions and factors such as outside noise. Octave was developed to detect when the DOCSIS 3.1 cable modems weren't using all of the bandwidth available to them.

"We were able to add additional channels on to that 5 to 40 capacity in order to drive even more upstream out of the capacity that we have," Nafshi said. "With all of these tactics in place, we're able to drive up upstream capacity by 42% just by relying on virtualization, real time telemetry, and machine learning and AI onto the access network.

Nafshi said the next step for Comcast's DAA platform was tying in the fiber visibility with power grid and power network visibility. Comcast is working on tracking its fiber distribution with a new network monitoring technology called XMFR, which it also developed internally.

RELATED: Comcast notches 10G milestone with a trial of 1.25-Gig symmetrical speeds in Florida

"We're not just the largest broadband provider, we also operate the largest power grid powering this immense network," he said. "So how do we tie in the power grid visibility into that fiber visibility and be able to deliver more upstream capacity with this technology, implementing what we call mid split and high split, which will enable us to drive upstream speeds of up to one gig in a symmetrical way/

"And then ultimately delivering on our 10G vision, which will enable us, riding on top of this virtualized DAA architecture, to deliver speeds of multiple symmetrical gigs across our access network. That's our access network and access network story."