Comcast is farther along the virtualization path than you might think, largely due its X1 video platform that turned 10 this year.
While AT&T traces its virtualization roots to its Domain 2.0 whitepaper in 2013, along with the announcement the following year that it would have 75% of its network functions virtualized by next year, Comcast's virtualization effort started in 2009.
In 2009, rumors started to percolate at trade shows about Comcast's next-gen video platform, which at the time was called Excalibur before it was later renamed X1. In 2012, Comcast's cloud-based X1 video service came out of cloaking mode at the NCTA's The Cable Show during an onstage presentation by Chairman and CEO Brian Roberts.
The XI platform, which initially supported QAM and IP video services, has been a game-changer for Comcast in its battle to reduce churn by its video subscribers. Other cable operators, such as Shaw Communications, Cox Communications, Rogers Communications and Videotron, have since licensed the X1 platform.
On the sidelines of last week's SCTE-ISBE Cable-Tec Expo in New Orleans, Comcast's Tony Werner and Matt Zelesko spoke to FierceTelecom about the company's virtualization efforts, Comcast's cultural transformation and the edge.
"Our path on virtualization started quite awhile back, and it started really when we did the X1 platform." said Werner, president, technology, product, Xperience organization at Comcast Cable. "All of that is virtualized. The reason we did that is because we wanted a video platform where we could move incredibly fast, and that's what we do. We can make a feature change literally in hours if we want to.
"Then as you go into the rest of it, everything goes down its (X1's) path. Certainly Remote PHY in the nodes is part of virtualizing the network. We're well underway on that and consider it the right architectural approach for the future."
Werner said Comcast recently crossed its 100th major release for X1 as the platform continues to evolve with new features such as a cloud DVR and a voice-controlled remote control.
"It was transformational for us, both from a product perspective, but I think also from a culture perspective," said Zelesko who was promoted to his chief technology officer role in May.
Also on the virtualization front, Comcast is rolling out virtual cable modem termination systems in its network, but doesn’t say how widely they are deployed. Comcast launched its SD-WAN service two years ago on its software-defined networking (SDN) ActiveCore platform, and has since added other virtual network functions (VNFs), including security firewalls and a managed router, to ActiveCore for its business customers.
RELATED: Comcast Business beefs up cybersecurity with Fortinet and Akamai
On the open source front, Comcast is a member of both the Open Source Foundation (ONF) and the Open Compute Project (OCP). Comcast has deployed ONF's Trellis software for white boxes in multiple markets and has rolled out some open source software as part of its Distributed Access Architecture (DAA) buildout.
RELATED: Comcast puts open source networking software into production
"If we hadn't had the 10 years of experience with XI, all these other virtualization things that we're doing would be a lot harder," Werner said. "I think a lot of things that people don't realize is virtualizing is easy to say, but it really does take a lot of years of culture building, of practice building, of discipline to be able to understand how you do releases, how you do triaging when there's issues, how you do roll back, how you maintain security, how you do all of those pieces.
"And unless you learn them with experience over a period of time, it's not something that you just all of a sudden read a book on and do."
Werner said that Zelesko has been a huge champion for changing and building the culture at Comcast. For the most part, Comcast is building its internal teams organically as the company further defines its culture.
"One of the ways we've been really successful is by operating, essentially, as a set of federated startups," Zelesko said. "So a lot of our teams have their own identities and some of those micro cultures."
Some of the teams develop out of acquisitions that Comcast has made. Zelesko said Comcast pulls the teams together every Friday for a few hours so that all of the team members know what the others are doing.
"So we've tried to create an environment where even though you've got this federated startups kind of operating independently, we have a framework to learn, and a framework to understand what those (other teams) are doing," Zelesko said. "When you talk about moving fast, when you talk about taking risks, so much of it comes down to creating a blameless culture. One where we understand that there are failures, that there are things that'll happen, and it's how we respond to those things, how we learn from them, as opposed to punishing the team that had a mishap.
"So I think that has been the biggest cultural shift. In order to move fast we had to take risks. We take that risk not at the expense of the customer, but we do it internally in a way where we're constantly learning and increasing that cycle for learning."
Zelesko and Werner both said broadband -- both residential and business broadband -- was fundamental to Comcast's success. Comcast is actively virtualizing its core network functions and its access network to build the best possible broadband network for its customers.
"When you look at the virtualization of the access network, we are replacing rows and rows of data center racks with single racks," Zelesko said. "It is more economical for us. It's way more flexible for us. At the end of the day, as we need to reconfigure the network, as we have to augment capacity, we can do it at a much better cost and in a much better way."
As Comcast becomes more of a national company, virtualization enables the company to be more agile while also reducing its carbon footprint.
Comcast is also virtualizing its network edge. Comcast has seven national data centers and uses a combination of private and public cloud for the majority of its services. Comcast also has 1,500 edge locations, according to Werner.
"A lot of things we like doing in these seven national data centers because it's just so efficient," Werner said. "Because they're just rows and rows of common compute architecture that Matt and his guys can turn up VMs (virtual machines), write software in, spin it out and go. And for 99% of the services, we can do it out of those data centers and they're all redundant, they all back each other up in the rest of it.
"Then going out to the other 1,500 sites, that's where, as edge compute requirements come and super low latency. We've got the architecture to do it, but we've not announced any plans or anything along those lines."
When cable's 10G technology is ready, Comcast will be well positioned to offer lower latency services at the edge.
Comcast is starting to use Kubernetes, microservices and containers in parts of its network. Werner said the click of a button by a consumer using X1 triggers anywhere from 20 to 30 services.
"There's a quote I like, which is, 'Now that we've shifted to microservices, every outage is a murder mystery,'" Zelesko said. "And it does feel like that sometimes. We've created a team of detectives. When we have an outage, we swarm on it and have a culture that can go and solve that very quickly, which is exciting."