Comcast - The challenge of personalized, mobile, on demand TV
- Summary:
- CTO of Comcast, Sree Kotay, was speaking at MongoDB’s user event in New York this week, where he spoke about the importance of high performance.
Comcast, one of the US’s leading network providers, has inevitably been disrupted by this trend. No longer is linear television viewing enough to satisfy the demands of the ‘always connected’ American citizen.
In countries like the US, consumers increasingly want to be able to watch what they want, when they want and to be served more choice than ever before (preferably personalised). However, the demands this places on a company like Comcast’s technology should not be underestimated.
Speaking at MongoDB’s annual user conference in New York this week, Comcast CTO Sree Kotay explained how Comcast has been forced to reposition itself as a technology company that supports content, rather than a content company that uses technology. Kotay said:
I thought I’d talk to you a little bit about how the whole NoSQL mindset that the MongoDB team so excels at, has really informed our thinking about how we approach giant ideas in our infrastructure.
If you look at our engagement and if you look at the kind of technology that we have to deal with, we have 27 million customer relationships. So about 20-25% of US households. If you look at the network we have built, it’s about 630,000 miles of network that we have laid. Traffic on the Comcast network peaks at more than 30 terabits of data per second (Tbps). We are only 20 percent of the US.
People just don’t think of us as a technology company and I can kind of understand why. Over the last four or five years we have been going through this profound transformation inside Comcast, because we feel like the experience, building a better product and better technology is going to become fundamental to media consumption.
Brazil
To give an idea of the challenge at hand, Kotay described Comcast’s broadcasting of the Olympics - for which it pays close to $1 billion for the rights to air. The last time that the Olympics was held in a similar timezone to the US was when it was in Atlanta 20 years ago and over that two week period Comcast aired 172 hours of coverage.
This year in Brazil, Comcast is going to be providing over 6,000 hours of coverage over the same two week period, across a number of devices (not just on TV), on demand, personalized, via its X1 product. He said:
The best way to show you the intersection of media and technology is to show you what we have planned for the Olympics this year.
If you use the X1 product, you would say that that’s what TV ought to look like in 2016. The reason that we think that that’s so important, focusing on product excellence and technology excellence, is there is just this need to connect with all this great content.
This year we are providing over 6,000 hours of coverage over that same two week period. Which if you try to watch back-to-back is about 270 days worth of content. Obviously you can’t watch all of that, what we are trying to do is enable people to follow the team, the countries, the sports that they want.
Watch all that content and really realise that long-tail on every screen in their life.
Challenges and opportunities
He gave the example of X1’s ‘Last 9’ capability, which allows viewers to pull up a little bast that has the last nine things that they watched - whether that be on demand, on a web video, a linear show, or something on their DVR. Kotay explained:
We have a very classic micro-services framework architecture. We aggregate the last things that you watched, what your progress might be on them, whether that show is still on, what might be on that linear station that’s airing. The fact that everybody does it at once, translates into some pretty huge concurrency.
So the micro=services aggregator that has to power Last 9 at peak, has around 100,000 requests per second. Around 5.5 million per minute. And might do about 2.5 billion requests in a peak hour.
Even in the off-peak times, it equates to about a million requests per minute. Television just gets viewed a whole hell of a lot. So we love the whole micro-services model and the convenience of the APIS, but it really does require us to think about not just the data model but a whole system model behind it.
Comcast’s micro-services model, supported by MongoDB, also allows the company to think in terms of not just availability, but performance and outcomes. Kotay made the comparison to uptime - which is typically measured by the number of 9’s after 99%. So for example, 99.99% would be to two 9s availability. He said that performance should be measured in this way too:
And so when we think about that, one of the ways we really think about driving a really high performant experience is by focusing on performance. So we treat our measurements around performance in the same way that we do around availability.
We talk about performance not in the averages, because the average is for the average. But we talk about performance in the context of 9s. We start measuring performance at two 9s, then we talk about three 9s, four 9s and five 9s. It’s the same term we use for availability, why wouldn’t we use it for performance as well?
What we are really talking about is all of the database principles, applied to deployment. Applied to how you think you’re rolling out your services, not just your data in the data model.
What we found is that this can improve not just your velocity, but your quality. Both. We were excited when we launched X1 when we were doing updates every two weeks to the product. These days we do multiple updates a day. This year we will do something like 16,000 updates over the course of the product.
However, this new micro-services model, with a focus on outcomes and performance, with regular updates, also has its challenges. Particularly when it comes to testing. Kotay said:
We have 20 million devices that run X1 and we update all of them every three weeks. So you even have to take a different approach to testing. So we have this idea of stochastic testing and stochastic deployment.
We can’t run all of the regression test cases for all of those devices at all times. So what we do is run a subset of tests on a subset of devices and rotate them across the whole population. Same thing on the deployment side. We never want to rollout to the same 1%, or the same 10%, you want to be deterministic but different every time.
If you really want to have a culture that drives towards high performance, that is a really a culture around delivering high results, you have to be ruthlessly focused on outcomes. So we view excellence as being not a skill, but a mindset.
