Over the past two days, there’s been some news articles suggesting that some ISPs, like Verizon, may be purposely throttling or slowing Netflix’s video content over their last mile network, because of the recent Net Neutrality rulings. While that sounds like a great headline and something to get people all worked up over, that’s not what’s really taking place. The reality is that business decisions that Netflix and ISPs make regarding quality of service determine how good Netflix or any other content looks when streamed to consumers. Both Netflix and ISPs are constantly having to add capacity to their networks, of all kinds, and make decisions on how much money they want to spend versus the level of quality they want to offer.
To make it simpler, if Netflix or the ISP can deliver content right now and the video starts up within three seconds and has SD quality, they have to decide if it is worth spending the additional money to make it startup within one second and have HD quality. From a business decision, it does not help any ISPs business to improve the quality past a certain point. Each ISP decides what that point is and the quality that is good enough, versus how much they would have to spend to improve it. These decisions that Netflix and ISPs make determine how much transit, peering and other technical resources they deploy and what impact it has on their bottom line. While the technical pieces like servers, transit, peering etc. is what makes streaming services work, it’s the business decisions made behind the scenes that determines the user experience, not Net Neutrality.
From a technical standpoint, it’s also extremely complicated how Netflix’s content is delivered as they have a total of three different CDNs, all delivering content in different ways. Some content is delivered inside last mile networks via Netflix’s Open Connect program, some is delivered via third party CDNs like Level 3 and Limelight Networks and other content is delivered via servers Netflix controls outside the last mile. When it comes to ISPs, no two are identical in footprint and number of subscribers so there are many variables. Netflix buys transit from many different providers, which come at different price points and with different SLAs.
None of the big ISPs buy much in the way of transit and those that do, buy it primarily for minor destinations they can’t get to through peering. The majority of U.S. originated traffic comes through peering points into the big ISP networks and ISPs who choose to, can make those runs congested for many hours a day. The whole process of getting Netflix content, or any video for that matter at scale, to consumers is very complex. Many of those who write about Netflix streaming don’t understand all of the pieces involved, how they all tie together and are quick to point fingers as things like Net Neutrality, which isn’t the case. There are also a lot of issues on the consumer side, inside the home, that affect the quality of video being delivered where many times, it’s not the ISPs fault for the poor experience.
I had an issue over the weekend where Netflix would not stop buffering on my Xbox 360, but once I used the Roku, streaming was perfect. After spending time to diagnose the problem, it turns out my Xbox 360 was having NAT problems with my router and even though it had a static IP, it had to be flushed out and setup again. In a lot of these types of cases when there is a problem, most consumers don’t know how to do a trace route, assign a static IP, log into their router and really diagnose the problem. ISPs can and do have issues, but I’m willing to bet that more than 50% of the time the poor QoS is due to the device, WiFi or local network issue inside the home.
Some are suggesting that because Verizon, Comcast or other ISPs have been dropping in their ranking via Netflix’s “ISP Speed Index Results“, it’s evidence that these ISPs are purposely making Netflix streaming from their network look bad. That makes no sense at all and would only hurt the ISPs business. If it looked bad enough and impacted enough customers, many would leave. Some can’t as they only have one ISP in their area, but many have choices. Also, keep in mind that the ISP ranking that Netflix provides is NOT comparing apples-to-apples nor does it even say what exactly it is defining. As usual, no one seems to question the data that many of these companies present to the market. Netflix does have a lot of very detailed performance data and intelligent video player that makes real-time delivery decisions based on that data, but Netflix doesn’t share those metrics or their methodology of how they say one provider is better than another.
Google Fiber is at the top of Netflix’s ISP Speed Index list, yet has the smaller footprint of any ISP in the top 10 of Netflix’s rankings. While Google refuses to reveal subscriber counts or uptake numbers for Google Fiber, research reports predict it will be another two years before they have 3 million subs. So even if they have 1 million subs today, it’s easier and cheaper for them to make the Netflix experience on their network the best it can be, compared to Comcast that has 20 million Internet subscribers. You can’t compare the two fairly.
In addition, what exactly are Netflix’s ISP results measuring? At the bottom of the Netflix page, it says the results “reflect the average performance”, but then they don’t define what methodology is being used to define performance. The only thing they show is the “Average Speed in Mbps”, but what exactly does that mean? That’s the speed at which the streaming is being delivered, but that measurement alone doesn’t show startup time, buffering, or a whole host of other factors. Where is Netflix methodology of exactly what is being measured and shown in their ISP charts? Netflix was very smart when they launched the “ISP Speed Index Results” as it puts the pressure on ISPs to be seen in a good light from their customers. It also means that in most cases, instead of the customer complaining to Netflix when they get bad quality video, they will instead complain to their ISP, but that may or may not always be fair. Some ISPs simply don’t appropriately add capacity between their network and all of their peers, while other times it’s Netflix, not the ISP who has made decisions that impact quality.
While there is this idea that Netflix has moved most of their video delivery to inside the last mile, by working with ISPs, the reality is that in the U.S., Netflix is still delivering a lot of their own streaming, via third party CDNs Level 3 and Limelight Networks as well as Netflix’s own caches. From what I have seen, only Cablevision, Suddenlink and Google Fiber have allowed Netflix to put caches inside their networks so many times the quality issue a consumer is having might not be the ISPs fault. It could be the CDNs, the connection between the CDN and the ISP or a whole host of other factors. It’s true that some ISPs won’t add more capacity when they need it, but they have made the decision that it does not make sense from a business standpoint, based on the return. At some time, the ISP has to decide the trade off between quality and how much money they spend upgrading their network. So while no ISP wants to come out on record and say it, many see Netflix as a threat and don’t want to give a competitor any advantage in the market.
It’s easy to use Net Neutrality as an excuse for why there are quality issues with streaming, but that’s not accurate. Delivering video over the web has inherent flaws and it’s not like traditional broadcast distribution that scales much easier. It’s one of the main reasons why pay TV won’t be replaced by Internet streaming at scale. It’s too expensive to support so many eyeballs while also providing the best quality possible. Case in point. If all we wanted was to have better video quality, why doesn’t Netflix encode their videos at a higher bitrate, say 8-9Mbps and deliver all of their content that way? Reason is that the ISPs would have trouble delivering it with good quality, which is the whole reason why Netflix is trying to get ISPs to join Open Connect and allow Netflix to put servers inside the ISPs network. While that sounds like a simple and free solution Netflix is providing to ISPs, the reality is far from that.
This is the problem that Netflix is currently facing with most of the major ISPs in the U.S. who won’t join Open Connect, which has made Netflix’s current multi-tiered approach to video delivery complicated. Even though Netflix is not causing a lot of people to cancel their cable, many ISPs sell pay TV services, have their own OTT services and see Netflix as a competitor. Some ISPs have also told me they would rather deploy a solution like transparent caching which would help the ISP with caching all types of video content, not just Netflix’s. Netflix has to make some decisions about how they work with ISPs, what role third party CDNs will have delivering their content and how much of a network they want to own and operate on their own. The good news is that I think all of these things can be worked out over time, amongst Netflix, ISPs, CDNs and transit providers, but it’s going to require Netflix to change their current approach. I’ll have more on that in a follow up post.
Programming for the Streaming Media East show, taking place May 13-14 in NYC, is coming along nicely with more than 50% of the speakers now placed. We’ve gotten a lot of great proposals this year and spots are filling up much faster and earlier than past shows. With 50% of the program now filled, if you wait too long to get back to me about what session(s) you want to speak on, they may no longer be available. Many companies have the agenda and are looking it over, but if you wait weeks to decide, spots will be gone.
Over the past year, there has been speculation of when or if Apple would build out their own content delivery network for the purpose of delivering content to consumers. To date, all of the CDN deployments done by Apple have been for internal purposes only, but that’s about to change. For the past few months, those in the networking community have been discussing some of the new people Apple has recently hired, all of whom have a deep background in building out large-scale networks. These employees are part of a new group that was formed inside Apple last year, who are tasked with the build out of a new CDN for external use.