People ask me all the time how I am able to track the CDN market so closely, if I ever sleep, and how I can keep my finger on the pulse of so many different CDN topics. I’ve spent the last fifteen years tracking the CDN market, talking to vendors, customers, policy makers, telcos, carries, MSOs and Wall Street money managers. I’ve had the privilege of discussing business and technology topics with CEOs at the largest content companies and vendors all around the world. They have trusted me by giving me details they don’t give to others, without an NDA, and with the understanding that I won’t disclose the info without their permission. They know my single purpose and the reason why I am still in this industry is to provide as much education and help I can, free of charge, 24 hours a day, so the entire industry grows.

As a result of who I talk to, I have a lot of details on pricing, vendor revenue and growth, new contract wins, acquisitions and other private details. Whenever possible I use this data, when allowed, and share it in the market. Probably 90% of all the details I get from talking to customers and vendors I can never blog about or discuss, but that info still helps me to learn more about the market. It helps to fill in the gaps and provides me with data needed to keep a close eye on the market drivers and restraints.

I really do appreciate all of the people who read what I write, have such an interest in what I think about the CDN market and spend time talking to me. That is what drives me. So while I do have a lot of data and first hand knowledge on what’s really taking place behind the scenes, I wanted to point out that I don’t know everything there is to know about the CDN market and CDN related technology. The term CDN covers a lot of technologies now; streaming, large object delivery, web acceleration, FEO, LCDN/MCDN and a host of other related infrastructure subjects like peering, transit and the like. Someone recently asked me how I got invited earlier in the year by the U.S. Senate to go to DC and speak on topics pertaining to net neutrality, telecom mergers and content delivery architectures considering I’m not a network engineer. The answer to that question really highlights just how much help others give me.

I get to speak to the smartest people who are building, running and using the largest private and public CDN related networks around. There are a lot of people smarter than me and while I eat, sleep and breathe CDN, the access people give me and the time they take to educate me on topics is why I can do the detailed blog posts that I do, and get invited to speak at events. To many, the title and subject of this post is going to seem odd. But to those who know why I do what I do, they understand that I believe that there will always be more to learn, those who know more than you do and you will never know it all. One’s self-importance is not based on what they know, but what they can share with others, which is my only goal. Thank you to those who continue to help educate me on various CDN technologies, deployments and use cases.

For major sporting events, it is no longer enough to produce a traditional broadcast television program alone. Fans want more access to their favorite players, sports, and teams in real-time. Through live webcasting, fans now have a VIP pass to almost every aspect of a major sporting event. At the Streaming Media West show, [taking place November 17-19 in Huntington Beach, CA] you can learn from top sports brands how they are expanding their reach for large-scale sporting events with live streaming. The session will cover driving fan awareness of live, sports-related content with shoulder programming, social media integration, distribution of highlights over a variety of platforms, and more. Confirmed speakers for the session include:

• Moderator: Philip Nelson, Chief Relationship Officer, Newtek
• Mike McLeod, Senior Manager, Advertising Tech & Products, PGA Tour Digital
• Andre Mika, SVP, Programming and Production, 120 Sports
• Allen de la Cruz, SVP, Engineering, Major League Gaming

Register online using the code 200DR for a “Discovery Pass” and get free access to the keynotes, exhibit hall, discovery track sessions, and receptions at #smwest.

On Tuesday, M-Lab released a new study on the impact of network interconnection on consumer Internet performance. The report entitled “ISP Interconnection and its Impact on Consumer Internet Performance“, details findings based on the speed test results collected by its test servers for various ISPs throughout the country over a roughly two-year period. For those not familiar with M-Lab, they provide the largest collection of open Internet performance data used by the FCC, amongst others, for the Measuring Broadband America program.

M-Lab data shows that around May 2013, suddenly and simultaneously throughout the country, speed test results for many ISPs (AT&T, Comcast, CenturyLink, Time Warner Cable, and Verizon) experienced a sudden and significant decline in performance to a specific set of transit providers (Cogent, Level 3 and XO). Just as suddenly around March 2014 the performance returns to normal for most of these same ISPs. Coincidentally, a few other ISPs who Netflix had negotiated direct Open Connect connections (Cablevision and Cox) did not experience similar decline in performance. The data presented in the study confirms what myself and others have surmised about Netflix being ultimately responsible for the dramatic, simultaneous decline in Netflix performance for all non-Open Connect ISPs.

If you look at the M-Lab measured history of the congestion, you will notice that these timelines line up very closely with Netflix’s migration from 3rd party CDNs onto their own Open Connect platform. The performance impact also matches closely with ISPs that did not agree to provide Netflix with Free Peering while other ISPs that agreed did not experience a performance impact.

Looking at Figure 1 from the report (below), we can see that performance suddenly degrades for three of the four major broadband companies in the NY metro area according to an M-Lab test server housed on Cogent’s network in NYC around May 2013 and then performance suddenly improves for all three around March 2014. This tight coordination of impact for multiple ISPs simultaneously suggests that the cause was not something done by the ISPs, but rather by another entity. (Note: I added the heading and arrows to the chart)

What entity might be responsible? Well, figure 2 shows us that the fourth broadband ISP in the NY metro area testing on the M-Lab server on Cogent’s network, Cablevision (the only one of the four with a direct connection to Netflix’s Open Connect CDN) did not experience the same sudden drop/rise in performance over their link to Cogent.

Finally, M-Lab’s report also helpfully includes performance results for all four broadband ISPs in NY from a test server located on a different backbone connection (one that was not providing transit service to Netflix) showing no sudden performance changes for any ISP.

The report also shows that direct interconnection agreements between Comcast/Netflix increased performance for other ISPs. Unless there were performance issues further upstream of the interconnection, there should have been no impact on the interconnection agreement between Comcast/Netflix on other ISP networks. And according to M-Lab’s findings, performance issues on ISPs networks were not due to technical issues but rather the business deals between ISPs. They say, “we were able to conclude that in many cases degradation was not the result of major infrastructure failures at any specific point in a network, but rather connected with the business relationships between ISPs“.

While some may want to take this report as a smoking gun that ISPs are causing congestion, they may forget, not understand, or purposely leave out, the fact that large content providers control the delivery of their traffic and can AVOID congestion. A recent MIT study “Measuring Internet congestion: A preliminary report” pointed out the fact that the ISPs singled out in this report have multiple alternative paths to reach them. The report states that, “Congestion at interconnection points does not appear to be widespread. Apart from specific issues such as Netflix traffic, our measurements reveal only occasional points of congestion where ISPs interconnect. We typically see two or three links congested for a given ISP, perhaps for one or two hours a day, which is not surprising in even a well-engineered network, since traffic growth continues in general, and new capacity must be added from time to time as paths become overloaded.”

Most agree that when Netflix, again, moved their traffic off of these newly congested paths to direct connections, performance improved both for Netflix services as well as other services impacted by this new congestion. What is puzzling however is the timing of this improvement. If you look at the graph above you will notice that all ISPs improved simultaneously in Feb 2014. This is the exact same time that Netflix and Comcast migrated traffic to their direct connection. While it is understandable that Comcast would improve, no one has explained how a Comcast direct connection would improve AT&T, Verizon, and Time Warner unless there were additional problems between the Netflix server and their transit ISPs themselves. When Netflix moved this traffic their congestion within their transit ISPs improved other destinations.

What M-Labs is trying to do is good for the Internet, but they need to expose more of the end-to-end problem. If they truly want to understand Internet congestion and user experience, they need to not only focus on interconnect, but they also should expand their measurement to the quality of transit ISPs and acknowledge the choices content sources make when delivering traffic to their customers. For example, a measurement can identify if there are material differences between a variety of OTT sources such as Amazon Prime, Netflix, Hulu and YouTube on a given ISP. If Amazon Prime HD video quality was excellent, but another source was poor, it would be interesting to determine why that’s occurring, and what options the content provider has to improve their services.

While many were quick to blame ISPs for problems consumers were having with their Netflix streaming experience, we’ve now have a lot of data in the market showing that the choices Netflix made directly impacted the quality of their video and other services as well. Between this new M-Lab data, the interconnection findings published by David Clark at MIT/CAIDA, this data, and a recently published research report that says Netflix is using calls for greater net neutrality to drive down the prices they pay, it’s now clear just how much control Netflix really has over the quality of video they deliver.

When it comes to streaming video online, Netflix has been doing it longer than almost anyone and has some of the best quality video available. If you’ve ever wanted to know how Netflix gets such good quality and how they encode their videos, then you won’t want to miss their keynote on November 19th at the Streaming Media West Show. David Ronca, Director of Encoding Tools at Netflix will detail their encoding service and the key engineering decisions they made around video encoding. Some of the topics that David will be cover include scalability, progressive and parallel video encoding, and Netflix’s 4K video encoding pipeline. Register online using the code 200DR for a “Discovery Pass” and get free access to the keynotes, exhibit hall, discovery track sessions, and receptions. #smwest

This morning Amazon announced a new HDMI streaming device called the Fire TV Stick, which is already available for pre-order and will ship before black Friday. (Nov. 19th) Amazon is offering Prime members, or those who sign up for Prime, special pricing on the stick for only $19, for the next 48 hours, after which it will retail for $39. Many might be quick to compare this to Google’s Chromecast stick, but unlike Chromecast, Amazon’s Fire TV Stick won’t require another device to use it. Amazon’s streaming stick comes with a remote and most, but not all, of the functionality of their $99 Fire TV box.

The Fire TV Stick has a dual-core processor (Broadcom Capri 28155, dual-core 2xARM A9), 1GB of RAM, 8GB of storage, Bluetooth 3.0 and dual-band, dual-antenna Wi-Fi (MIMO). From a CPU standpoint, on paper it beats Roku’s Streaming Stick and Google Chromecast in terms of performance many times over. The Fire TV Stick comes with the same user interface as the Fire TV and has support for all of the current content channels on the Fire TV, with the exception of some gaming apps. Casual games can still be played on the stick, like Angry Birds, but some complex games, requiring more controls aren’t supported.

The remote that comes with the Fire TV Stick doesn’t have the same voice search functionality as the Fire TV remote, but it is compatible with the voice remote if bought separately for $29. Users can also use a free Android app on their phone to search for content with their voice and Amazon says an app for iOS is coming shortly. Non-gaming apps that are written for Fire TV will automatically work on the Fire TV Stick, which is also based on a flavor of Android. Amazon also confirmed that they are on track to have support for HBO Go on the Fire TV before the end of December, with the Fire TV Stick getting HBO GO in the new year.

Amazon told me they have “made a lot” of the Fire TV Sticks in preparation for the pre-orders, but in usual Amazon fashion, won’t disclose how many are available before they sell out or if they expect there to be a shortage during the holidays. Amazon has placed a buying limit of two devices per Prime member, at the special $19 price point.

While Amazon’s new stick is good for consumers, it’s bad news for Roku, which now has even more competition, at a lower price point. I love my Roku box and it still has more premium content channels than any other $99 or less device, but it’s only a matter of time before Amazon’s streaming stick, or Google’s new Nexus Player for that matter, catches up with Roku in terms of content choices. Roku simply can’t compete with Amazon or Google’s marketing dominance and their ability to package these products in with other services.

In the long run, the non-gaming streaming device space is going to be won by Apple, Google and Amazon. They all control multiple devices that tie into a larger ecosystem, make money from many avenues and at some point, Amazon or Google is going to get the price on these sticks down to where they are free. It’s not hard to imagine Amazon giving them away with a Prime membership or Google giving it away with a Nexus tablet or phone. At $19, the price point doesn’t have far to go before it reaches zero.

I’m pleased to announce that Cory Mummery, VP/GM of NFL Now will be the keynote speaker on the first day of the Streaming Media West Show, taking place November 17–19 at the Hyatt Regency Huntington Beach Resort & Spa in Huntington Beach CA. Come hear about their new cross-platform digital product, NFL Now, that packages all available NFL produced live and on-demand video from NFL Films, NFL Network, NFL games and all 32 NFL teams for viewing on multiple devices. Register online using the code 200DR for a “Discovery Pass” and get free access to the keynotes, exhibit hall, discovery track sessions, and receptions. #smwest

The $99 streaming device market just got a little more crowded with Google’s announcement of their new streaming box, dubbed Nexus Player. Built in partnership with ASUS, it’s the first device to run Android TV and also allows you to play Android games on your TV, with a separately priced gamepad. Unlike Google’s Chromecast USB stick, the Nexus Player includes an app guide, content recommendations and voice search controls. It’s also Google Cast Ready so you can fling content from Chromebooks and Android/iOS devices to your TV. Inside the device is a 1.8GHz Quad Core, Intel Atom chip, 1GB of RAM, 8GB of storage, HDMI out, with WiFi support for 802.11ac 2×2 (MIMO). The lack of an ethernet port is a big downside as I’ll take the reliability of an ethernet cable over WiFi any day. The box is up for pre-order tomorrow and will be in stores on November 3rd.

The initial content options available at launch are limited on the box with the major ones being Google Play, Netflix, Hulu Plus, YouTube, Vevo, Pandora, iHeart Radio and support for Plex. Other apps from those like the Food Network and PBS Kids are also available, but content choices are pretty limited right now. Other boxes, including Amazon’s Fire TV were also limited in their content choices at launch, so we can expect to see a lot more content come to Google’s new box fairly quickly. I expect it won’t take more than a year before Google’s $99 Nexus Player will be very similar to Roku, Apple TV and Amazon’s Fire TV, with regards to content choices.

With so many streaming options in the market when it comes to Smart TV’s, game consoles, connected Blu-ray players and $99 streaming boxes, Google is entering a very crowded consumer market. But the advantage Google, Apple and Amazon have over everyone else is that they all operate and control an end-to-end video ecosystem. In the long run, they will all be the winners in the $99 streaming box market, and devices from Netgear, Sony, Western Digital and others stand no chance with their boxes. Just within the last year alone, devices from Vizio (Co-Star), D-Link (MovieNite Plus), Hisense (Pulse), Sony (SMP-N200) and Seagate (GoFlexTV) have all been discontinued in the market.

Long-term winners in the space will be Apple, Amazon, and Google for dedicated streaming boxes/USB sticks, and Microsoft and Sony with their more expensive gaming consoles. Where this leaves Roku in the long-term is unknown, as they still have the best $99 streamer in the market today when it comes to content choices, but that gap between them and others keeps shrinking. Roku doesn’t have the marketing power of Apple, Amazon or Google so it’s going to continue to be a tough fight for them in a very crowded market.