At my recent EdgeNext Summit, there was a lot of discussion about how edge computing can be used to deliver next generation OTT viewing experiences with low latency. Traditional HTTP streaming formats such as HLS and MPEG-DASH have gained wide acceptance, making it relatively easy to reach viewers on almost any viewing device at global scale. However, these formats have significant limitations when used in traditional live streaming workflows, including slow startup, stream latencies, and latency drift over time. By utilizing edge compute resources, new methods and formats are emerging to address these limitations and improve the experience for viewers and content distributors.

Live streaming has some inherent challenges that impact infrastructure requirements differently from traditional VOD delivery. Demand for live events can grow quite quickly, requiring instant scalability. For example, the recent online trivia craze has seen demand for online streams grow from zero to over one million viewers in just a matter of minutes, creating massive challenges in ramping to meet this instant demand. Unlike traditional on-demand workflows where popular content can be pre-cached in multiple locations to reduce bottlenecks, live content must be ingested, packaged, and pushed to edge locations as it is created. And with growing viewer frustration around poor user experiences when it comes to streaming live events, caching and buffering of live content to gain scalability and ensure reliable playback becomes a greater challenge.  

Traditional cloud service providers offer hyper-scale data centers with computing resources, but they are in relatively few locations and often not located in densely populated areas where viewers may reside. Sending source video streams from these centralized cloud data centers to viewers in distant locations requires extensive middle-mile capacity and can create peering bottlenecks as viewership grows. To viewers, this often means slow or inconsistent startup of video streams, video quality degradation as more viewers watch the event, or even the inability to join popular live streams. These quality, reliability and scalability challenges are impeding the way consumers think of live streaming as a true replacement to cable TV.  

Edge computing can help alleviate these bottlenecks by sending source streams to edge servers that perform stream splitting and replication functions. By locating these edge servers in more locations across the globe, edge computing provides the ability to scale up to meet the demands of even the largest live events. By scaling at the edge, traditional bottlenecks are avoided and transit costs are reduced by serving streams from locations that are closer to viewers. And by distributing this capacity in metropolitan areas where viewers are located, it ensures lower latency and higher QoE by reducing the distance the packaged stream must travel to the viewer, reducing transit costs and helping to eliminate peering bottlenecks. The result is greater scalability to provide higher-quality viewing experiences.

One of the live streaming technologies that many think is ideally suited for edge computing is WebRTC. Source video feeds can be ingested through local ingest locations using traditional low-latency streaming formats such as RTMP. The source video can then be pushed to edge servers around the globe where the RTMP feed is converted to WebRTC through edge compute instances running in locations close to viewers. Unlike traditional HTTP live streaming formats, WebRTC uses UDP instead of TCP. UDP streaming takes advantage of modern fiber-based IP infrastructures that utilize hardware-based switches and routers to deliver higher sustained average bandwidth and picture quality to viewers. 

WebRTC also promises to open up new interactive workflows by allowing viewers to simultaneously watch live streams at the same time. With the growing popularity of live online gaming such as Fortnite and the legalization of sports betting in the U.S., the ability for content distributors to deliver synchronized interactive online live streaming will help drive the increased consumption of live content and the need for better edge computing functionality and scale.  

One of the first CDN companies that has implemented this approach to using edge compute for scalable live video streaming is Limelight Networks. And with their recently announced partnership with Ericsson to leverage its UDN Edge Cloud Platform to provide edge computing and delivery within carrier networks should provide even greater scalability and capacity for live video streaming at the edge. This will become increasingly important as 5G begins to be rolled out, as I noted in my recent blog Here’s Why Today’s Video Infrastructure Is Not Ready For 5G, And How Edge Technologies Can Help.

All CDN vendors are talking about ultra-low latency video delivery, but not a single one will tell you it’s easy to do at scale. It takes a lot of resources to deploy and adds overhead cost to operating their service. Aside from the technical challenges in scaling, CDNs also have to target specific customers who see the value and are willing to pay more for the improved user experience. In a recent survey I did of over 100 broadcast and media customers, 80% of them said they wanted ultra-low latency functionality, but were not willing to pay more for it. Many expect the functionality to be part of a standard CDN delivery service. 

By utilizing edge computing capacity within carrier networks as well as in globally distributed data centers to scale and distribute live video streams, content distributors want to reduce costs while ensuring the highest quality live viewing experiences on both mobile and fixed devices. Edge infrastructure will help make this a reality, and it’s going to take time for the business model to be figured out, but it will happen. I’m excited to see what new applications will be enabled in the live video world from the edge and with low-latency. They are coming.

Season 7 of Epic Games Fortnite kicked off today on PS4, Xbox One, PC and the Nintendo Switch and multiple ISPs I have spoken to have said it’s made up some of the largest percentage of traffic they have ever seen on their network. A few ISPs reported Fortnite accounted for more than 50% of their inbound internet traffic (peering/transit/CDN/caching) at peak today, with one telling me it was as high as 62% of their traffic.

Microsoft is delivering a lot of the traffic themselves from their own network, but I also see some coming from CenturyLink, Limelight Networks and I know additional third-party CDNs are also involved. While the volume of bits is high, the price point the CDNs are getting paid is very low, and it’s downloads, not streaming, so it’s rock-bottom pricing. Some CDNs even passed on doing the delivery due to the low price point.

In addition to Microsoft handling much of the delivery themselves, ISPs that took advantage of  Microsoft’s free virtual cache software are managing a lot of the downloads themselves. So far, I haven’t heard of any problems with the update on a network level, just that it’s a lot of traffic.

What is being billed as the first ever live PPV event for golf is going down in flames as Bleacher Report’s live stream of the Tiger Woods vs. Phil Mickelson match is experiencing massive failure. The stream, which costs $19.99 to watch, is having a long list of authentication issues, app problems and missing audio amongst other complaints. From what I can tell so far, the issues all appear to be on the front-end and not with the CDNs delivering the video as you can’t even get to the video to watch it. Whichever vendors(s) are responsible for the failures are costing Bleacher Report a lot of money and will need to issue massive refunds.

I’ve tried to get the stream on multiple apps and desktop and have gotten errors of “oops, something went wrong. Please try again later,” along with blank screens, no options to actually purchase the event and a message saying “check back closer to the event time”, even though the even started more than 30 minutes ago. Also a message of “this video cannot be played” is a frequent message many users are reporting. On the desktop, I was asked to sign in with my Google account, but when I tried, got an error 403 message saying “This app has exceeded recommended rates of users signing in with Google.”

Looking at Bleacher Report’s Twitter feed, users are fuming with the most common comment being that even after purchasing the event, a message of “this event is not purchasable yet”, appears. Other errors users are reporting include:

– Says too many devices are streaming
– Error retrieving assets for commerce
– I purchased and went back to app and it says not purchasable yet
– Stream isn’t working on computer or Roku TV app
– We paid for the match on the app but the app isn’t working
– Stream is not working at all on smart phone or Roku!
– There is no buy button

While Bleacher Report is using many third-party vendors for this failed webcast, that’s no excuse for the company to not even acknowledge the problems they are having. Two hours into the problems they haven’t posted a single Tweet updating users and many are leaving comments that the support number provided is busy and they can’t get through. I also haven’t seen Bleacher Report reply to a single user and their strategy of just ignoring the problem is about as bad as you can get. Making matters worse, all week Bleacher Report has been promoting the event on social media saying “Because watching live sports should be easy”. Ouch. But this is what happens when you don’t plan properly, don’t use vendors that know how to execute large scale webcasts properly and have a failover strategy. This PPV webcast now equals the MacGregor/Mayweather PPV from a disaster stand-point.

Updated 4:49pm ET: The stream is now working, for free, via the website at https://live.bleacherreport.com/tiger-vs-phil/

At the Streaming Summit, taking place as part of the NAB Show New York on Oct. 17-18th, Jai Krishnan, Product Manager at Chrome and Nathan Egge, Video Codec Engineer at Mozilla will share a look at AV1, the royalty-free video codec finalized by the Alliance for Open Media earlier this year. AV1 offers a generational improvement in coding efficiency over any web codec, and comes backed by nearly 40 companies, including some of the largest in the internet, tech, and video industries.

Their talk will cover why Google and Mozilla invest in web codecs, the benefits of AV1, where the codec stands going into 2019, and demos of AV1 already available in browsers today as well as on different platforms and for different use cases. (session details)

With 100 other speakers and 40 presentations and discussions, it’s going to be a great two days covering the entire streaming media technical and monetization workflow. Use my personal discount code of dan18 and get a pass for only $595. #streamingsummit

You can get the same result from a well-produced mobile phone stream as you can get from a production truck, satellite backhaul and encoding in a NOC. So how do you decide on which equipment and workflow pieces to use for different situations?

At the Streaming Summit, taking place as part of the NAB Show New York on Oct. 17-18th, one session of seasoned webcasting professionals will discuss what decision factors go into producing a live stream from a remote location, with a week of lead time with can’t-fail expectations. They will also outline how to arrive at a solution based on budget, allowable risk and fault tolerance while still producing a high-quality webcast for viewers. (Session details)

With 100 other speakers and 40 presentations and discussions, it’s going to be a great two days covering the entire streaming media technical and monetization workflow. Use my personal discount code of dan18 and get a pass for only $595. #streamingsummit