Some Tech Details from Netflix’s Boxing Stream and What To Look For on Xmas Day
Since Netflix’s live boxing event, I’ve spoken to more than half a dozen ISPs globally, as well as others who had direct insight into the traffic during the event. Based on those conversations, info from manifest files, error codes, and other sources I will not disclose, here’s part of what happened during the event and what to watch for during Netflix’s NFL games on Christmas.
Post-event, some quickly suggested that ISPs ran out of “capacity,” without even defining what capacity they were referring to. For most ISPs, almost all of Netflix’s traffic was carried on private network interconnects and appliances. Of the seven ISPs I spoke with, located in the US, Europe, and another location, none experienced any capacity issues with peering, transit, or private interconnects. In conversations with ISPs and what some ISPs posted online, they said:
- “We have plenty of capacity with them in the form of OCAs and Private interconnects.”
- “We had plenty of OCA bandwidth capacity and overhead available.”
- “None of our paths that feed our OCAs were at capacity.”
- “We had a healthy overhead of Netflix interconnect/OCA.”
- “None of our connections with Netflix were saturated.”
Multiple ISPs reported that they had issues with the fight and regular programming streaming from an OCA closest to them, but only the fight had problems, which is very telling. That’s why Netflix’s SVOD content and other non-Netflix streaming services looked fine. Netflix deploys caches differently based on the ISP and region of the world. Some ISPs have Netflix caches in some of their larger telco central offices and cable headends. In contrast, others have direct private peering connections with them at core sites, in addition to other connection options.
Netflix did not use any third-party CDNs for the boxing event and is not currently adding any third-party CDNs into the mix for the NFL games. Anyone who said that Netflix used them for the boxing event or wondered if they did should stop guessing. It’s public information, as you can see from traceroutes and by speaking with those involved in the event. Netflix is utilizing the same infrastructure to deliver live streams and VOD content; some statements on LinkedIn that Netflix is using a “separate” infrastructure for live streams are not accurate. Netflix spins up and down resources needed for their Open Connect servers and doesn’t have any hardware “limitations” across Open Connect for live, like some want to suggest.
Looking at data from manifest files and other sources, it was interesting to observe that Netflix was using 1-second segments, which was surprising. One of the key technical issues I observed was that manifest files were not refreshing at times, resulting in connection errors. I sorted through more than 1,000 posts on Twitter from users who provided screenshots of errors, and the three errors below were the most commonly reported, based on my review.
- Netflix Error TVQ-PB-101 (5.3): This error happens when an issue with data stored on the user’s device stops Netflix from playing. It requires a device restart.
- Netflix Error TVQ-PB-101 (3.2.400): This error occurs when a network issue prevents the device from accessing Netflix. Users who encounter the error on any device other than a Roku or smart TV are directed to contact Netflix on their website to help “investigate” the issue. However, Netflix’s support site provides no suggestions for resolving the problem.
- Netflix Error: UI-113 and NSEZ-403: This error occurs when a network issue prevents the user’s device from reaching Netflix and connecting to their account, with Netflix suggesting to “try again later.”
Some LinkedIn posts suggested that Netflix should have prepared better for the event by “simulating” more traffic in advance, which is a ridiculous statement. Those making such comments demonstrate a lack of understanding of how live events work. No company can simulate 65 million simultaneous streams across any network in a real-world test. As Netflix said post-event, “It’s impossible for our engineering team to test that magnitude of traffic and viewership unless they have a real, live thing, which is what happened.” It’s also important to note that Netflix had no previous streaming telemetry or viewing data specific to live content in any significant quantity, so it couldn’t determine from a regional standpoint where the most viewership would come from. They were using VOD viewership data to try and figure out live traffic, and it’s easy to warm up caches for VOD, something you can’t do with a live stream.
To optimize stream performance during a live event, you need sufficient real-time network telemetry and observability data to inform your tuning decisions. The boxing event was Netflix’s first large-scale live streaming event on its platform. I don’t know how many logs Netflix was processing per second, but for a recent live event with an AMA of just over 10 million, the content owner told me they were processing 15 million log records per second. With Netflix having so much data from the boxing event to put towards tuning QoE for the NFL games, I would expect the user experience for the NFL stream on Christmas to be vastly improved over the boxing event.
While no one knows how many simultaneous viewers Netflix will get for the NFL games on Christmas, I expect it to be less than the boxing event. The boxing stream attracted viewers from various regions worldwide, as the event was more of an entertainment spectacle than a professional boxing match. The NFL games will see increased traffic from the US, and Netflix knows which US regions will have the most viewership for the game based on the team’s local markets. That said, Beyonce performing at halftime is an entertainment event, not a sports one, so I would expect Netflix to see a surge of non-football viewers if they can time the halftime show accurately.
A not-often-mentioned but critical point is that Netflix’s boxing event was initially scheduled for July 20th. If the event had occurred on that date and not been postponed to November 15th due to Tyson’s injury, Netflix would have had 5 months between the boxing event and the NFL games on Christmas to help tune its network. However, due to the schedule change, Netflix’s 5-month window to prepare for Christmas dropped to 45 days. Those who produce live events for a living understand the level of what Netflix is looking to achieve at a scale not seen before, with a timeline that is unheard of. Adding to the complexity is that a Netflix executive was recently quoted as saying that Christmas is the largest viewing day on their platform.
Using the simultaneous streams methodology, I estimate viewership will be in the 40-50 million range for the NFL stream on Christmas. The wild card in the guess is that Netflix has never given global streaming rights to any platform for an NFL game. In 2023, the Las Vegas Raiders and Kansas City Chiefs drew an average TV audience of more than 29 million viewers on CBS, making it the most-watched Christmas Day game in 34 years. If you compare that number to TV broadcasts from previous years, note that out-of-home viewing was not included in Nielsen final nationals until 2020. Between global rights, large viewership on Christmas Day, and Beyoncé as the halftime show, there are elements to the stream for which no previous data exists to make a comparison.
After the boxing event, numerous posts on LinkedIn disseminated false information, fabricated numbers, and opinions masquerading as facts, using terms without clear definitions. Before you share any of these posts, please stop and look at the source of the info, any data or numbers presented, and the words used. These posts and their sharing don’t help the industry; they set us back. When discussing the challenges of delivering video at scale with excellent QoE, conversations should be based on facts, not guesses or pitches for P2P products from vendors. Netflix publishes a significant amount of technical information about Open Connect publicly and presents at multiple shows that detail, to varying degrees, what they have built.
My post does not capture all of the technical elements and challenges Netflix had during its boxing event. Still, several pieces of information were shared on public platforms and in private conversations, offering a limited insight into what took place. Some additional details I have can’t be disclosed but have been used on background. For Netflix’s games on Christmas, I have assembled a team of industry professionals from across the infrastructure trenches who will have access to real-time stream data. They will contribute information to me during the event and remain unnamed as I blog about it. If Netflix pulls off the NFL games with minimal hiccups, which is my hope, I’ll have nothing to blog about, which would be good. Their success in streaming the NFL games would be a success for the entire streaming industry, and we should all be rooting for them. In this case, the progress of one would be the progress of all.