Since Netflix’s live boxing event, I’ve spoken to more than half a dozen ISPs globally, as well as others who had direct insight into the traffic during the event. Based on those conversations, info from manifest files, error codes, and other sources I will not disclose, here’s part of what happened during the event and what to watch for during Netflix’s NFL games on Christmas.

Post-event, some quickly suggested that ISPs ran out of “capacity,” without even defining what capacity they were referring to. For most ISPs, almost all of Netflix’s traffic was carried on private network interconnects and appliances. Of the seven ISPs I spoke with, located in the US, Europe, and another location, none experienced any capacity issues with peering, transit, or private interconnects. In conversations with ISPs and what some ISPs posted online, they said:

• “We have plenty of capacity with them in the form of OCAs and Private interconnects.”
• “We had plenty of OCA bandwidth capacity and overhead available.”
• “None of our paths that feed our OCAs were at capacity.”
• “We had a healthy overhead of Netflix interconnect/OCA.”
• “None of our connections with Netflix were saturated.”

Multiple ISPs reported that they had issues with the fight and regular programming streaming from an OCA closest to them, but only the fight had problems, which is very telling. That’s why Netflix’s SVOD content and other non-Netflix streaming services looked fine. Netflix deploys caches differently based on the ISP and region of the world. Some ISPs have Netflix caches in some of their larger telco central offices and cable headends. In contrast, others have direct private peering connections with them at core sites, in addition to other connection options.

Netflix did not use any third-party CDNs for the boxing event and is not currently adding any third-party CDNs into the mix for the NFL games. Anyone who said that Netflix used them for the boxing event or wondered if they did should stop guessing. It’s public information, as you can see from traceroutes and by speaking with those involved in the event. Netflix is utilizing the same infrastructure to deliver live streams and VOD content; some statements on LinkedIn that Netflix is using a “separate” infrastructure for live streams are not accurate. Netflix spins up and down resources needed for their Open Connect servers and doesn’t have any hardware “limitations” across Open Connect for live, like some want to suggest.

Looking at data from manifest files and other sources, it was interesting to observe that Netflix was using 1-second segments, which was surprising. One of the key technical issues I observed was that manifest files were not refreshing at times, resulting in connection errors. I sorted through more than 1,000 posts on Twitter from users who provided screenshots of errors, and the three errors below were the most commonly reported, based on my review.

• Netflix Error TVQ-PB-101 (5.3): This error happens when an issue with data stored on the user’s device stops Netflix from playing. It requires a device restart.
• Netflix Error TVQ-PB-101 (3.2.400): This error occurs when a network issue prevents the device from accessing Netflix. Users who encounter the error on any device other than a Roku or smart TV are directed to contact Netflix on their website to help “investigate” the issue. However, Netflix’s support site provides no suggestions for resolving the problem.
• Netflix Error: UI-113 and NSEZ-403: This error occurs when a network issue prevents the user’s device from reaching Netflix and connecting to their account, with Netflix suggesting to “try again later.”

Some LinkedIn posts suggested that Netflix should have prepared better for the event by “simulating” more traffic in advance, which is a ridiculous statement. Those making such comments demonstrate a lack of understanding of how live events work. No company can simulate 65 million simultaneous streams across any network in a real-world test. As Netflix said post-event, “It’s impossible for our engineering team to test that magnitude of traffic and viewership unless they have a real, live thing, which is what happened.” It’s also important to note that Netflix had no previous streaming telemetry or viewing data specific to live content in any significant quantity, so it couldn’t determine from a regional standpoint where the most viewership would come from. They were using VOD viewership data to try and figure out live traffic, and it’s easy to warm up caches for VOD, something you can’t do with a live stream.

To optimize stream performance during a live event, you need sufficient real-time network telemetry and observability data to inform your tuning decisions. The boxing event was Netflix’s first large-scale live streaming event on its platform. I don’t know how many logs Netflix was processing per second, but for a recent live event with an AMA of just over 10 million, the content owner told me they were processing 15 million log records per second. With Netflix having so much data from the boxing event to put towards tuning QoE for the NFL games, I would expect the user experience for the NFL stream on Christmas to be vastly improved over the boxing event.

While no one knows how many simultaneous viewers Netflix will get for the NFL games on Christmas, I expect it to be less than the boxing event. The boxing stream attracted viewers from various regions worldwide, as the event was more of an entertainment spectacle than a professional boxing match. The NFL games will see increased traffic from the US, and Netflix knows which US regions will have the most viewership for the game based on the team’s local markets. That said, Beyonce performing at halftime is an entertainment event, not a sports one, so I would expect Netflix to see a surge of non-football viewers if they can time the halftime show accurately.

A not-often-mentioned but critical point is that Netflix’s boxing event was initially scheduled for July 20th. If the event had occurred on that date and not been postponed to November 15th due to Tyson’s injury, Netflix would have had 5 months between the boxing event and the NFL games on Christmas to help tune its network. However, due to the schedule change, Netflix’s 5-month window to prepare for Christmas dropped to 45 days. Those who produce live events for a living understand the level of what Netflix is looking to achieve at a scale not seen before, with a timeline that is unheard of. Adding to the complexity is that a Netflix executive was recently quoted as saying that Christmas is the largest viewing day on their platform.

Using the simultaneous streams methodology, I estimate viewership will be in the 40-50 million range for the NFL stream on Christmas. The wild card in the guess is that Netflix has never given global streaming rights to any platform for an NFL game. In 2023, the Las Vegas Raiders and Kansas City Chiefs drew an average TV audience of more than 29 million viewers on CBS, making it the most-watched Christmas Day game in 34 years. If you compare that number to TV broadcasts from previous years, note that out-of-home viewing was not included in Nielsen final nationals until 2020. Between global rights, large viewership on Christmas Day, and Beyoncé as the halftime show, there are elements to the stream for which no previous data exists to make a comparison.

After the boxing event, numerous posts on LinkedIn disseminated false information, fabricated numbers, and opinions masquerading as facts, using terms without clear definitions. Before you share any of these posts, please stop and look at the source of the info, any data or numbers presented, and the words used. These posts and their sharing don’t help the industry; they set us back. When discussing the challenges of delivering video at scale with excellent QoE, conversations should be based on facts, not guesses or pitches for P2P products from vendors. Netflix publishes a significant amount of technical information about Open Connect publicly and presents at multiple shows that detail, to varying degrees, what they have built.

My post does not capture all of the technical elements and challenges Netflix had during its boxing event. Still, several pieces of information were shared on public platforms and in private conversations, offering a limited insight into what took place. Some additional details I have can’t be disclosed but have been used on background. For Netflix’s games on Christmas, I have assembled a team of industry professionals from across the infrastructure trenches who will have access to real-time stream data. They will contribute information to me during the event and remain unnamed as I blog about it. If Netflix pulls off the NFL games with minimal hiccups, which is my hope, I’ll have nothing to blog about, which would be good. Their success in streaming the NFL games would be a success for the entire streaming industry, and we should all be rooting for them. In this case, the progress of one would be the progress of all.

Here’s a link to the discussion on this topic on LinkedIn.

The economics of the CDN business have never changed. Vendors spend money to put capacity in place, hoping that over time, they get more bits to deliver and the economics of scale kicks in. It’s a gamble with a high risk of failure. Many vendors bet on 4K/8K/Metaverse/AR/VR “insert non-sense consumer app here,” but they didn’t get the additional volume they were betting on. As a result, they could not grow their business at the level they needed to to get to a scale where they could lower their costs enough to make the business work.

Deployment architecture is also a significant factor in determining success. CDNs must be able to deliver content with the QoS that customers demand while keeping their costs down. Efficiency is EVERYTHING. Akamai realized this early on and started working with ISPs to deliver customer content from caches within ISPs or connected to ISPs via peering/transit/interconnects. For some reason, Limelight thought it best not to work with ISPs to deploy caches in large quantities, with less than 10% of their traffic ever being served via ISP relationships. Instead, Limelight focused on delivering bits from “dense Super PoPs” that were far more expensive to build and maintain. Edgio’s direct cost on the Limelight network was around $0.0005 per GB delivered, while Akamai’s cost is less than half that.

When you sell a delivery service for less than it costs you to deliver it, you only survive based on the money you can raise to keep the business going. When interest rates increase on borrowed money, vendors can no longer afford the interest payments, and they go out of business. It’s simple economics. What happened to Edgio is the same thing that happened to iBEAM, Panther Express and other CDNs from twenty years ago. The times have changed, but the business economics haven’t. See www.cdnlist.com

The biggest problem with the CDN business has ALWAYS been vendors and industry people suggesting the total addressable market (TAM) for CDN delivery services is larger than it is. When executives at any company don’t live in reality and don’t understand the market, they will go out of business! Limelight’s filings disclosed that less than 20 customers comprised 70+% of its business. Over many years, we saw how Akamai’s delivery business was directly impacted by its “top six internet platform accounts.” As an industry, we have the data to see the “true” size of the CDN market and the expected growth level. The fact that many want to ignore the numbers and make up their own is what sets false expectations, leading to failure. A handful of customers comprise the largest percentage of overall revenue for all CDN vendors. A shift in strategy by any of them and vendors are impacted, more often than not, negatively.

Regarding capacity, there is not a shortage. Vendors are willing to pay to build out more capacity if customers will pay a fair price, which they are willing to do. But it must be built out realistically. Large customers still have Akamai, Amazon, Fastly, and CDN77 as options, and within specific geographic regions, other CDN vendors are strong in those locations. Some of the largest companies have also supplemented their delivery with DIY and first-party CDN capabilities, with a few other large content owners currently looking at that as an option. 4K quality video is not growing much as a percentage of total bits delivered by CDNs, and content owners optimizing their encoding has, in many cases, lowered their total bit delivery year-over-year.

The economics of the CDN business have not changed, and every vendor has been trying to replicate what Akamai has done. Akamai offers a delivery service at a scale that they routinely disclose is profitable. They achieved this by diversifying their business from only offering delivery services to other non-CDN services, like security, with higher margins. This allows them to generate cash to re-invest in the business and build more scale while lowering costs. Amazon has done the same thing with its CloudFront delivery business, to the tune of more than $1.1 billion in revenue this year (I’m not disclosing the source of the number), within a larger AWS business that will do over $110 billion in revenue in 2024. The economics of the CDN business have never changed over the past thirty years, only the vendors’ execution in the industry.

In regulatory filings (two 8-Ks and one 10-Q), more details have emerged in Bending Spoons plans to acquire Brightcove, including some terms for employees who stay on with the new company, the number of aggregate shares of company common stock tied to RSU awards, the exact number of Brightcove employees, a disclosure of a “material weakness in internal control over financial reporting,” and other company details. I have highlighted some of the numbers below, and Mark Donnigan and I recorded a special podcast discussing the proposed Brightcove acquisition, which you can listen to here.

• As of Sept 30, 2024, Brightcove had 621 employees, not 720 like multiple people keep reporting
• If the deal is not completed by August 24, 2025, Brightcove would have to pay $7.8M in a termination charge
• Brightcove’s CEO, CFO, CLO all have been given a $200K bonus to stay on with the Company or one of its subsidiaries through the Closing
• Employees who stay with the new entity cannot have their salary or base hourly rate “reduced by more than 10%” for 12 months.
• Brightcove disclosed a “material weakness in internal control over financial reporting.” Fortunately, Brightcove says the material weakness “did not result in any material misstatements in our financial statements or disclosures, and there were no changes to previously released financial results.”
• Brightcove has a minimum commitment of $6.6 million over two years for content delivery network services, hosting and other support services.
• On March 20, 2023, Brightcove granted 1,563,688 premium-priced options to some of its employees under its 2021 Stock Incentive Plan. The options have a strike price of $7.00 and vest in equal installments over three years following March 10, 2023.
• An aggregate of 1,904,276 shares of Brightcove’s common stock were subject to outstanding company stock options, and an aggregate of 7,579,892 shares of company common stock were subject to outstanding company RSU awards.
• While not in the filing, I find a record of 53 patents granted to Brightcove. However, in Q1, the company sold an unknown number of its patents to Adeia for $6.0 million. I don’t know how many patents Brightcove currently owns.
• At the end of Q3, Brightcove had 1,923 premium customers, which was slightly down YoY. Outside of premium customers, Brightcove had 469 “starter” customers, paying $350 monthly, which was slightly down in Q3 YoY.

Further details on the deal can be found in this blog post.

Akamai has provided more details tied to their $125 million bid for customer contracts from Edgio’s security and content delivery businesses as part of Edgio’s bankruptcy auction. These contracts will give “several hundred net new customers” the necessary support to migrate smoothly to Akamai. For the fourth quarter of 2024, Akamai expects this transaction to add approximately $9-$11 million in revenue. For the full year 2025, Akamai anticipates this transaction will add approximately $80-$100 million in revenue and have approximately $25-$30 million of transition service costs.

Akamai agreed to pay certain costs for Edgio to operate its network during the transition and wind-down period, estimated to be $15-$17 million in Q4, until Edgio ceases its content delivery network in mid-January 2025. Akamai Technologies has acquired select customer contracts from Edgio’s businesses in content delivery, security, and non-exclusive license rights to patents in Edgio’s portfolio. The transaction does not include acquiring Edgio personnel, technology, or assets related to the Edgio network.

Overall, Akamai’s costs will be $150-$160 million in the deal, and they expect to generate between $89-$111 million in revenue over the next five quarters. Akamai’s revenue estimates do not include the revenue potential for Akamai selling these new customers cybersecurity and cloud computing services or additional revenue generated from the potential growth of the delivery contracts.

For those who will make statements saying the deal was good or bad for Akamai, it can’t be determined unless you have details of Akamai’s conversations with customers, especially the larger ones. Edgio had some very large delivery contracts with Amazon Prime Video, Sony, Samsung, and Microsoft, and it’s unknown what customers might have committed to Akamai in their conversations. This is the third deal for Akamai, where they acquired CDN contracts from a competitor, and the company has excellent insight into the value of those contracts for their business.

Bending Spoons, a Milan-based mobile app development company, announced plans to acquire Brightcove in an all-cash transaction valued at approximately $233 million. Brightcove shareholders will receive $4.45 per share in cash for each share of Brightcove common stock, with the deal expected to close in the first half of 2025. This deal is a natural progression for the company, and over the past 18 months, I spoke to multiple PE companies that were looking to take Brightcove private. It was only a matter of time before this happened.

While Brightcove has struggled to grow revenue over the past four years, it has a clean balance sheet and has projected a non-GAAP loss from operations of $0.1M — $1.1M for 2024. The company ended Q3 with $27M in cash and cash equivalents. Brightcove’s 2024 revenue guidance is $197.7M – $198.7M, so Bending Spoons is valuing the company at about 1.2x projected 2024 revenue. Brightcove’s 2023 revenue was $201.1M, 2022 revenue was $211.1M, and 2021 revenue was $211.09M.

If the acquisition goes through, this won’t be Bending Spoons’s first company in its portfolio tied to video. Earlier in the year, they acquired StreamYard (also known as Hopin) and previously acquired Evernote, WeTransfer, Issuu, Filmic, and Meetup. Earlier this year, Bending Spoons raised $155 million in its latest equity financing round, giving them a post-money valuation of $2.55 billion. It raised $340M in 2022, and earlier this year, Bending Spoons CEO and Co-Founder said the company had raised over $500M to date. In 2022, Bending Spoons announced it surpassed $100 million in annual revenue, and during a 2024 media interview, the CEO disclosed the company’s revenues were $392M in 2023 and expected to surpass $500M in revenue for 2024.

While it’s unknown what Bending Spoons’ plan is for Brightcove, the company has a history of mass layoffs and shutting down US operations for companies it acquires. After acquiring Evernote, Bending Spoons closed its operations in the US and Chile, laying off almost all the staff in those areas, and moved the business to Europe. After acquiring WeTransfer, bending Spoons laid off 75% of its employees. After Bending Spoon’s acquisition of Filmic, they laid off the entire staff, and after acquiring Meetup, they moved operations to Europe, resulting in mass layoffs.

Much has been written about how Bending Spoons regularly cuts the employee headcount of companies it acquires to operate them profitably and have a clear, tight product focus. While that’s not good for the employees of the companies it acquires, it’s simple business economics. While I have no insight into how Bending Spoons might staff Brightcove after the acquisition, I expect them to do a large round of layoffs as there will be much job overlap from the previous companies they have acquired. In a 2024 interview, the CEO stated that Bending Spoons had 300-400 employees. At the end of 2023, Brightcove had 671 employees.

New data shows that 4K bit delivery growth is still flat across CDNs. In 2022, I detailed how many streaming services were optimizing their encoding and, in some cases, reducing the highest rung in their bitrate ladder to save money. CDNs I spoke to then were very open about seeing a drastic reduction in the volume of video bits delivered due to bitrate optimization.

According to new data this month from CDN77, 4K video playback is still not significantly driving traffic growth across the CDN industry. Across all their customers, while 4K titles make up less than 15% of available content and approximately 10% of initial requests, they account for less than 5% of overall playout time on their network. Other CDN vendors I have spoken to recently reinforce what CDN77 is seeing, with some CDNs telling me that 4K bit delivery still makes up 5% or less of the total bits they deliver each month, flat from last year.

CDN77 recently analyzed its logs and spoke to several customers to get insight into the latest trends regarding 4K content libraries and playback. They found that 4K video takes up 12-15% of their clients’ content libraries, meaning up to 15% of titles are available in 4K. 8-10% of all initial requests are made for 4K, but interestingly, 4K represents less than 4% of the overall playout time.

This discrepancy is partially caused by the fact that while users (or players) frequently request 4K content, it often switches to a lower profile during playout, reducing overall 4K playout time. Interestingly, some clients admitted adjusting their ABR algorithms not to push 4K as a default profile even if end users’ connectivity allows. Instead, they force lower quality (720p/1080p) as a default, leaving 4k as an option for the client to choose manually.

Also, some streaming services, like Max, charge more monthly to get 4K streaming as an option. Their With Ads and Ad-Free Plan offer up to 1080p, but to get 4K, you must sign up for their Ultimate Ad-Free plan at $20.99 monthly. The cost for the 4K plan is more than double their base plan at $9.99 per month. And while no third-party CDN delivers video for Netflix, they charge $22.99 monthly for a plan with 4K streaming, more than 3x the cost of their Standard with ads plan at $6.99 monthly. Today, fewer subscribers are taking plans that give them access to 4K content, and most live events don’t even offer 4K video quality as an option. Amazon’s TNF games, YouTube’s NFL Sunday Ticket package, MLS Season Pass and Friday Night Baseball on Apple TV+ max out at 1080p. For all the talk in the industry about how sports events must be in 4K, content owners are very open that offering 4K as an option does not drive more viewership or revenue to the streaming service.

What we saw just after the pandemic with content owners looking to cut costs in their video workflow has not changed. There is still pressure to cut costs at the lower end of the bitrate spectrum, as content providers tend to lower bitrates for smaller qualities, ultimately offsetting total bandwidth volumes. Some within the streaming industry keep talking about the “growth” and “need” for 4K video quality, but as the data shows, those statements are not grounded in facts, and the data from CDN providers says otherwise.

Related Posts:
Rate of Video Traffic Growth Declining Across CDNs and ISPs As OTT Services Optimize Encoding Bitrates, See Little Demand for 4K Quality

With Streaming Services Cutting Bitrates to Save Money, Vendors That Bet on 4K for Revenue Growth Have Lost

Facts matter, except apparently in the aftermath of Netflix’s boxing event. This was Netflix’s 8th 9th live event (open to correction), not their first. Everyone is an expert in CDN and can “fix” Netflix’s live streaming, even though they don’t know the problems. Multiple vendors claim their cloud/edge/P2P/”insert anything here” solution could have solved the problem when some vendors are in beta with zero deployments. Many people share and compare numbers that are completely made up but state them as fact.

Having a fruitful conversation on the topic is hard since many refuse to take the time to educate themselves on the real numbers of past events. The volume of inaccurate posts I’ve seen on LinkedIn wastes people’s time. Stop posting made-up numbers. Here are a few points I keep seeing that need to be corrected:

• FALSE: “It was Netflix’s first live event, so it will take them time to get it right.” This was Netflix’s 8th 9th live streaming event. (Love is Blind reunion, Netflix Cup, Chris Rock Special, 30th annual Screen Actors Guild Awards, UDUM 2023, The Netflix Slam, Tom Brady Roast, Joe Rogan Special)
• FALSE: “The NFL games on Christmas will need capacity for 100+/110M concurrent streams”. The Las Vegas Raiders and Kansas City Chiefs drew an average audience of more than 29 million viewers on CBS — which made it the most-watched Christmas Day game in 34 years. Even with Beyonce as the halftime show, Netflix is not getting 110M concurrent streams.
• UNKNOWN: “Only 0.1%/0.15%/1%/5% of the audience had a bad stream.” No one knows the percentage of overall users that had bad QoS, and people are pushing out numbers with ZERO sources, public or private. Using reports from downdetector is NOT a valid source!!!
• FALSE: “Netflix spent ~$50M on the event.” Based on what the fighters said, the purse alone was more than $80M combined. Add marketing and production costs, and Netflix spent well over $100M to produce the event. I don’t know if they shared in the revenue from the venue ticket sales, and I have found no source of Netflix commenting on the topic. I also don’t know what Netflix made on sponsorships.
• FALSE: “Akamai had major problems delivering the videos.” Netflix used its Open Connect infrastructure exclusively, and no third-party CDN was used. Some said, “If Netflix relied exclusively on its Open Connect CDN…” Why is anyone guessing? Do some traceroutes and talk to the vendors. All third-party CDNs were very open when asked and said they had no involvement in delivering the stream.
• FALSE: “Netflix supplemented its own CDN with vendor CDNs for all their overflow traffic.” See above.
• FALSE: “Netflix failed to test its capacity load properly.” You can’t simulate 65M concurrent streams in any real-world testing environment. Many want to point to “capacity” with ISPs, which I know, in many cases, was NOT the problem, as the ISPs told me.
• FALSE: “The event can only be compared to the Super Bowl.” Super Bowl streaming viewership peaked at 8.5M last year; on TV, it peaked at 126.3M in 2017, and none of those 126M viewers had QoS issues.
• FALSE: “Netflix has 8,000+ servers.” According to the last number they released, Netflix has close to 20,000 servers deployed across 175+ countries. In 2023, a half-rack of eight servers could deliver 200,000 concurrent streams. (Netflix did not say if that was VOD or live streams)
• UNKNOWN: “Estimates put sign-ups from this event at 10% / Netflix signed up millions of new users.” Based on what source? 10% of what number?

For those making up numbers and publishing them without sources, and you have the title of “Research Director” or “Analyst,” shame on you. Playing fast and loose with numbers is not acceptable. All my numbers in the post are sourced from public blog posts and/or press releases from Netflix, CBS, and Paramount.