Archives

Comcast to Deploy Qwilt’s Open Caching Platform at Hundreds of Locations

Qwilt and Comcast announced a deal that will see Comcast deploy Qwilt’s open caching platform inside its network at hundreds of locations over the rollout, with the first locations going live in the next 1-2 months. The companies are not disclosing the capacity Comcast will deploy within their network or the percentage of total video traffic that Qwilt’s Open Edge platform will deliver across Comcast’s network. However, with the addition of Comcast to Qwilt’s federated CDN strategy, Qwilt says their total on-net coverage of US broadband households will be 55% at the end of this year. While Qwilt has full US regional coverage like other commercial CDNs, its on-net coverage, with many embedded edge delivery nodes, distinguishes its operation and business model.

While Qwilt started as a point solution selling to ISPs, the company has transitioned to becoming its own federated CDN, selling delivery across multiple ISPs directly to content owners. It is starting to compete more with traditional CDN providers like Akamai, Fastly, and Edgio, using its federated CDN model to get a percentage of traffic from customers who use a multi-CDN approach. I have previously highlighted how I have seen Qwilt in the mix for a small percentage of live event traffic from Prime Video, Peacock and other large content distributors. With the Comcast deal giving Qwilt a much larger footprint, the company hopes to get a larger share of customer traffic going forward.

In the federated CDN model, Qwilt shares revenue with ISPs, as they will do in their deal with Comcast. A big advantage that Qwilt has over traditional CDNs is that the ISP provides the hardware—allowing Qwilt to add capacity at a fraction of the cost of competitors since Qwilt does not have to spend capex on edge infrastructure since the service provider owns and operates that part of the network. Comcast claims that by deploying Qwilt within their network, Comcast will create “the most distributed content delivery network (CDN) in the US.” While their wording is too generic, Comcast means most distributed within an ISP environment. Comcast will not have the capacity that Akamai or select other CDNs have in the US. Being widely distributed is crucial to offering a CDN service, but so is capacity, which must go hand-in-hand.

Sponsored by

What Many Are Getting Wrong in the Meta and Deutsche Telekom Peering Dispute

Meta and Deutsche Telekom are having a peering dispute, with the companies ending their interconnect deal. While the two sides are blaming each other, Meta is under no legal obligation to pay Deutsche Telekom for access to their network. Deutsche Telekom is under no obligation to give Meta peering for free. As Deutsche Telekom correctly points out, “Meta alone decides how Meta traffic is routed to Deutsche Telekom’s network.” Meta has switched to using transit providers instead of a direct interconnect to connect to Deutsche Telekom’s network.

In its blog post, Meta says, “given the court ruling concerning the unprecedented and unacceptable fees demanded, we are now routing our network traffic through a third-party transit provider.” Meta doesn’t say how Deutsche Telekom’s fees compare to Meta now paying transit providers to connect to Deutsche Telekom’s network. Deutsche Telekom disclosed that Meta used to pay for access to their network but stopped paying in March of 2021. Why did Meta stop paying? Was there a capacity issue? SLA issue? It’s rare for a platform provider to pay for a direct interconnect to a network and then just stop paying. Is Meta comparing Deutsche Telekom’s fees to what they had for free? Any fee seems “unacceptable” compared to what a company got for three years at no cost.

Meta says Deutsche Telekom “is using its market power to put its subscribers in Germany behind a de facto paywall, potentially restricting their access to internet services that do not pay Deutsche Telekom.” This is not accurate. Deutsche Telekom has not restricted or blocked any of Meta’s apps or services, and Meta knows this, which is why they used the word “potentially.” There is no business reason for Deutsche Telekom to restrict any of Meta’s services, as that would only harm Deutsche Telekom’s customers, who would complain to Deutsche Telekom, not Meta.

I see some in the media saying instead of Meta, “continuing to pay extortionary network fees for a direct connection,” which isn’t accurate. Meta hasn’t been paying anything for the past 3+ years. And what are the fees exactly? Who’s to determine what is considered “extortionary.” I also see some suggestions that Instagram, Facebook, and WhatsApp will “slow to a crawl,” which again is inaccurate. As Deutsche Telekom stated, “the rerouting of data traffic in the night from Tuesday to Wednesday went smoothly.” We see no reports of any Meta apps or platforms having QoS issues, and the media needs to stop suggesting that there are user issues when they do not exist.

While this all boils down to a business dispute between two companies, it’s important to point out that a German court ruled in this contractual dispute between Deutsche Telekom and Meta based on the principles of German contract law. Meta may not like the outcome of the ruling, but it’s the court that decided. Meta is free not to pay Deutsche Telekom and instead move to a transit provider model as they have done.

All of this aside, settlement-free peering and paid interconnects are the basics of how the Internet works so well. I disagree entirely with Deutsche Telekom’s statement that “regulations are needed to ensure the settlement of disputes.” Regulators have no idea how any of this works, and any time they do get involved, it is always behind closed doors. It becomes grandstanding by politicians and lobbying organizations paid by both sides, getting in the middle of the debate. Many step forth with studies and papers implying they are neutral when they get paid by lobbying organizations and have conflicts of interest. The public doesn’t get the details of the contracts or costs, and any shared data is always used to skew the argument without the complete data set being shown.

Regulators won’t fix this, nor should they be allowed to. The biggest problem in this whole discussion is the media. Too many suggest blocking, throttling or slowing down of Meta’s services is taking place when it isn’t. Or speculate, with no proof, that Deutsche Telekom won’t put enough capacity in place with Meta’s transit provider, which again, we know isn’t a problem. Analysts and media members need to stop taking sides and report the facts. All the what-ifs, what could, and what might happen are speculations on their part, and you can’t have a rational discussion and solve business problems focused on theory. Stick to the facts.

Deutsche Telekom Post: Meta is not above the law
https://www.telekom.com/en/company/details/meta-is-not-above-the-law-1079704

Meta post: Why We’re Having to End Our Direct Peering Relationship With Deutsche Telekom
https://about.fb.com/news/2024/09/why-were-having-to-end-our-direct-peering-relationship-with-deutsche-telekom/

Too Big to Be Ignored: On Average, 5-10% of Streaming Subscribers Experience Poor QoE

For OTT services, the largest content streamers now serve around 40-80 million subscribers in the US every month. Outside of Netflix, they rely on third-party CDNs to deliver all or nearly all of their video. Delivering this content requires a high threshold of HD or better video quality, high bitrates for viewing on large screens, immediate player startup, no buffering, and zero dropped packets during the stream. In other words, a consistently high Quality of Experience (QoE) for end users and that’s magnified even further when streaming live events. On average, for 90-95% of their subscriber base in the US, third-party CDNs provide enough data to measure QoE so that streamers are, for the most part, confident their content is being delivered with an optimal experience. But what about the other 5-10% of their audience?

The Blind Spot Downstream
For subscribers further downstream from where third-party CDNs are deployed, it’s challenging to ensure QoE and almost impossible to validate it. This ‘blind spot’ typically exists in more rural networks, or what many large content providers call “the 1% networks”. In this case, it does not refer to the financial elite but rather the QoE-poor or, more accurately, the QoE-unvalidated. Talking to content distributors, it’s not uncommon for them to tell me they lack visibility into what’s happening in these harder-to-reach 1% networks, mainly due to the inherent function of commercial CDNs. QoE could be good, bad, or somewhere in between, but there’s no telemetry to know. Furthermore, 1% is a figure of speech, not a quantifiable reality; the actual figure could be considerably higher—and many believe it probably is.

In particular, the continued viewership growth in live sports streaming can increase traffic by 25-50% or more for some networks, requiring sufficient capacity to ease bandwidth congestion. For commercial CDNs and ISPs, these traffic spikes can be more easily absorbed, as capacity can be spread over a larger area. However, for the smaller ISPs serving 1% of networks, capacity is often constricted unless extra capacity is purchased in advance, called an RSVP fee, and often at a premium, and that’s if capacity is even available.

Then there’s the content delivery blind spot. With live streaming, QoE metrics are often received directly from the player in real-time. Content streamers can easily detect content delivery and streaming performance issues for commercial CDN and ISP networks. For the harder-to-reach 1% networks that are much further downstream, however, tracking content delivery performance is difficult, if not impossible. This means subscribers can easily suffer from buffering, slow startup times, dropped packets, and low bitrates – when they want to stream content the most.

Live streaming from the NHL, NBA, and US Open sporting events has quickly become the most-streamed content across digital platforms, with the NFL dominating TV viewership in the US. When QoE is suboptimal during the big game, the blind spot in a content delivery pipeline suddenly becomes a glaring problem for 1% of network subscribers. This is a common problem for ISPs who receive these complaints from viewers expecting a better experience.

Do the Math
While I’ve called out the 1% of viewers being impacted, in reality, it’s more. One percent of subscribers may sound like a minimal amount, and not much to be concerned about, but content owners will want to think again. If you aim for the best but plan for the worst and then do the math, you’ll realize that 1% is significant. For a content streamer with 40 million subscribers, one percent is 400,000 subs. At $7/month for an average plan, that’s $2.8 million in monthly subscription revenue or $33 million annually.

Moreover, 1% is just an idiom referring to harder-to-reach networks generally. That 1% makes up 5-10% of an OTT platform’s subscriber base when it comes to the blind spot of content delivery and streaming performance. With 5% of network subscribers potentially experiencing poor QoE, a content streamer with 40 million subscribers is now looking at 2 million subscribers and $14 million in MRR—or $168 million annually.

Another angle to consider is customer churn and acquisition costs (which streamers don’t disclose, even to Wall Street). These costs vary widely across organizations; Even without knowing the exact numbers, keeping existing customers is almost always more economical than acquiring new ones and a much easier lift. As for winning back churned customers? That cost just sounds steep. Suffice it to say that if you can’t validate the QoE for your hard-to-reach subscribers, OTT platforms and broadcasters are risking customer churn and lost revenue. They are risking brand reputation, media partnerships, and exclusive streaming rights for the content people want.

QoE in 1% networks has been obscured and underrepresented for far too long, and there’s a lot more at stake than most people think. It’s just that no one seems to be shedding much light on it, let alone doing the math. This gaping hole has only gotten wider for as long as I’ve been in the streaming media industry. Between more people leaving big cities to work remotely or cut living costs and increasing demand for streaming video (especially with live sports), this is a growing problem. Having a blind spot in a content delivery pipeline and potentially delivering suboptimal QoE shouldn’t be acceptable anywhere, especially nowadays, and it deserves more attention.

The last-mile CDN
Part of the problem is that no broadcaster or OTT platform will disclose what percentage of their users had a good experience after live events. Nor will they discuss max bitrates, average startup times, average viewing times, rebuffering or their CDN strategy’s impact on their viewers. As a consultant, I’ve worked on some of these large live events for broadcasters, having access to their monitoring platforms during the events and have seen the problem firsthand. And while NDAs keep me from giving out numbers, a larger percentage of users are impacted than most would think. Talking off-the-record to many of the live event teams at any of the large OTT platforms confirms the scale of the impact.

This market problem was why I profiled Canada-based Netskrt a few months ago after it raised about $7.2 million in its Series A round. The company focuses on delivering video to remote and rural communities for specific video applications with hard-to-reach subscribers to help smaller ISPs manage live event traffic. Effectively, they provide a last-mile CDN for ensuring content delivery and streaming performance for 1% networks. Plus, it’s way more cost-effective than building your own cache, and it’s certainly smarter than ignoring your QoE blind spot. Netskrt says they plan to discuss rural network-related challenges in the coming months and release some QoE-related data to debunk some common misconceptions. That data will be interesting to see, as well as how well the company adds another 15 Tbps to its US footprint and expands it to include Brazil, the UK and Italy. Expect to see me blog more about this topic when Netskrt provides some data.

DAZN’s Private CDN Deployment Is a Good Use Case for Regional Broadcasters

The future of content delivery for broadcasters has changed dramatically over the past few years and will continue to do so, especially for live events. Some have predicted that in the next five years, broadcasters will need to deliver TV-sized prime-time audiences, with streaming media technology being the primary means of distribution.

This means that current peak streaming audiences of about 3-4 million in a country with a population of 65 million will transform into peak audiences of 20-25 million for the same event, using the viewer figures published about the Euros 2024 Football championships as an example. This means that streaming services’ capacity, quality, and security requirements will become dramatically more important for broadcasters in the years ahead.

While I don’t see streaming concurrent viewership of 20-25 million as the norm, at a regional level, we are starting to see some very large concurrent audiences for one-off live events and specific sports matches. DAZN is a good example of a platform that sees large concurrent audiences for Serie A and Spanish LaLiga, where the audiences are large, and the demand for the content is strong.

As part of DAZN’s multi-CDN distribution approach, working with MainStreaming, DAZN built and operates a private CDN-as-a-Service, deployed and managed in close partnership with the largest ISPs in DAZN’s core markets where DAZN Edge is deployed. Not every streaming platform or broadcaster can, or should, build their own CDN; for most, it doesn’t make sense. But in cases like DAZN’s, especially at a regional level, a private CDN-as-a-Service can work well at the core of a multi-CDN strategy.

From a QoE perspective, DAZN says the private CDN has proven its worth, fine-tuning configurations and deployment plans with MainStreaming to ensure the best possible performance to support regional spikes from large audiences tuning in for specific games. The chart above, published in 2023, shows why DAZN’s private CDN at the core of a multi-CDN model provides the base performance it requires.

The internet is constantly in flux as demand on networks fluctuates without notice. Network connections change without warning as ISPs adapt their networks. Yet this ability to change continuously creates different network paths for streaming video to traverse. For example, BGP Routing tables for IPv6 connections (see below) show a general growth trend over time. Still, connection counts have gone down and up quite dramatically, immediately changing how video traverses the internet.

Unlike broadcast, which has standards about QoE, in the streaming industry, there are no agreed-upon definitions of what HD or 4K streaming even means, let alone the term “Broadcast-grade” streaming. I like how MainStreaming defines the term as six key components, and I’m curious to hear in the comments session how others define it:

  • Deliver media with consistently high quality and low latency; and
  • Scale to many millions of viewers; and
  • Achieve predictable delivery costs with economies of scale; and
  • Take real-time QoS actions from real-time QoE analytics; and
  • Protect against Piracy through CDN-embedded controls; and
  • Deliver in Ultra-Low Latency to meet or beat an App’s speed

Broadcast-grade streaming is necessary for broadcasters and streaming platforms to protect their business as they move to a more streaming-first strategy. While this brings new opportunities to engage audiences better and increase revenues, it also brings extra threats to content security, advertising revenue, subscription revenue, and brand reputation. Private CDN platforms will need to adapt to address all those concerns and, in many cases, will need to integrate with other vendors in the video workflow who are specialists in addressing some of those problems.

Edgio Files For Chapter 11 Bankruptcy in Preparation for Sale of Business Units or Entire Company

Edgio announced that it has voluntarily filed for Chapter 11 bankruptcy, and I expect the company to be delisted from the NASDAQ by the end of the day or shortly after. (Updated: Edgio was delisted on Monday September 18) This news does not mean the company is going out of business. Edgio will receive approximately $15.6 million in financing from Lynrock that, following approval by the Court, is expected to ensure that Edgio continues to operate like it is today and throughout the sale process and Chapter 11 Cases.

The purpose of the filing is so that Edgio can sell off part of the business it no longer wants or the entire company, which should allow for the continued operation of the Company’s business under new ownership. It wouldn’t be an asset sale but a business unit sale or the entire company. There is a 40-day bid procedure process, and once all bids are in, the court will decide which is best for the company.

Edgio says they have recently engaged in discussions with several parties that could be interested in acquiring all or part of the Company’s businesses and assets, and by using the Court-supervised sale process, they seek to get the highest or best bid for those assets. As part of this process, Edgio has entered into a “stalking horse” asset purchase agreement with its primary lender Lynrock, which has agreed to acquire assets of the Company through a credit bid for $110 million of the existing secured debt held by Lynrock.

Whatever assets might be sold, Edgio says it is targeting the sale process to be completed in approximately 80 days, if not sooner.

Note: I have never bought, sold, or traded shares in any public CDN, and even in my managed portfolios, Akamai, Fastly, Cloudflare, and Edgio are excluded.

Olympics Averaging 4-5 Million Viewers AMA, Across Peacock and NBCU Digital Platforms

Here’s a list of streaming viewership numbers from NBCU for the Olympics across Peacock and NBCU Digital platforms, averaging 4-5 million viewers. NBCU clarified that these numbers are AMA (Average Minute Audience), even though their releases don’t say that.

  • August 3: coverage was streamed by 𝟰.𝟰 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
  • August 1: coverage was streamed by 𝟰.𝟰 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
  • July 30: coverage was streamed by 𝟱.𝟬 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
  • July 29: coverage was streamed by 𝟰.𝟱 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
  • July 28: coverage was streamed by 𝟲.𝟬 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
  • July 27: coverage was streamed by 𝟰.𝟳 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
  • July 26: coverage was streamed by more than 𝟮.𝟱 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms

The above language is taken directly from NBC Sports press releases.

Paramount Global Has Discussed Selling Pluto TV Back to Co-Founder Tom Ryan

As Paramount Global evaluates its strategy for all of its linear and streaming assets, multiple sources tell me that Pluto TV’s co-founder and current CEO of Paramount Streaming has been in discussions with the company about repurchasing it. These discussions started before the Skydance deal was announced, but I’m told no valuation has been placed on Pluto TV, and no official offer numbers have been exchanged.

Interestingly, during the Paramount Global all-hands meeting on June 25th, employees say the company didn’t mention or include any reference to Pluto TV during the newly announced go-forward plan. Those who sat in smaller internal meetings also say the brand was absent from the discussions. Since the Skydance deal was announced, Skydance has featured Pluto TV as one of the six core pillars of the company on slide ten of its presentation. However, that doesn’t guarantee that Pluto TV, or any other content brand, will be part of the company going forward. During a call with investors after the Skydance deal was announced, Jeff Shell stated, “Current management is also talking about a couple of transactions that, if they get the right price, we’ll be supportive of.” So Paramount Global’s current management team can sell off assets and make deals they think will help the new management team as long as the new management team is consulted.

The financial metrics of Pluto TV and its platform usage have been a mystery since Viacom acquired it for $340 million in cash in January 2019. In 2022, several Pluto TV executives said the company exceeded $1 billion in revenue in 2021 and was profitable. However, I have not encountered any SEC filing that provides detailed P&L metrics for the company or mentions its profitability. If I missed a filing, I would welcome correction. When a co-founder of Pluto TV mentioned on LinkedIn that the company was profitable, I found it interesting and reached out to Paramount Global’s IR department, which said they “Would not confirm” Pluto TV’s financials. Executives I privately asked within the company said they were unwilling to comment on Pluto TV’s financials, even off the record.

Previously, Viacom, and now Paramount Global, have not broken out revenue, detailed profitability, advertising ARPU, or discussed how they define a monthly active user on Pluto TV’s platform, which totaled 80 million when they last reported it about nine months ago. Pluto TV disclosed that in 2023, users streamed seven billion hours of video, up 40% from 2022, but total viewing hours alone isn’t a useful metric. In a LinkedIn post two months ago, Tom Ryan said Pluto TV had “record viewing hours” in Q1, but no numbers were highlighted. 

It’s unclear how Pluto TV defines profitability, considering they get much of their content from Paramount Global at no cost. However, one thing is clear, Paramount’s content is of significant importance to Pluto TV’s business. The potential sale of Pluto TV could profoundly impact its operations. One could make a valid argument that if Pluto TV were sold and had to license Paramount’s content, it would negatively impact its balance sheet and skew the numbers drastically. Pluto TV executives have discussed the importance of Paramount’s content to their business with the media, being quoted as saying, “With the acquisition, the access to the content that we’ve been able to get from Paramount has just been a pivotal moment for us.” Another executive said, “Paramount brings a healthy chunk of premium content to our platform.” Without Paramount’s content, Pluto TV would be a very different service and be less attractive to consumers. 

The potential sale of Pluto TV would be less valuable to a new owner if it did not include exclusive access to Paramount’s content. The Skydance management team said it is looking at all options regarding its content strategy to determine which content it should keep exclusive to its platforms and which it can license to others. It’s possible that Paramount could generate more revenue by not restricting content exclusively to Pluto TV, regardless of who the owner is. Whatever strategy they decide on, the value placed on Pluto TV by any potential new owner will depend on what content they get from Paramount in the deal and whether or not it comes with exclusivity.