What’s amazing to me is how much data they’re pumping out — their bandwidth bills must be insane. Or they did some pretty amazing negotiating contracts considering Netflix is like what, $10-15/mo? And there are many who “binge” many shows likely consuming gigabytes per day (in addition to all the other costs, not least of which is actually making the programming)?
Extortion fees? Netflix wanted Comcast to provide them with free bandwidth. While some ISPs may appreciate that arrangement as it offloads the traffic from their transit links, Comcast is under no obligation to do so.
You could argue then that Comcast wasn't upgrading their saturated transit links, which they weren't with Level 3, but to assume that every single ISP should provide free pipes to companies is absurd.
So no business should ever have to pay for bandwidth because customers of said business are already paying? Or should I get free internet because businesses are already paying ISPs for their network access?
That sounds reasonable? If I'm an end-user, what I'm paying my ISP for is to get me data from Netflix (or youtube or wikipedia or ...); that is the whole point of having an ISP. If that means they need to run extra links to Netflix, then tough; that's what I'm paying them for.
You're paying them for access to the ISPs network. For an internet connection, this also comes with access to other networks through your ISPs peering and transit connections.
If you know of any ISP that would give me 100+ Gbps connectivity for my business, please let me know as I'd love to eliminate my bandwidth costs.
Businesses have to pay for bandwidth to get data to the customer's ISP, but they generally don't pay the customer's ISP for the same bandwidth the customer has already paid for.
Netflix did not have to pay Comcast either. One of their ISPs Level 3 already had peering arrangements with Comcast to deliver the content. Instead of paying their own ISP, Netflix wanted to get free bandwidth from Comcast. There's a difference.
Comcast threatened to throttle customers' bandwidth, refusing to deliver the speeds they had promised. The data was available to Comcast, customers had paid for the service of delivering that data, but Comcast wouldn't provide the full service they had sold unless Netflix paid them more.
The deeper issue is that Comcast is lying to their customers, promising them more bandwidth than they are able to deliver, so when Comcast's customers wanted to use all the bandwidth they bought to watch Netflix, Comcast couldn't afford to honor their promises.
But Comcast has a monopoly in the markets they serve, while Netflix exists in a competitive market, so Comcast got away with it.
No ISP can guarantee speeds outside of their network. Once it leaves their network it's considered best effort. Comcast has about 30 million subscribers. If they were to guarantee bandwidth out of their network, and if every subscriber had a laughably slow 10Mbps connection, Comcast would need 300Tbps of connectivity to every single company and network. For this reason, every ISP in the world "throttles" in one way or another.
They don't, and I never suggested they did. Netflix had their own ISPs. Netflix wanted the Customer's ISP to give them free bandwidth so they could offload the traffic from their other ISPs.
One of the funny truths about the business of networks, is that the one who "controls" the most bandwidth has the #1 negotiating spot at the table.
Take for example, if say Verizon decided to charge more for bandwidth to Netflix... if Netflix said "no" and went with another provider, then Verizon's customer's would suffer from worse access times to Netflix.
Verizon has the advantage in that they have a huge customer base that no one wants to piss off Verizon. So it cuts both ways. Bandwidth becomes not a cost at this scale, but instead a moat.
My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world.
ISPs would have been more than happy to show the middle finger to the WFH engineers, but not to the binge-watching masses.
> My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world.
Couldn't agree more.
We see the opposite when it comes to broadband monopolies: "barely good enough" DSL infrastructure, congested HFC, and adversarial relationships w.r.t. subscriber privacy and experience.
When it became worthwhile to invest in not just peering but also last-mile because poor Netflix/YouTube/Disney+/etc performance was a reason for users to churn away, they invested.
This isn't to say that this is all "perfect" for consumers either, but this tension has only been good for consumers vs. what we had in the 90's and early-mid 00's.
The US is still not ready. The vast majority of people have access to only over subscribed coaxial cable internet, with non existent upload allocation.
Anecdotally, I'm always very impressed with the connectivity I see in the US versus my home connection in the UK. I'm fairly certain that the last mile from my suburban home to the cabinet is made of corroded aluminium.
UK is certainly worse. Even in a well populated suburb 10min from Birmingham, I know a family that can only get ADSL or some other service like that at 2Mbps.
I have been spoiled with symmetric 1Gbps up and down fiber for a few years, and using the internet is like turning on the electric or gas or water, you do not ever have to think about it.
I live in central London (zone 2) and the fastest wired broadband service available at my house is slower than 20Mbps. (I'm playing with LTE now, but so far it's been a bit of a mixed bag.)
The binge-watching masses are easy to satisfy. All it takes for the stream to work is average speed of a few megabits per second, but there's so much caching at client end that high latency and few seconds of total blackout every now and then don't really matter.
Not sure if you realize which ISP you picked, but Verizon and Netflix actually had peering disputes in 2014 which gave me quite the headache at my then-current employer.
> I heard they are paying something between $0.20 (eu/us) to $10 (exotic) per TB based on the region of the world where the traffic is coming from
They're likely paying even less. $0.20/TB ($0.0002/GB) is aggressive but at their scale, connectivity and per-machine throughput, it's lower still.
A few points to take home:
- They [Netflix, YT, etc] model cost by Mbps - that is, the cost to deliver traffic at a given peak. You have to provision for peaks, or take a reliability/quality of experience hit, and for on-demand video your peaks at usually 2x your average.
- This can effectively be "converted" into a $/TB rate but that's an abstraction, and not a productive way to model. Serving (e.g.) 1000PB (1EB) into a geography at a peak of 3Tbps per day is much cheaper than serving it at a peak of 15Tbps.
- Netflix, more so than most others, benefits from having a "fixed" corpus at any given moment. Their library is huge, but (unlike YouTube) users aren't uploading content, they aren't doing live streaming or sports events, etc - and thus can intelligently place content to reduce the need to cache fill their appliances. Cheaper to cache fill if you can trickle most of it during the troughs as you don't need a big a backbone, peering links, etc. to do so.
- This means that Netflix (rightfully!) puts a lot of effort into per-machine throughput, because they want to get as much user-facing throughput as possible from the given (space, power, cost) of a single box. That density is also attractive to ISPs, as it means that every "1RU of space" they give Netflix has a better ROI in terms of network cost reductions vs. others, esp. when combined with the fact that "Netflix works great" is an attractive selling point for users.
Sorry for the naive question, but to offer those prices, there are two options: A) Amazon is losing money to keep Netflix as their client, B) They are doing profit even with $0.20/TB, which means the $120/TB is, at least, 119.80 profitable. Wow.
Netflix has appliances installed at exchange points that caches most of Netflix. Each appliance peers locally and serves the streams locally.
The inbound data stream to fill the cache of the appliance is rate limited and time limited - see https://openconnect.zendesk.com/hc/en-us/articles/3600356180... The actual inbound data to the appliance will be higher than the fill because not everything is cached.
The outbound stream from the appliance serves consumers. In New Zealand for example, Netflix has 40Gbps of connectivity to a peering exchange in Auckland. https://www.peeringdb.com/ix/97
So although total Netflix bandwidth to consumers is massive, it has little in common with the bandwidth you pay for at Amazon.
$0.20/ TB costs is not for their agreements with Amazon, bulk of their video traffic is directly peering with ISPs, AWS largely serves for their APIs, and orchestration infrastructure.
Amazon and most Cloud providers do overcharge for b/w. You can buy a OVH/Hetzner type box with a guaranteed un-metered 1 Gbps public bandwidth for ~ $120/month easily, which if fully utilized is equivalent 325TB / month or $3-4/TB, completely ignoring the 8/16 core bare metal server and attached storage you also get. This is SMB/ self-service prices, you can get much deals with basic negotiating and getting into a contract with a DC.
One thing to remember though not all bandwidth are equal, CSPs like AWS provide a lot of features such as very elastic scale up on-demand, a lot of protection up to L4 and advanced SDN under the hood to make sure your VMs can leverage the b/w, that is computationally expensive and costly.
AWS {in,e}gress pricing strategy is not motivated by their cost of provisioning that service. Cloudflare had a good (if self-motivated) analysis of their cost structure discussed on here a while ago
Hence, to a large extent, the devices described in the presentation. They’re (co)located with ISP hardware, so the bulk data can be transferred directly to the user at minimal / zero marginal cost
