This sounds like a pragmatic approach, and pragmatism has always driven internet engineering. From the ISPs perspective, Netflix's needs are diverging from a typical free peering arrangement.
The issue is, what triggers the costs? There's no good definition of what threshold requires a content provider to connect to the ISP network. Because ISPs are consolidating so rapidly, there's a huge opportunity for abuse.
"From the ISPs perspective, Netflix's needs are diverging from a typical free peering arrangement."
I would argue that point. I pay for Internet access, not "All of the Internet MINUS Netflix" access. As a result my ISP SHOULD (doesn't mean they will) ensure that all it's peering connections are of sufficient capacity to provide me with whatever bandwidth I choose to use up until I run into the rate limits for the plan I'm on.
My ISP sells me 25/5 I think so the idea that I can't get 3Mbps worth of Netflix is suspect.
Of course you have to consider that when you're not saturating your last-mile connection it could be a bottleneck anywhere. If it's 15 hops from me to a server if any of those 15 hops is saturated, that's the limiting factor no matter how much capacity there is before or after. That's why closing a single lane on a freeway for as little as 10 feet can really screw up traffic. You have to merge down and deal with reduced capacity even though it's not miles and miles of lane closure.
What we've been told via one of Netflix's backbone providers (Level 3) is that Netflix isn't the slowdown, the backbone provider isn't the slowdown, but the ISPs are the slowdown. How can they say this with certainty? Because they're peered with various ISPs, the connections are saturated, and the ISPs refuse to upgrade their side of the peering connection unless they're paid.
That might sound reasonable at first brush, the ISP will incur costs associated with upgrading the peering connection. BUT so will Level 3 incur costs. So on that front they should cancel out, Level 3 pays for their side, ISP pays for their side, upgrade gets done and peering connection gets un-saturated.
Now wait, you might say, the backbone provider is pushing all this traffic on the ISP and it's not the ISP's fault that this deluge of data is headed it's way, but rather the backbone provider's fault. And there's some meat on those bones under one condition: the ISP doesn't have customers who are paying it for internet access. So that argument is not valid. The ISP's customers are the ones requesting the data that the ISP has promised to deliver and then the ISP wants to get paid twice. First by the customer for internet access, and again by the backbone provider for access to customers.
If it's so painful to provide the product to your customers that you have promised to them, perhaps you need to sell the business and get into another one. That is what literally every other company in the world does:
1. Offer to sell a thing to a customer
2. Customer agrees to buy a thing
3. Provide the thing to the customer
4. Customer pays for a thing
5. Profit!
These ISPs are so incompetent that they can't manage step 3 without getting paid extra, again, not just by a customer but also by their supplier. It boggles the mind!
I'm not an ISP advocate... My hatred of time warner runs deep into my soul.
But, I just completed an Office 365 implementation for a huge enterprise. Email for businesses of our size is similar to netflix in many ways.
Guess what we found out early on? 365 is different than web surfing and providing services to consumers and business partners. The model we used to deliver Internet wasn't effective for a variety of reasons, so guess what we did? We paid a bunch of money to establish a peering connection to the Microsoft network at our expense.
Netflix's solution is was to peer and to place a bunch of storage gear at ISP points of presence for free. That works, but may not be a solution that makes sense for the ISP for a variety of reasons. The issues that we're witnessing now is the lame ultimatum driven behavior that happens in contract negotiations.
IMO, we need rules that prohibit ISPs from discriminating against certain traffic. But there needs to be a line where exceptional bandwidth destinations have to take measures to be good citizens, which may require payment for connection or hosting.
"Netflix's solution is was to peer and to place a bunch of storage gear at ISP points of presence for free. That works, but may not be a solution that makes sense for the ISP for a variety of reasons."
Here's the thing. The ISP is over-subscribed, i.e. they've sold probably 25 more "contracts" for service than they can actually deliver. They advertise 25/5 Mbps service to customers but allocate perhaps only 1/0.2 Mbps (only 4% of what they're advertising for sale) across their back-haul and peering stations for their customers. This is under the premise that the network is bursty and people download webpages in a second or two and then read them for several minutes, etc. That assumption might have been valid 10 years ago. It's much less valid today.
So the situation isn't that Netflix is DDoSing the poor little ISP which is doing it's best but just can't keep up. It's that the ISP has KNOWINGLY oversold itself and it's now getting caught in the act and it's wildly pointing fingers at anyone and everyone it can. And it's attempting to re-frame the argument in a fashion that makes them look like the victims when in fact they're the villains.
In reality what Netflix is doing is bending over backwards to try and make their customers (who also by definition are an ISPs customers) get the level of service that the customer has already paid for and has a very legitimate claim against their ISP for in terms of transport from their house all the way to the ISPs peering stations. The ISP isn't upgrading their peering links unless they get paid to do so. They are holding their customers hostage and demanding ransom from people who their customers have quite literally already paid to be able to communicate with.
I can understand how this can be confusing because the ISPs are doing an AMAZING job of framing the argument in terms that make them look like the victims of horrible abuse. But that's simply not the case. They signed up for this "abuse" when they advertised certain speeds and started billing customers for it.
They're basically upset that customers are asking them to make good on their promises because they thought they'd never have to. I am sympathetic to their plight in the sense that I know how much it sucks to make a utilization assumption and to price it accordingly and to get it wrong. But having gotten it wrong in the past doesn't make it OK to continue to get it wrong always and forever.
EDIT: I just reconciled my 100x, 4% math which didn't make any sense.
Does the contract actually say they guarantee 25/5 Mbps? I strongly doubt that.
Bursty traffic and steady traffic cost different, so I think, in order to be honest, ISP should sell two differently priced 25/5 Mbps connection, one priced same as now, and one much more expensive. I predict people will not move to the more expensive pricing. So while there is some deception going on, I doubt people actually want to pay for steady traffic.
The contract says "up to" whatever because an ISP can't make any claims about how fast your link to any and all websites will be, once the traffic leaves their network.
Let's imagine for a moment a very simple case. A network with nodes A, B, C, D and E
A <-> B <-> C <-> D <-> E
Now let's suppose that (A,B) has a 1Gbps link. And (B,C) has a 10 Gbps link. And (C,D) has a 10Mbps link. Finally (D, E) also has a 1Gbps link. What is your ability at A to receive data from E?
It's the minimum bandwidth of all the links that limits your rate. So clearly we can see that the available bandwidth from (A,E) is 10Mbps because that's the limiting factor.
Let's suppose that my ISP consists of just the link (A,B) and they advertise speeds up to 1Gbps. And then let's assume that I can't (at node A) download from node E as fast as I'd like. The problem is that the ISP doesn't have any meaningful control over the link (C,D) because their network only covers the link (A,B). That's the reason that ISPs advertise "up to" a particular speed, because they have no control over what happens once you leave their network.
Now let's suppose that the ISP actually covers A, B and C. In other words they have both a consumer facing link, (A,B) which is the cable modem to POP link. And they have a back-haul link, which is (B,C) to their peering station, which is at node C. Node D is on the backbone providers network and is the other end of the peering arrangement from the ISP to the backbone provider. E is just about anyone else on the backbone network, perhaps Netflix.
In this case the ISP does have some sort of control over the bottleneck link. They can't unilaterally upgrade it like they could (A,B) or (B,C) but with the cooperation of D the 10Mbps link could get upgraded thus providing better (A,E) service to me. Ostensibly when I purchase 25/5 service I am paying to get 25/5 all the way up until the ISP doesn't have control over the network anymore. But here the ISP has DELIBERATELY bottlenecked me ON THEIR OWN NETWORK by refusing to upgrade their half of the (C,D) link.
I am 100% sympathetic to the idea that ISPs don't have any control once traffic leaves their network and thus the 25/5 isn't a guarantee. But the idea that they won't make a good-faith effort to provide 25/5 transit across their own network to a backbone provider but still advertise "up to" 25/5 to me as a customer is completely reprehensible.
If you don't have to make a good-faith effort to provide the 25/5 then why not advertise "up to" 1Gbps or 100Gbps? Customers are never going to see those kinds of speeds but by the logic that "up to" language absolves them of any and all responsibility to actually provide it, why not advertise it?
The reason is that they would get sued for fraud. They have no reasonable ability to claim the 1Gbps number over a link that can never ever get close to 1Gbps, either cable or DSL. What's the difference if they move the bottleneck a single link or two away but still inside the ISPs own network? As far as I can tell it just makes it harder for customers to figure out and thus less likely that they will get caught. The fact that it's confusing and many people don't know how networks work doesn't make it not fraudulent. It just means that it's highly unlikely that enough people will figure it out and start a class-action suit.
Great discussion, all. I wonder if the ISPs have thought about moving customers to 95th percentile bandwidth billing instead, so they can sell packages that are more honest about their overselling and actual expected speeds / usage: http://en.m.wikipedia.org/wiki/Burstable_billing#95th_percen...
Related; I haven't seen anyone point out yet that Comcast (and most other ISPs IIRC) already do have a bandwidth cap, and it looks like they're now on an even more complex model where the pricing / overage varies by market: http://customer.comcast.com/help-and-support/internet/common...
> The ISP isn't upgrading their peering links unless they get paid to do so.
At the same time NetFlix doesn't want to pay to expand their CDN capacity/diversity. Their motivations are exactly the same as ISPs -- cut costs & maximize profits.
The issue is, what triggers the costs? There's no good definition of what threshold requires a content provider to connect to the ISP network. Because ISPs are consolidating so rapidly, there's a huge opportunity for abuse.