How can Comcast or Verizon claim to offer you X MBps if they refuse, or are unable, to provide downloads from the Internet, including from Netflix, at X MBps? How is that not fraud or a breach of contract?
If billing rates were by volume of data transferred, just like every other utility, that would incentivize ISPs to maximize data flow. But as things stand they're not incentivized to increase bandwidth because they make the same money whether you check email once a week or stream Netflix 24/7. This seems obviously broken to me, but apparently most people can't stomach the idea of paying by volume, so we're stuck with the current system.
The problem with pricing by volume is that the average consumer can't really manage their volume.
Would the consumer be responsible for paying for:
-DDOS (not only can you knock them offline, you can cost them big money)
-Spam (Both received and botnet sent)
-Retransmissions for traffic the ISP dropped due to congestion
-Ads (adblockers could reduce bandwidth costs by a third)
I personally would love volume pricing. But I also would not want to provide tech support next time my parents get hit by malware and get hit with a $1000 isp bill.
Definitely valid concerns, but they don't strike me as inherent problems with the idea, rather technical problems that can be overcome with some ingenuity and incentive.
It's the incentive that's lacking at the moment. My machine may send out bursts of spam at 3am, or a website ad may weigh 10MB, but as long as I can still check Facebook what do I care? But I would care if I were billed for those transfers. I'd start shopping for devices that helped me self-throttle. I'd start caring whether websites were bloated. That would pass the incentive on to those providers, etc.
The architecture of the Internet can only offer a 'best effort' class of service so it's impossible for ISPs to guarantee speeds. At best they can offer an SLA to give the customer concessions (free/discounted service) if they fail to meet the terms of the SLA.
That's true. But in this case the ISP isn't even making a good-faith effort to provide X Mbps worth of transport across their own network.
I get that once the traffic hits the broader 'net that all bets are off. That's true and nothing can be done about it.
But in this case the ISPs aren't even making a "best effort" on their own network. They're deliberately not upgrading congested links where their peering counterparties are willing to upgrade. That is not a "best effort" class of service.
And the only way to break that claim is to give MORE MBps than that. I think the fcc is saying "ok, let's be realistic, your 'up to' is gonna be held to a standard higher than rando48 on hacker news saying I personally guarantee internet downloads of up to 1 billion MBps to ever citizen on earth."
Netflix became Level3s problem with Comcast because of unbalanced level of traffic flowing from L3 to comcast.
There is a simple solution to unbalanced Peering that Netflix could implement – just setup their clients to upload dummy garbage data at 70% max uplink speed whenever their client software is running. All of a sudden traffic will go both ways and they will be able to peer as an equal party.
This alone proves Peering agreements based on traffic amounts are a joke.
Streaming service that generates one way traffic is simply 'too nice' to comcast. Flood those asshats with /dev/random.
This brings to mind an interesting question: Does the FCC even have the authority to regulate net neutrality?
Even if the US government did somehow manage to regulate the US-based telcos, the FCC would have no legal way of regulating net neutrality outside of the US. An effective net neutrality policy could even harm our economy, if its a rule that nobody else needs to play by.
If we want access to a free internet, we need to figure out how to democratize the infrastructure. As long as we depend on fiber and copper controlled by big companies, net neutrality will continue to slip through our fingers like water.
> Does the FCC even have the authority to regulate net neutrality?
It depends on how the FCC classifies the ISPs. If the FCC classifies the ISPs as a common carrier, then yes that will basically force them to honor net neutrality. Due to <politics>, the FCC has been reticent to classify the ISPs as common carriers (instead classifying them as infomation services), but it has tried to regulate them as a common carrier in some ways (but not in all ways), ie net neutrality.
Recently verizon took the fcc to court over this and the gist of the court's ruling was 'The FCC has the power to enforce net neutrality on the ISPs iff it designates the ISPs as common carriers.'
> An effective net neutrality policy could even harm our economy, if its a rule that nobody else needs to play by.
An effective net neutrality rule could just as easily bolster our economy if no one else decides to play by that rule. What makes you think that net neutrality isn't a competitive advantage?
> If we want access to a free internet, we need to figure out how to democratize the infrastructure.
It's hard to democratize the infrastructure because it is so expensive to keep and maintain, short of nationalizing it like the interstate system, though based on the previous paragraph I don't think that's what you meant.
>An effective net neutrality policy could even harm our economy, if its a rule that nobody else needs to play by.
Can you expand on this? I don't see it. What does it matter to America, if Brazil allows telcos to meter speed to subscribers based on bribes paid by third parties?
It affects American businesses and therefore affects the American economy.
For instance, let's say Brazil's telcos started throttling their citizens' access to Netflix, but not to Braziliflix, a hypothetical Brazilian version of Netflix. That could prevent Netflix from gaining traction in Brazil. Meanwhile, Braziliflix would be free to compete on an even playing field for customers in the US.
Cool, that means consumers in the US will have two viable alternatives, Netflix and Braziliflix, so there will be competition and people will enjoy better level of service. And the two companies will try to out-innovate the other, hiring engineers and lowering prices. Sounds like economic gain to me.
Meanwhile, consumers in Brazil enjoy shitty services and exorbitant prices...
While that scenario is certainly better for American consumers, it's not better for Netflix. Both companies will have to innovate and compete in America, but Braziliflix will essentially get the Brazilian market for free.
That's correct. Note that I never said it was the FCC's job to prevent protectionism in other countries. I was simply answering deciplex's question of how net neutrality could harm the American economy if it were only enforced in the United States.
I must have missed the other discussion. I looked around the big threads but couldn't find much about FCC's authority when it comes to international networks. Feel free to link it.
On US soil with US citizens and US companies, it should be simple. But it is a complicated world in which we live, so it sounds like the FCC could only really protect equality in the best of circumstances.
For the sake of conversation, lets say the FCC isn't powerful enough. Lets say the internet is just too powerful, and there is no hope of the FCC or any other organization to regulate it. Would net neutrality be lost forever? Is it lost already?
IANAL, but I'd imagine that they'd fall under NDAs and trade secrets rather than copyrights (although I suppose the text of a contract could be copyrightable).
Still, I'd be curious to see if a FOIA request would work.
But there's a specific exemption for: "Trade secrets and commercial or financial information obtained from a person and privileged or confidential, 5 U.S.C § 552(b)(4)"
Years ago I lost a fight with the FCC for data on cell phone carrier reliability. The FCC has great insight into who has the fastest or most reliable cell network, but as part of mandating outage reporting, Congress promised the carriers everything would stay secret.
Transparency into the deals is nice, but doesn't change the fact that the deals have to exist in the first place. This seems in the same line of thinking with FCC's earlier "commercially reasonable" deals. "We'll give them fast lanes..but don't worry...the deals will be commercially reasonable...and now transparent, too".
Somehow it doesn't make me feel much better about it.
This sounds like a pragmatic approach, and pragmatism has always driven internet engineering. From the ISPs perspective, Netflix's needs are diverging from a typical free peering arrangement.
The issue is, what triggers the costs? There's no good definition of what threshold requires a content provider to connect to the ISP network. Because ISPs are consolidating so rapidly, there's a huge opportunity for abuse.
"From the ISPs perspective, Netflix's needs are diverging from a typical free peering arrangement."
I would argue that point. I pay for Internet access, not "All of the Internet MINUS Netflix" access. As a result my ISP SHOULD (doesn't mean they will) ensure that all it's peering connections are of sufficient capacity to provide me with whatever bandwidth I choose to use up until I run into the rate limits for the plan I'm on.
My ISP sells me 25/5 I think so the idea that I can't get 3Mbps worth of Netflix is suspect.
Of course you have to consider that when you're not saturating your last-mile connection it could be a bottleneck anywhere. If it's 15 hops from me to a server if any of those 15 hops is saturated, that's the limiting factor no matter how much capacity there is before or after. That's why closing a single lane on a freeway for as little as 10 feet can really screw up traffic. You have to merge down and deal with reduced capacity even though it's not miles and miles of lane closure.
What we've been told via one of Netflix's backbone providers (Level 3) is that Netflix isn't the slowdown, the backbone provider isn't the slowdown, but the ISPs are the slowdown. How can they say this with certainty? Because they're peered with various ISPs, the connections are saturated, and the ISPs refuse to upgrade their side of the peering connection unless they're paid.
That might sound reasonable at first brush, the ISP will incur costs associated with upgrading the peering connection. BUT so will Level 3 incur costs. So on that front they should cancel out, Level 3 pays for their side, ISP pays for their side, upgrade gets done and peering connection gets un-saturated.
Now wait, you might say, the backbone provider is pushing all this traffic on the ISP and it's not the ISP's fault that this deluge of data is headed it's way, but rather the backbone provider's fault. And there's some meat on those bones under one condition: the ISP doesn't have customers who are paying it for internet access. So that argument is not valid. The ISP's customers are the ones requesting the data that the ISP has promised to deliver and then the ISP wants to get paid twice. First by the customer for internet access, and again by the backbone provider for access to customers.
If it's so painful to provide the product to your customers that you have promised to them, perhaps you need to sell the business and get into another one. That is what literally every other company in the world does:
1. Offer to sell a thing to a customer
2. Customer agrees to buy a thing
3. Provide the thing to the customer
4. Customer pays for a thing
5. Profit!
These ISPs are so incompetent that they can't manage step 3 without getting paid extra, again, not just by a customer but also by their supplier. It boggles the mind!
I'm not an ISP advocate... My hatred of time warner runs deep into my soul.
But, I just completed an Office 365 implementation for a huge enterprise. Email for businesses of our size is similar to netflix in many ways.
Guess what we found out early on? 365 is different than web surfing and providing services to consumers and business partners. The model we used to deliver Internet wasn't effective for a variety of reasons, so guess what we did? We paid a bunch of money to establish a peering connection to the Microsoft network at our expense.
Netflix's solution is was to peer and to place a bunch of storage gear at ISP points of presence for free. That works, but may not be a solution that makes sense for the ISP for a variety of reasons. The issues that we're witnessing now is the lame ultimatum driven behavior that happens in contract negotiations.
IMO, we need rules that prohibit ISPs from discriminating against certain traffic. But there needs to be a line where exceptional bandwidth destinations have to take measures to be good citizens, which may require payment for connection or hosting.
"Netflix's solution is was to peer and to place a bunch of storage gear at ISP points of presence for free. That works, but may not be a solution that makes sense for the ISP for a variety of reasons."
Here's the thing. The ISP is over-subscribed, i.e. they've sold probably 25 more "contracts" for service than they can actually deliver. They advertise 25/5 Mbps service to customers but allocate perhaps only 1/0.2 Mbps (only 4% of what they're advertising for sale) across their back-haul and peering stations for their customers. This is under the premise that the network is bursty and people download webpages in a second or two and then read them for several minutes, etc. That assumption might have been valid 10 years ago. It's much less valid today.
So the situation isn't that Netflix is DDoSing the poor little ISP which is doing it's best but just can't keep up. It's that the ISP has KNOWINGLY oversold itself and it's now getting caught in the act and it's wildly pointing fingers at anyone and everyone it can. And it's attempting to re-frame the argument in a fashion that makes them look like the victims when in fact they're the villains.
In reality what Netflix is doing is bending over backwards to try and make their customers (who also by definition are an ISPs customers) get the level of service that the customer has already paid for and has a very legitimate claim against their ISP for in terms of transport from their house all the way to the ISPs peering stations. The ISP isn't upgrading their peering links unless they get paid to do so. They are holding their customers hostage and demanding ransom from people who their customers have quite literally already paid to be able to communicate with.
I can understand how this can be confusing because the ISPs are doing an AMAZING job of framing the argument in terms that make them look like the victims of horrible abuse. But that's simply not the case. They signed up for this "abuse" when they advertised certain speeds and started billing customers for it.
They're basically upset that customers are asking them to make good on their promises because they thought they'd never have to. I am sympathetic to their plight in the sense that I know how much it sucks to make a utilization assumption and to price it accordingly and to get it wrong. But having gotten it wrong in the past doesn't make it OK to continue to get it wrong always and forever.
EDIT: I just reconciled my 100x, 4% math which didn't make any sense.
Does the contract actually say they guarantee 25/5 Mbps? I strongly doubt that.
Bursty traffic and steady traffic cost different, so I think, in order to be honest, ISP should sell two differently priced 25/5 Mbps connection, one priced same as now, and one much more expensive. I predict people will not move to the more expensive pricing. So while there is some deception going on, I doubt people actually want to pay for steady traffic.
The contract says "up to" whatever because an ISP can't make any claims about how fast your link to any and all websites will be, once the traffic leaves their network.
Let's imagine for a moment a very simple case. A network with nodes A, B, C, D and E
A <-> B <-> C <-> D <-> E
Now let's suppose that (A,B) has a 1Gbps link. And (B,C) has a 10 Gbps link. And (C,D) has a 10Mbps link. Finally (D, E) also has a 1Gbps link. What is your ability at A to receive data from E?
It's the minimum bandwidth of all the links that limits your rate. So clearly we can see that the available bandwidth from (A,E) is 10Mbps because that's the limiting factor.
Let's suppose that my ISP consists of just the link (A,B) and they advertise speeds up to 1Gbps. And then let's assume that I can't (at node A) download from node E as fast as I'd like. The problem is that the ISP doesn't have any meaningful control over the link (C,D) because their network only covers the link (A,B). That's the reason that ISPs advertise "up to" a particular speed, because they have no control over what happens once you leave their network.
Now let's suppose that the ISP actually covers A, B and C. In other words they have both a consumer facing link, (A,B) which is the cable modem to POP link. And they have a back-haul link, which is (B,C) to their peering station, which is at node C. Node D is on the backbone providers network and is the other end of the peering arrangement from the ISP to the backbone provider. E is just about anyone else on the backbone network, perhaps Netflix.
In this case the ISP does have some sort of control over the bottleneck link. They can't unilaterally upgrade it like they could (A,B) or (B,C) but with the cooperation of D the 10Mbps link could get upgraded thus providing better (A,E) service to me. Ostensibly when I purchase 25/5 service I am paying to get 25/5 all the way up until the ISP doesn't have control over the network anymore. But here the ISP has DELIBERATELY bottlenecked me ON THEIR OWN NETWORK by refusing to upgrade their half of the (C,D) link.
I am 100% sympathetic to the idea that ISPs don't have any control once traffic leaves their network and thus the 25/5 isn't a guarantee. But the idea that they won't make a good-faith effort to provide 25/5 transit across their own network to a backbone provider but still advertise "up to" 25/5 to me as a customer is completely reprehensible.
If you don't have to make a good-faith effort to provide the 25/5 then why not advertise "up to" 1Gbps or 100Gbps? Customers are never going to see those kinds of speeds but by the logic that "up to" language absolves them of any and all responsibility to actually provide it, why not advertise it?
The reason is that they would get sued for fraud. They have no reasonable ability to claim the 1Gbps number over a link that can never ever get close to 1Gbps, either cable or DSL. What's the difference if they move the bottleneck a single link or two away but still inside the ISPs own network? As far as I can tell it just makes it harder for customers to figure out and thus less likely that they will get caught. The fact that it's confusing and many people don't know how networks work doesn't make it not fraudulent. It just means that it's highly unlikely that enough people will figure it out and start a class-action suit.
Great discussion, all. I wonder if the ISPs have thought about moving customers to 95th percentile bandwidth billing instead, so they can sell packages that are more honest about their overselling and actual expected speeds / usage: http://en.m.wikipedia.org/wiki/Burstable_billing#95th_percen...
Related; I haven't seen anyone point out yet that Comcast (and most other ISPs IIRC) already do have a bandwidth cap, and it looks like they're now on an even more complex model where the pricing / overage varies by market: http://customer.comcast.com/help-and-support/internet/common...
> The ISP isn't upgrading their peering links unless they get paid to do so.
At the same time NetFlix doesn't want to pay to expand their CDN capacity/diversity. Their motivations are exactly the same as ISPs -- cut costs & maximize profits.
You've got that backwards. From this very high level the Internet is PULL ONLY. Netflix doesn't start streaming data to customers just to waste bandwidth and fuck up your network with a flood. That's a DoS/DDoS attack.
The "flood" is actually your paying customers who thought that you were telling the truth when you promised them X/Y internet service. They are trying to utilize the service they have paid for and you are preventing them from doing so.
To not provide them with the service that was promised could be construed as fraud informally if not legally. I'm sure there is plenty of language in an ISPs retail contract that says that customers can fuck right off for any and all reasons and no matter what the ISP is not at fault. But while it might be legal it's definitely dishonest.
You don't have to peer with anyone... you wouldn't have any customers, but that's certainly your choice. You just can't charge for peering, given that it's your retail customers who are paying you to deliver them access to the Internet.
Well, sorry, but both the ISPs are the ones initiating connections to Netflix. If they think Netflix is flooding then, they can just stop asking it to send data (and deal with their clients afterward).
I agree that nobody should be forced to peer. But if a content provider is "flooding" an ISP, refusing to peer will just cause those packets to arrive via more-expensive transit. How bad do you want to spite Netflix?
Refusing to peer won't cause those packets to arrive at nearly the rate they were arriving previously, which means my network won't flood, which is less expensive for me.
I don't want to spite Netflix at all, I just don't want my network flooded.
(I probably also shouldn't have promised folks I could deliver all of these packets, either, but that's another issue)
Why would Netflix pay more for single network peering than full transit? Presumably Netflix could, hire a better ISP than Cogent and actually get the capacity they're trying to buy.
Cogent isn't the problem, from a technical standpoint.
The issue is that the last-mile ISPs have hardware that gets overloaded by the responses to the requests their clients are making, e.g. Bob sends out 10 packets to Cogent, and Cogent sends 10 million packets in response.
The last-mile ISP isn't prepared to deal with those 10 million packets all at once.
> This is quite literally a "have a cake and eat it too" type of situation.
How is NetFlix any different? They make the choice to lower the cost to deliver their service by choosing a CDN with inadequate existing peering relationships with ISPs. They also severely limit client side functionality because it makes their service cheaper/easier to support on a wide range of devices. NetFlix collects this data about ISP performance so why don't they say 'oh hey sorry ISP X user. We can't let you sign up because you'll have poor performance!' -- instead they happily take the money.
ISPs aren't stopping people from signing up for Netflix because they won't allow the maximum performance subscribers' service plans specify. Accepting pay for service isn't dishonest. Accepting pay for service you don't provide is.
Netflix has Internet Connection Speed Recommendations (https://help.netflix.com/en/node/306). ISPs offer service plans that cost more for higher maximum speeds. Is it Netflix's responsibility to ensure particular subscribers receive particular qualities of content, without regard for the contents' source qualities, the subscriber's bandwidth, the policies of their ISP, or any other range of factors in the last mile? Netflix is offering content, not quality. It doesn't sell differently priced subscriptions based on "maximum video quality", it offers a range of bitrates at a flat rate and sends the one that allows the best experience for your bandwidth.
Although, that's sort of the issue, isn't it? It's not "your" bandwidth that determines that now. It's not the plan you're paying your ISP for that's the limiting factor, it's the ISPs particular traffic shaping policies. Netflix isn't slowing its traffic down or decreasing video quality prejudicially based on your ISP or internet plan. It's the opposite, but they're being asked to pay, rightfully or not, so that some of their subscribers won't have their bandwidth throttled by their own ISPs.
I think ISP should sell "25 Mbps" connection and "Up to 25 Mbps" connection, priced differently. Then customers will (I predict) choose "Up to 25 Mbps" connection.
> Why would Netflix pay more for single network peering than full transit?
Because the interconnection between your transit provider and Comcast is frequently saturated and Comcast is not going to fix it since they would prefer to be paid.
