How do we, IT savvy people capable of monitoring, testing and verifying this stuff, use our collective skills to monitor each ISP and communicate who is doing the best/worst job?
As long as there is at least one ISP provider who is not throttling traffic, we should reward that ISP that does not discriminate on web traffic, and punish the ones that do. You might have to show some differences across different regions and cities, but this is critical. I don't know enough about monitoring web traffic to figure this out but it sounds like an awesome project to spearhead. How do we get started?
It's just that I'm just a little sick of the finger pointing, and would prefer to do whatever I could to fix the problem or use the markets to promote the good companies and punish the bad ones. If there is a market where all the ISP providers are throttling then you have a better case for government regulation. Who's with me?
As someone who is in DC doing net neutrality advocacy on a daily basis, I love this question. One of our major challenges is that ISPs have data on their networks and we don't. That leaves us in a position to speculate about what is "really" happening and hoping that someone out there happens to have the technical skills to potentially uncover a problem. The HN community's ability to evaluate claims made by ISPs as legitimate or illegitimate based on technical constraints is incredibly valuable.
Interestingly though, there is an argument that even ISPs don't have the right data, and that this is a reason - or at least an excuse - for throttling. See http://www.bobbriscoe.net/presents/1005ftw/1005ftw-briscoe.p...
There is an ietf working group (ConEx) on exposing the kind of information necessary, with the idea that this would provide the right kind of incentives for ISPs not to throttle. I'm not sure that ISPs wouldn't anyway have the incentive to be a gatekeeper, but it would be a good idea to remove any possible excuse.
Or, you could refer to the product you're interested in buying, as they can provide information about how well different internet providers can service their product:
I want to express my gratitude for your thoughtful and more effective approach. The aha! nature of this article and the sensationalistic title lead me to believe that it's a knee jerk reaction.
thanks - the concern is legit, it might be throttled like the article said. OR the guy on the phone is inept and has no idea what he is talking about. Either way the threat is real possibility but whether it is actually happening is hard to verify.
Lack of competition though is a hard barrier to climb. If there are only say 2 ISPs in your area it seems like you are pretty much SOL. And that is where government intervention needs to happen.
The primary problem we'll have is the 'reward' system in locales with only one provider. You'll have to focus on markets with multiple providers. The solution involves building an escrowed Bitcoin fund which does payouts based on provider's aggregate throughput. How that is implemented, I have no idea.
I was thinking a lot simpler - keep tabs on each ISP in each market. Determine who has the least discriminatory network. Once enough data is collected you can make recommendations. ISP X is the best, ISP Y is the worst, anyone who cares needs to switch from ISP Y to ISP X. No bitcoin needed.
You can have a coordinated event, a cancellation day where all users of ISP Y cancel on the same day and switch to ISP X. Make a big deal about it. Tell your friends about it.
The one concern I have is in certain markets there might not be enough competition. If all the providers in that market are throttling then you have a stronger case for the government to get involved. Heck, there might even be collusion in those markets. But you might be able to bypass any government enforcement of Net Neutrality if there are enough well informed consumers punishing the bad ISPs and patronizing the good ISPs. It aint rocket science.
This is one of the reasons I'm a loyal Sonic customer, despite paying more dollars for less bandwidth, relative to my friends and coworkers who use, for example, Comcast.
By far the most correct observation. These guys are just following a script - they probably don't even know what half of the words mean. Please don't read anything into it or assume the outsourced support is somehow privy to anything at all that goes on inside Verizon other than what the FAQ system he's using tells him on screen.
But the proof is in the pudding, the speed tests speak for themselves, no?
That's exactly it -- multiple, independent yet individually unreliable measures can actually produce very reliable aggregate evidence through convergence. This is particularly true if one uses abductive reasoning, but IIRC, there's statistical proof behind this idea as well...
That is to say that the speed tests say something and the chat says something, but together, they begin to paint an troublesome picture. Combine that with the timing and it starts to look bad. More data points will add to that convergent picture, of course...
As they say, a leading question's answer is usually no.
We need this to be verified by someone with intricate knowledge of whatever is going on inside Verizon's network. The number of people with the exact knowledge of what is going on will be necessarily limited.
I'd be with you if you'd said, "By for the most likely correct observation."
But you're asserting the truth of this position, not its probability, and I don't think anyone participating in this discussion thus far has sufficient information to do that.
The netflix data itself points to a gradual lowering of bandwidth to their service. Now here's 2 scenarios :
1) Verizon decided to enter some router configurations that affect the packet forwarding to Netflix (well, most likely FROM netflix, as the internet is not actually symmetrical). What we should see as a result of this is a sudden change in bandwidth.
2) The load on the interfaces in between is changing, either because of more netflix subscribers, more AWS traffic, or more netflix traffic. Since these are things happening slowly over time, you'd see a gradual worsening of traffic conditions.
I think it's pretty fucking likely that what's happening here is 2). Now granted, Verizon could solve this problem by cooperating with Netflix. But for them that's an expense with no upside, so likely they're demanding Netflix make it worth their while to upgrade the interconnects.
This is one of the basic problems with the internet today. It used to be the case that bandwidth between providers was much more plentiful than last-mile bandwidth. So we upgraded last-mile bandwidth. Problem : doing that resulted in an exponential increase in load on the core network and the interconnects between different networks. Needless to say, nobody can keep up with exponentially rising demands (especially without subscriber growth). The basic problem is that a linear increase in bandwidth for the customer results in an exponential increase in costs for the ISP (so -surpise- small ISPs don't feel the effects nearly as badly).
So ISPs are desperate. Their costs are spiralling out of control, with everyone demanding they follow this exponential curve, but of course their customers not willing to pay for it (and all the money in the world would only buy time if thrown at this problem). Demanding the ISPs pay for it is simply not going to work. Right now, yes, they could theoretically pay for it, but that won't last long.
Plus Netflix is being a rather bad netizen themselves. The polite thing to do is to carry the traffic to very close to the user on your own network, and only use other's links, especially transit, as a last resort ("cold-potato" routing, for content providers). Netflix has caches, but no own network.
There's other problems that result from this cost problem. Already transcontinental and other long-haul links are strained to the limit. This has resulted in a massive degradation of non-local internet traffic, and it's getting worse fast.
It is really a very simple problem. You have consumers, and you have producers. And you get more and more of each. To a limited approximation you can assume everyone on the internet is both. To simplify things, let's say everyone uses a tiny amount of everyone else's services. Now given N participants in the network, what is the load on the core network ?
N * (N - 1) =~ N^2 (assuming full duplex links, which would be generally correct for the internet)
So adding more participants in the internet increases resources required to give every existing participant in the network his "old" speed. These need to be paid for. But unless everyone want to start paying an amount for your internet connections that rises with the square of the size of the internet, we're going to see progressively worse and worse throttling. Right now the strategy is fast becoming limiting interconnect capacity to prevent the core network overloading, which would have far worse consequences.
While, yes, historically this has mostly been monopolies cheating the market, that's less and less true. This is going to get worse, fast. And since those resources are obviously not going to be made available, there's only one thing to say :
> The basic problem is that a linear increase in bandwidth for the customer results in an exponential increase in costs for the ISP
> N * (N - 1) =~ N^2 (assuming full duplex links, which would be generally correct for the internet)
1) N^2 is not exponential, it's polynomial.
2) Expanding capacity isn't even N^2, it's just linear.
If you double each residential customer's bandwidth, you "only" have to double the capacity of your uplinks to other networks. The number of endpoints or total nodes in the network is entirely irrelevant, because you have the same 50Mbps whether you're drawing it from one AWS server or a thousand BitTorrent peers. The only way you would get the result you're assuming is if every endpoint had a fully meshed connection to every other endpoint, i.e. each customer gets 50Mbps to each Amazon server, a separate 50Mbps to each Google server and a separate 50Mbps to every single other user on the internet, for a total of several hundred billion Mbps for every customer. That's not how it works.
And in practice it's even less expensive. It's sub-linear. Because just giving everybody a connection which is twice as fast doesn't mean everybody is going to immediately double their usage. A large fraction of users, given a faster connection, will do with it only what they currently do with the slower connection. Their pages will load faster but they won't load more pages, so the load they put on uplinks to other networks will remain the same.
There are no exponentially increasing costs. There are only the same costs as there ever were: Cisco et al come out with a new router that puts twice as many bits through the same piece of cable and if you want those bits you pay them a fixed cost and swap out your old model for the new one, and in another few years you do it again.
> Because just giving everybody a connection which is twice as fast doesn't mean everybody is going to immediately double their usage.
My understanding is that peak load is the issue, not total data transferred in a month. If Netflix downloads/buffers video content as fast as it can, and people mostly use Netflix at the same time of day, then giving them more bandwidth is a potentially big problem
Peak load is what you have to design capacity for, but that is hardly unique to Netflix. They seem to have it sorted for FiOS/U-verse/etc, don't they?
The argument they might make is that their services keep the data closer to the users, but that's such a cop out. They could do the same thing for Netflix with transparent proxies if they wanted to (but the result would be to make Netflix more usable instead of less, which is adverse to their interests as competitors). And in any event the uplink to a large peer is a totally inconsequential part of the cost of operating a network. The thing which is expensive is upgrading the last mile to handle a large amount of video traffic, but that cost is the same whether the traffic is Netflix or FiOS TV.
In the UK, I believe the main practice revolves around broadband, whereas in US it's all around phone (VoIP or otherwise). If you could enable competition within the same last mile service, you'd a)give a definite incentive to make sure the entire network delivers the best experience and b)have alternate sources of revenue to maintain the last mile network.
The current duopoly that exists SoCal in most areas (1 cable provider and 1 DSL provider) is a really terrible consumer experience. For those who have access to FiOS, it's a tad better, but not by much.
If you had access to 10 providers at once with comparable services, there would be real incentive for them to offer the best possible experience and lowest possible prices to consumers... but of course that would hurt corporate profits and so on and no politician getting those donations wants that...
What we really need is the equivalent of Glass-Steagall for internet services. Prohibit the company that operates the last mile from owning or being owned by anybody that offers end-to-end connectivity or over the top services to end users.
The problem with just local loop unbundling is that the last mile provider has the incentive to favor their own services. They can operate their own ISP division at a loss or with zero margins (more than made up for by profits in the last mile division) which disadvantages competitors who have to lease the last mile and thereby maintains the status quo of no competition.
I think people get confused by the Startup rule of Descriptive Speech which states that all growth is exponential, all interfaces are gorgeous, and all teams are composed of A players.
The thing is, it's not even close to doubling. I recently exchanged my 30Mbps ADSL to a 100Mbps fiber. In practice this means a jump from 0.8Mbps (yay long copper links!) to 50Mbps, which is most probably limited by the core. Thus, for vast residential areas (this is near the center of a 80k people town in a metro are of about a million), we are talking about multiplying the last mile by a factor of roughly 100x.
Ask yourself whether that ISP allowing speeds to languish for more than a decade such that they provide one ~100x increase instead of six or seven generations of ~2x increases has made their total costs lower or higher. The fact that they've been screwing over their customers to save a buck for a long time is not a thing that should inspire sympathy for their expenditures.
What 100x performance difference do you think exists between 30Mbps ADSL and 100Mbps fiber?
The answer is that there exists a box that mounts on a telephone pole. The purpose of the box is to terminate hundreds/thousands of DSL lines and connect them all to the central office with a single fiber optic cable. It allows you to get 30Mbps out of your 30Mbps DSL instead of getting .8Mbps, because the box is close enough to your house to get full speed unlike the central office. It saves the phone company from having to string a new strand of fiber to the premises of every individual customer who lives too far away from the central office for high speeds over twisted pair copper.
The first generation would have been actually installing the boxes, which they apparently never even did. Then you periodically upgrade the boxes to provide faster speeds as the technology improves and/or the cost comes down, e.g. from 1.5Mbps DSL to 3Mbps to 6Mbps to 12Mbps etc.
Giving everyone a guaranteed connection is a theoretical networking problem. The brute force solution as you've mentioned, is a crossbar switch, which is implemented in O(n^2).
However, theoretical networking problems like this have been solved many years ago. The Clos Network can provide full duplex links at significantly better big-O
I don't fully remember the details, but I believe you grow at approximately N*log(n). The problem is that the original Clos paper was published before big-O notation was invented... and that this was just "one other homework problem" that a professor gave to me about 5 years ago when I was in college.
So my memory is fuzzy, and the math is undocumented on the internet :-(
The internet is built on top of unreliable datagrams, which means the connectivity problem is even simpler. It satisfies the conditions of a Rearrangeably nonblocking Clos network. So the Big-O is even smaller than the above networking problem.
Clos wrote his paper specifically for phone connections. You cannot disconnect people for no reason while they're in the middle of a conversation. However, unreliable IP packets can be disconnected and rearranged, allowing you to use a cheaper form of the Clos network.
Either way, (like my professor from half a decade ago...), I'm going to leave the asymptotic complexity of "Strict Clos networks" and "Rearrangeably nonblocking CLOS networks" as an exercise up to the reader.
Mostly because I don't remember the solution... As a hint, replace the crossbar switches (in the wikipedia page) with a recursive Clos network and solve for the Recurrence relation. Use a 2x2 crossbar as the base case.
Have you thought about what such an architecture would mean bandwidth-wise on a WAN network ? I mean, dear God. This would actually require MORE links than full mesh.
In other words, core network capacity (meaning total bandwidth in all links) if you have a clos scales even worse than n^2, with n the bandwidth you deliver to users. Even worse : you need as many long-haul links as you have subscribers (because your subscribers are connected to different locations).
Is there question in your mind as to why ISPs don't do that ?
I think you're right about a lot of things in this post, but the one thing I disagree with is that internet traffic is going to continually follow an exponential growth curve.
The main driver behind increased internet traffic is the migration from television to internet for video content. I think that trend is going to continue over the next decade or so until everyone uses the internet for video content. But after that I think you'll see internet traffic level off to an extent.
ISPs need to build out their systems to handle that. I can't see anything else causing a dramatic rise in traffic so once they build out their system to handle video content it should be good for awhile.
Unfortunately, most ISPs are also cable companies, so naturally they are resistant to spending money just to shoot themselves in the foot. But this trend is going to continue whether they like it or not. It's probably going to be a really painful process for the ISPs and their customers. Netflix is going to be trapped in the middle. The ISPs will want to choke netflix out and take over their business. With net neutrality gone they just might.It's going to take some good PR and strategy for netflix to come out of this fight alive. If I was a betting man I'd say they'll put up a good fight but eventually be bought out by one of the big ISPs.
That's at best the average cost. You can also deliver a few terabytes for something like $3 in postage if you ship a tape, but that average cost would be deceptively low if you were trying to use that bandwidth to watch Netflix.
The real cost is building the infrastructure to handle higher and higher peak bandwidth. Suppose you're Comcast with 20 million subscribers, half of whom are trying to watch 5Mbit Super-HD Netflix during prime time. That's 5 terabit, and even if you could agree with Netflix's providers to divide it up geographically, it's still an outrageous amount of bits to figure out how to ship.
I don't understand: If under total load, with every customer pulling the maximum possible bandwidth simultaneously, Comcast could give you 250Kbps, you want them to sell a product that only gives you 250Kbps all the time just so their advertising will be accurate?
Like everything else, there's fine print: 20Mbps or whatever is described as a maximum speed, and many people actually see burst speeds that high. But as it would be unreasonable to be upset you can't get 20Mbps to Nigeria, it's also somewhat unreasonable to be completely upset at Comcast that you can't get 20Mbps to Netflix if Netflix is paying for cheap ISPs that won't fairly peer. I'm not sufficiently informed to take sides in that dispute, but I can easily imagine that Cogent (for example) is dumping way more traffic into Comcast than Comcast is dumping into it, and trying to get away without paying for that disparity.
> If under total load, with every customer pulling the maximum possible bandwidth simultaneously, Comcast could give you 250Kbps, you want them to sell a product that only gives you 250Kbps all the time just so their advertising will be accurate?
No, their advertising just needs to be accurate! They need to say "250Kbp bandwidth, max peak 20Mb". The real number has to be the lead, not the BS peak number. There's a parallel here to audio amplifiers. Advertising the peak wattage and not the RMS is stupid and deceptive.
There are two key differences: the first is, as other people noted, that they should advertise that range rather than always marketing the highest number. “256KB guaranteed, up to 20Mbs” isn't any harder to sell than telling people they won't always be able drive 65MPH.
The bigger issue, however, is whether those limits are based on the underlying limits of the network or artificial caps: this is currently completely opaque. It'd be much better if they were forced to publish any traffic shaping performed so the consumer can actually make an informed decision. This might lead to other questions such as whether they should be refusing to deploy Netflix OpenConnect nodes which would be healthy – and no doubt a key factor in why they won't talk about it unless forced.
It is practically and theoretically impossible for Comcast (a residential ISP with asymmetrical connection speeds) to have a balanced traffic relationship with any Tier 1 ISP. Every typical internet behavior of a Comcast subscriber follows the pattern of sending a small request, and receiving a large amount of data (webpages, music, video, etc.) in return.
The cost of transferring each bit is exponentially decreasing as network equipment tracks Moore's law. Their costs are going down, not up.
What this is really about is damaging competitors to their cable TV operations. Wail about Netflix using a lot of bandwidth, never mind that the cost of providing that bandwidth is falling as fast as the demand for it is rising, and you can put on a good show for the regulators as to why you need to destroy Netflix and push everybody back onto U-verse and cable TV.
That depends entirely on whether you see bandwidth costs as revenue or an expense. For many of these companies it's both, but generally they make most their money from consumers, and in that case it's an expense (to the degree it's got an ongoing price at all, given peering arrangements).
Didn't ISP's in the USA get a whole bunch of money in the past from the government to improve infrastructure but they didn't which now led to the current state of USA not being an internet leader?
Telcos got subsidies in the form of tax breaks to improve infrastructure. They spent their own money though. They talked about wiring America with fiber when trying to get the subsidies through congress, so that's why people claim they didn't live up to their "bargain."
Today this has been twisted into a false narrative: "the teclos got paid 200 billion to build fiber and never did".
A reduction in debt is a form of income though. If I owe someone $200 dollars. But then I only have to give them $100 I now have gains of $100 vs 0 if I had to pay the full amount.
To be fair, often politicians will refer to tax breaks as a cost, as in lost revenues, to the Treasury as if something was paid out. It's common enough that it's understandable about the confusion of what the narrative actually means.
Scenarios don't allow you to make blaming statements. We don't actually know anything about the connection between Verizon and AWS, other than it's slow. Making statements like "they're demanding Netflix make it worth their while to upgrade the interconnects" is speaking for others based on a presumption you had.
Saying they are 'desperate' makes them sound fearful. Verizon made $70B in gross profit last year. I can't see how they would be afraid of upgrading their network - it's part of the business.
This would all be correct, if not for the Netflix Open Connect Appliance. Netflix is moving content closer to the user, so only the last mile feels the affects of the traffic.
alphadogg had an interesting, albeit dead question:
"So, how does this analysis account for the fact that, in many reported cases like this, other services work just fine at high-speed? Your point 2 is that more consumers and producers are flooding the net as a whole, but that would cause widespread issues, not just premier services issues, right?"
This is a boon for indie ISPs who are too small to host a cache as the bandwidth is cheap and it makes Super HD streams available. I wouldn't call NetFlix a bad netizen.
It isn't even just "indie" ISPs who do it. Cablevision's Optimum Online, which is probably the biggest internet provider on Long Island and Westchester has a peering relationship with Netflix, and lands at #2 on the speed chart, just behind Google Fiber:
http://ispspeedindex.netflix.com/usa
It is also worth noting, that FIOS was #6. I do wonder if that has really changed or not.
And this is the reason why I cannot make a complaint about Verizon that will stick. Verizon purposely hires or makes it seem they hire crappy technical people. But when a person does 'know' either through some confidential/proprietary memo or through technical know-how, people begin to have doubts on the quality of the information.
I had a deep fight with Verizon over a chromebook 3g service that they no longer offered, and I tried to get a copy of the internal memo that said the service was cancelled, but they would not give it to me. And everybody thought that the person I spoke to was lying.
How did you conclude that this was a "remark from an outsourced, minimum wage, offshore worker" ? The author was talking to a Verizon customer support.
Also I do not think that Verizon would be targeting Amazon's S3 platform, even if it was a small subsection of it. Too many webservices use it and Amazon is no small player either. There are easier ways to limit Netflix bandwidth through data stream throttling like they do for torrents.
It's more about low available bandwidth + more users, visible in the afternoons, when he is at home, in his residential area.
Comparing speed there with what he has at the office, after office hours, he gets an 'yes' to his questions from a guy most likely located on a different continent.
I still don't see the net neutrality issue in the article.
I have experienced exactly this on Comcast. Suddenly, about two months ago, Netflix and Hulu became terrible to watch. I thought maybe it was my cable modem, so I upgraded it and still no luck. So I signed up for a VPN service and tested watching Netflix and Hulu both on the VPN and off.
Sure enough, when on the VPN, I get the highest quality picture. When I disconnected the VPN, Netflix dropped down to about 500kbps. Terrible.
An excellent test. If a VPN makes something faster, despite the fact that it adds overhead to the process, that seems like proof that it's being throttled.
Not necessarily. It might just be bypassing the congested route that everyone else on your ISP is using. Comcast and Verizon have had disagreements peering with Cogent, one of Netflix's ISPs:
It's not throttling, but rather the peering points along the "best" route between Cogent and Comcast/Verizon are overloaded, resulting in lousy connections between them. Look at this traceroute hop from Comcast to Cogent I just recorded:
7 he-0-10-0-1-pe03.111eighthave.ny.ibone.comcast.net (68.86.83.94) 21.377 ms 20.379 ms 19.387 ms
8 be7922.ccr21.jfk10.atlas.cogentco.com (154.54.13.161) 67.126 ms 67.747 ms 69.083 ms
40 ms to go one short hop in, I believe, the same data center.
It's not being throttled. There is a peering dispute between netflix's CDNs and the ISPs. Netflix's CDN's don't want to pay for delivery to the ISP. Normally CDNs peer with each other for free if the balance in each direction is equal. The problem is that any CDN that delivers netflix is hugely unequal since it's a huge one way stream.
The ISPs could just eat the cost, and let it be delivered for free. Or Netflix's CDNs could just pay the peering fee. Or Netflix could use different CDNs.
The reason the VPN works is because it sends the data over a different CDN that isn't saturated because of the dispute.
At least that is how I understand it, but I'm not an expert (or even a novice).
Umm wait, isn't such a dispute exactly what we call net-neutrality? Consumers pay ISPs for internet access, and now ISPs also want money from websites that provide a service to the consumers. But filtering those websites until they pay the ISP to be let through is the opposite of net neutrality. If what you're saying is true and ISPs are asking Netflix (or subsidiaries) to pay them to be let through unthrottled, then this is indeed the net neutrality problem as the OP reports.
Edit: Wait I think I understand. I messed up peering and ISPs, right? Because the CDN connects to the internet via peering, and they're charging money (which does make sense to me), not the ISPs.
Oh, and the even more galling part... when watching Hulu on regular Comcast (without the VPN), the shows average around 500-900kbps. When it flips over to displaying the commercials, it jumps to 8Mbps or more!
This is actually not malicious at all. An individual ad gets served to many more people than any given show, and as a result they'll get cached closer to the destination. Still pretty obnoxious though :)
I had this exact same problem with Sonic.net in SF. It was miserable. Each time I called they would tell me everything was fine and that it wasn't on their end- ignoring all proof.
On the verizon topic, soon after they got rid of the unlimited data tier my speeds would drop significantly after I reached 200-250MB of data transfer, but when I cycled the power and updated the roaming the speed would go back to normal until it hit the 200-250MB again. It wouldn't surprise me at all to learn that this was done on purpose or that they're currently limiting streaming traffic.
Yes it was mobile phone data, I should have been more clear in the original reply. Looks like I can't edit it now.
I had no problems with data speed/cap before they phased out the unlimited tier. I was grandfathered in, but like I said, it was never the same. I primarily worked from the phone during that period and it was a constant struggle. Which is why it wouldn't surprise me if Verizon, across all of it's services, throttled traffic.
Interesting I have been watching Netflix more than ever but I havent had any problem. I'm also using Comcast in Atlanta I wonder if its based on region?
It's only tangentially related to region. Comcast bring all of Netflix's traffic onboard in NYC and their East Coast routing is handled in New Jersey. Their eastern seaboard routing is about the only area where they're not having massive internal issues. Both their Denver (Midwest) and Oregon (PNW) iBone routing centers are having serious capacity issues that are manifesting as packetloss, giant latency increases and generally shitty route-arounds in the hope that it'll relieve pressure.
Why does Comcast use so few peering locations? Why do they route packets from their network to a university in the same city across half a dozen states?
Because its cheaper to build and maintain only a few hubs. It's also cheaper for them to only onboard traffic a specific centers. JFK is the big US routing hub for Cogent and has the benefit(?) for Comcast of being where all the other big US ISPs onboard Cogent traffic. This lets them collectively bully Cogent (and other providers) into paying higher access fees.
Why do they route packets from their network to a university in the same city across half a dozen states?
I couldn't really say, likely its more related to what that university is paying for. I don't see any unis in Comcast's BGP announcement meaning they're probably going over Comcast's "Business" or "Corporate" network (higher priority traffic, same network). Comcast does not have very fine-grain regional routing, especially in the Midwest, so you'll see connections that cover a short LOS take a long path due to lack of routing
Net Neutrality is fine. The FCC just needs to reclassify internet providers as common carriers rather than information services.
To quote from the Washington Post [1]:
"Broadband is currently classified by the FCC as an information service, a category that gives the agency a fairly limited set of regulatory options. If Internet providers were classified instead as common carriers, the FCC's rule would likely stand. In fact, the federal ruling on Tuesday upheld the FCC's net neutrality rules as a matter of principle; the problem is that the agency effectively tried to apply its powers in the wrong context."
I think the ISPs are going to eventually regret this. Once you get a reputation for having a crappy product, it's hard to lose it. And they will get that reputation.
Their own advertising has often touted how well their service can stream Netflix. People are already primed to judge them on it, because the industy's own advertising has taught them to.
I too have Verizon FIOS and have no issues to speak of, other than the cost. But I don't assume that everyone else has had similar experiences. Typically it works out that areas with choices tend to get better service while the experience degrades as customers have fewer choices.
Personally, I wish they'd quit trying to convince me to increase my bill by a significant amount by including TV, of which I've made it quite clear I do not want.
>Verizon FIOS has an excellent reputation and the product is far superior to the other choices we have.
Well, that's the crux of the problem, isn't it?
The shitty TWC service I pay for is probably far superior to the more expensive, 500kbps, line-of-sight based wireless network of the only other competitor I know of in the area. Not exactly a glowing testimonial though, is that?
This is why I think streaming is a broken model: it spikes usage at the same time every day. BitTorrent (or just off-line downloading) is so much better because you can amortize the download over the whole day and take advantage of off-peak bandwidth. We just need to convince copyright holders that its in their best interest to deliver a superior product.
Except streaming is the only viable option due to how content licensing works. You could do slow (200Kbps) download of content you plan on watching in the evening while you're at work, but because of restrictive licensing the cost becomes prohibitive ($10-40/night to rent and watch the same thing as on Netflix)
That sounds like a problem for content licensing, not me. Promoting a broken model because the model that model is built on is broken as well hardly bodes well for this burgeoning industry.
Do you know how many homes in your neighborhood have FiOS? I am wondering if it's a congestion issue - have you checked RTT in the day vs afternoon? What about looking at wireshark to see how many duplicate Acks/Re-transmits you're seeing?
As only the first-hop device appears to be different in the tracert, I doubt it's something upstream - if they were throttling, they'd probably do it at the network core - before it left the Verizon network for transit, which would apply to both circuits.
I'm on Verizon FIOS and since some time after the ruling Netflix has been taking 10+ seconds to buffer initial SD quality where before it was always less than a second. There are times where it takes 15 or 30 minutes to switch to HD and sometimes it never even does.
Meanwhile I can download games from Steam at 3+ MB/s, torrent at the same rate, and so on.
Maybe Netflix suddenly got a ton of demand shortly after the net neutrality ruling or changed their software, but frankly Verizon doesn't get the benefit of the doubt. They've done so many sketchy things that at this point if they aren't doing this on purpose they need to prove it.
I think Time Warner has started doing this now as well. Right after the ruling, Netflix quality was so poor that I bought a new modem. No improvement. I will have to check the other AWS services I use including my own stiff. Thanks for posting.
There is a lot of complexity involved with network connections. There are many plausible alternative explanations for your and the author's problems. One or two people talking about their experience isn't the kind of data you would need to jump to the conclusion that your ISP is throttling your internet connection.
Unless they were told to specifically say what they did, which seems unlikely, the chance of a first level technical support person having any clue what sort of QoS policy is implemented on the network is basically zero.
That sort of person would not have access to see such things directly and wouldn't have the skill set to interpret configuration.
That guy doesn't probably have any idea about what he's talking about. Call center types wouldn't know this and "throttle cloud services" just translates to "throttle the internet." The cloud is the internet.
if only we had a distributed group of people, all of whom with a vested interest in figuring this out, that could help us with doing speed tests in variety of configurations…
A rough look at the data suggests that Verizon DLS used to be around a stable 1.4 Mbps until October 2013 and is on a continuous decline since then with a current low of 1.17 Mbps. That would support the OPs theory, but it's only a single datapoint that may have other underlying causes.
I believe that's a few months after Netflix started heavily using Cogent as a provider, and thus Cogent's peering disputes with Verizon and Comcast began:
My company uses Cogent as an ISP, and remote workers have had nothing but trouble trying to use VPN and other connections back home via Comcast or Verizon because the Cogent/Comcast+Verizon peering points are severely overloaded, dropping packets at a 5%+ clip at some times during the day.
Totally true, but let's be honest even if the chat representative understood English perfectly and denied up and down that Verizon was throttling, he probably doesn't have the expertise to know one way or the other what is actually happening.
I have also noticed a similar effect on Verizon FIOS, where some video services work perfectly well during the day, but around 5pm or so start getting incredibly slow to buffer. Whether this is intentional throttling I'm not qualified to say, but it makes me quite hesitant to use Verizon in the future.
I've noticed this when streaming from the PBS Apple TV app, but not so much the Netflix app or website.
Occum's razor would suggest the problem is likely with the video services themselves, and/or congestion from other people who are now home from work watching videos, than something malicious from Verizon.
I'm very lucky to live in the Netherlands if I compare the average speeds, even with the mediocre ISPs, with the rest of the countries where Netflix is active.
You can't really compare a small densely populated country such as the Netherlands to a huge country with large rural areas such as the United States. If you live in an urban area here (such as the Bay Area) you get excellent speeds as well.
Someone should write a speed test website that looks to read from a set of common cloud services and then compares the speeds and records this based on ISP and location.
This shouldn't be hard to setup and you'll get a ton of page views probably for ever.
One can then see degradations over time and based on ISP. Would be even nicer to have some controls in there too so you can see if well known cloud services are slower than non-cloud services.
I'm a little confused by the average speed reported in their speed index. Not sure about the others, but I know Comcast and AT&T both offer several speed tiers. Is their average speed reported here affected by the percentage of customers who purchase the lowest tier?
I've updated the blog post to also include traceroutes from the 2 sites. I'm not a networking expert, but since the traffic is ultimately going through the same network - peering should be a non issue. Feel free to let me know if this is significant or not.
There are really too many variables to have any idea. For example, in both instances I see that your traceroute identifies the hop to your router, and you're using the 192.168.1.0 private network. Since your business example shows actual reverse dns lookups (from the looks of it), I'm guessing it uses a commercial switch.
Unfortunately, there's no way to determine the actual cause of the slow down.
For instance, you said that the slow down happens after 5pm, which quite literally is when 80+% of the working force gets off work. It could also be an artifact of AWS, and not Verizon FIOS.
Also, 2 instances does not a trend make. There's no way to know if you're one of those people who were sent a letter because they were using terabytes of data a month, and they're simply protecting the network traffic of 99% of their other users. Granted, I assume you aren't, but I can't be 100% sure.
For a truly independent test, you'd need the same modem/access device, without a local switch (i.e. a machine connected directly to the modem), and you would need VPN access to test both identifiable and non-identifiable traffic.
If you could verify that Netflix traffic through the VPN, on the same network, without any interference through the router, was delivering much more bandwidth than the same Netflix traffic, but in the open (not through the VPN), and could do so for a statistically significant portion of the Verizon FIOS user base, then you'd have reason to complain. Until then, I don't see your claim being taken very seriously.
> Unfortunately, there's no way to determine the actual cause of the slow down.
Is networking the most frustratingly complex field in technology? As someone who admittedly knows little about it it all seems like a pile of hacks stacked on top of one another. It's the only explanation I can come up for why it's near impossible to figure out why a network isn't behaving correctly.
We'd need more samples to make any serious conclusions, but it does look like you're taking the same route from VZ's gear to AWS. That would rule out any kind of peering spat or congested port theories. If the paths are indeed the same, it means that VZ is CoS'ing down residential traffic over business. You'll need to get more traceroutes during different times of day, too.
The net neutrality issue is a tragedy of the commons. Say you have a common green area and everyone shares it. People come and picnic and everything's great until a family of giants moves into town and sits on the entire green.
Netflix is like that family of giants.
The solution is to build a bigger green, but how is that going to be funded. Customers don't want to pay for it and feel they're already being overcharged. Netflix doesn't want to pay for it just because they are streaming giant files. Such costs are not built into their ability to make profits.
Your analogy seems off to me. It works as long as you admit that the reason the family of giants is there because everyone else wants them to be there.
Plus, both the customers and Netflix already pay into the system, it's just that the companies in charge of it want more simply because Netflix is a competitor to some of the services they offer.
So there's no issue with a business line? Sounds like they're NOT throttling Netflix per se but rather residential lines. That would make sense since businesses pay more and generally consume less.
Throwaway here, so take it for what it's worth but I swear my experiences are true.
I deal with Verizon Wireless (as far as I can tell, the article is about the other Verizon entity) reps often, I've heard this mentioned many times in an almost off-handed, positive, revenue-generating manner.
That they're positioning themselves to charge providers like Netflix for "priority" on their network. Except the way they describe it, it sounds like they will be deprioritizing or otherwise "shitifying", for lack of a better word, the provider's traffic. Netflix was specifically mentioned.
It's never mentioned on internal slides, but there's always some vague term on a slide that brings up this topic.
This is going to happen, people, unless legislation interferes with their plan and even then they may be able to loophole it. Net neutrality is the enemy, as far as Verizon (and possibly the other big telcos) are concerned.
I see it more as an opportunity for more VPN services. I can imagine the ads proclaiming something along the lines of getting the bandwidth you're already paying for.
Maybe there's something in the transition to IPv6 that might help with this? But then I guess they could do the opposite and throttle everything but what's on their ransom list.
Oh dear. Looks like the work left for Google Fiber isn't done yet. Hoping they'll make a run in New York (or at least upstate where I intend to move), but that's a long shot.
I wonder how much independent ISPs could make if they stick to a small list of customers, but take up the "We Don't Throttle" stance.
Google Fiber is the worst idea ever. You know this company makes the vast majority of its income through advertising and collecting data about you, right?
I understand that it's a great service for a great price, but the question is how long that remains sustainable. Right now they're sitting on such a pile of money that it doesn't matter whether they provide internet access in a few cities at a super cheap price. They benefit enough in reputation and in extra usage of other services when people are able to use very high quality internet.
Proof of this is that Google disallowed running servers (the very thing that makes the internet the internet, end to end communication) up until their reputation was almost completely tarnished in the hacker community. Only then did they relent and allow servers to be run. They're not much different from other for-profit companies.
Google disallowed servers initially as a non-insignificant number of people thought of the service as an easy entry into mass virtual hosting as soon as they got it. That started to affect throughput[1]. As mentioned numerous times elsewhere, their ultimate goal is to embarrass ISPs into providing better service. That seems to be working in Texas ATM. The hobbyists that complained got an exemption in their TOS for non-commercial use[2].
Of course, Google is collecting information about me. As is every other ISP (Verizon and Comcast hijack domain misspellings to redirect to their search pages), which is why VPNs and Tor are so popular right now.
> (Verizon and Comcast hijack domain misspellings to redirect to their search pages)
What?! Thought only OpenDNS (and similar services) did that bullshit, and in their case I can understand. In the case of an ISP... you're paying them already!
They totally did. I used to be on Comcast and they did it all the time. There's a setting to turn it off, apparently, but I was never able to. I switched to FiOS fairly recently and it was the same. Any time I browse to an invalid URL, it went to the Verizon search capture page that I had to manually opt out.
They would be stupid to allow servers. Buy your own business line if you want that. Google doesn't want others losing bandwidth because some people want to run servers.
There is a reason the internet was designed with end to end connectivity in mind. If my ISP randomly start filtering packets, like AWS traffic or incoming TCP SYN packets, net neutrality is broken and I will break the contract with that ISP right away. Contract or not, they won't get another penny.
It's a different matter if I consume 5TB of data per month and it's costing them a lot of money on bandwidth to upstream providers. But filtering servers altogether is plain old evil. It doesn't cost them anything and they don't have to decide what's right for me.
I don't think Google looks for traffic originating from a server specifically. I doubt it is really even possible as it would look just like traffic from a normal computer.
The prohibition on servers has very little to do with network bandwidth. It has a lot to do with people bandwidth - overhead associated with support, security and compliance.
>Google Fiber is the worst idea ever. You know this company makes the vast majority of its income through advertising and collecting data about you, right?
You never actually made a point here, do you have one? Obviously google makes money through advertising so getting you fast internet allows you to see more ads, is that a bad trade off?
Well my response to Google Fiber is somewhat triggered by a book called Ready Player One, in which a big company (with some similarities to Google) controls lots of things online, including many consumer's connections to the internet. They then of course abuse this, and it would be incredibly easy for Google to do the same and invade our privacy further.
So my point is, if it wasn't clear already, privacy invasion. As I mentioned, they are like any other for-profit company and not interested in doing what's best for consumers. Their goal of making other ISPs provide better service sounds so noble, but I doubt it.
OK, this guy lost his job, I am sure they record all the conversations, and just a grep -i "cloud providers" should be enough to find the conversation, sorry man,
I'm not sure I believe any of this for reasons others have pointed out. But let's say it's true. Is this an argument for or against Net Neutrality?
By most accounts I read NetFlix is responsible for 30% + of evening internet traffic. Kind of staggering. If an ISP actively throttled Netflix people would leave that ISP for another one that didn't throttle Netflix Netflix is clearly a hugely popular service with enough people that it could exercise that clout. Netflix can sign agreements and tout such providers (heck, bundle in sign up offers etc) So the market can solve this problem.
Secondly, why are they throttling this service? Because they can't charge effectively for it. We all love the simplicity of "all you can eat" pricing. Yet Netflix essentially exploited that model to their own benefit to the point now that model is under risk.
I find it fascinating that a service that has (legally and ethically and followed standard biz practices and sensibilities) grown to gobble 30% of all resources and effectively squeezes out other traffic is seen as a victim here. I'm not trolling so would love to hear what I'm missing.
Netflix has an ISP tracker on their site that shows performance by ISP. Trust me, if they know your ISP is a "throttler" they have the ways to get this in front of you and highlight "Hey you're having a crappy experience because you're using X and they throttle us. Y wants your business and will give you a $150 coupon to change. Click here now"
I had no idea someone could be this naive. WE'RE AT THE MERCY OF AN OLIGOPOLY. THEY HAVE MEETINGS BEHIND CLOSED DOORS TO COLLECTIVELY RAISE THEIR PRICES AND PROVIDE MINIMUM SERVICE. THEY CAN FUCK US AS MUCH AS THEY WANT, AND WE CAN'T DO SHIT BECAUSE THERE ISN'T ANY COMPETITION. THERE ISN'T ANY COMPETITION BECAUSE THE BARRIER TO ENTRY FOR A NEW ISP IS SO HIGH THAT IT IS LITERALLY IMPOSSIBLE; EXCEPT FOR COMPANIES WITH THE RESOURCES COMPARABLE TO GOOGLE. ONLY FUCKING GOOGLE.
Jesus Christ. I can't believe you are savvy enough to pay for your own internet service and you don't realize this.
>people would leave that ISP for another one that didn't throttle Netflix
That assumes people have a choice for their ISP.
If I want high speed internet I have two choices. Both of them are huge national companies.
If they decide to go the throttle route, where do I go?
>Secondly, why are they throttling this service? Because they can't charge effectively for it.
That's an interesting take considering how they offered a product at a certain bandwidth and I agreed to pay for that bandwidth. If I use it to capacity, the services I receive from third parties are somehow 'exploiting' the model? I don't think so.
If the post office offers to ship anything under 10lbs for a flat rate, is a company taking advantage if they offer, say, a 10 lbs cheese of the month package using that USPS service?
There is an important distinction you are failing on.
"NetFlix is responsible for 30% + of evening internet traffic"
This is false. Netflix is not responsible for the traffic usage. PEOPLE WHO WATCH NETFLIX are responsible for this traffic.
I also think you are incorrect on the "all you can eat" pricing. Of course, I am not privy to the actual business deal that Netflix has with their internet provider... but I very much doubt that they have unlimited bandwidth. They, like their customers, are most likely paying for how much they use.
There's your flaw. Most people have 0, 1, or 2 choices for ISP. And those ISPs offer similarly sucky service.
Compare US bandwidth rates to modern countries, and you'll see that the US businesses are not in any way competing with each other to the benefit of the consumer.
Somewhat. Most people in the US only have a choice between the specific type of line they use (Cable, DSL, Satellite). There are typically only one provider for either Cable or DSL in an area, and you're not even guaranteed to have an option between either.
I'm really struggling with this concept. I've lived in 3 different metro areas in the US and always had more than one option. I can see it for rural areas but in terms of % of population. Can't find anything definitive on the internet though.
I've lived in three metro areas, and only one had significant choices in provider, and that was a decade ago, 2004. I had the choice of three cable providers, and I was just on the edge of a local ISP's service area so I was able to get a decent price and great service.
I lived in northern Bothell, in Seattle, and I had one ADSL provider (not even 'fast' DSL) and one cable provider to choose from. In the metro area I live in currently, I've had exactly one DSL and one cable provider, and I've moved across the city and the variety didn't change.
How is it squeezing out other traffic? A packet is a packet, under the net neutrality model all packets are routed equally. Capping packet flow from a provider would be like Subaru owning chunks of the national highway and setting a limit on the number of Toyotas that are allowed to be driving at a time. If Verizon doesn't have the capacity to support the evening demand, they should increase their shitty infrastructure so they can deliver the service that they sold to their customers.
And what if I want to compete with netflix myself? They have the cash to make a deal with Verizon, I don't.
Most networks see a degradation in performance at peak times. That reduction in performance can result in all types of behavior. You may not watch a clip, may click away from a site etc that's loading too slowly (and that would otherwise load fine and capture your view a few hours earlier)
It's simple. NetFlix is already paying for that traffic. The users are already paying to get access to the Internet.
The ISPs are trying to make complicated arguments so that they can get away with charging the same traffic twice.
Verizon has special right of way access for its fiber cables and should be regulated as such. I can't change ISPs and the ISP is only able to exist because of exemptions from City, State and Federal governments.
"We treat all traffic equally, and that has not changed," the statement read. "Many factors can affect the speed of a customer’s experience for a specific site, including that site’s servers, the way the traffic is routed over the Internet and other considerations. We are looking into this specific matter, but the company representative was mistaken. We’re going to redouble our representative education efforts on this topic."
Although in the comments there's a guy who differs:
"Chris Walsh wrote:
Verizon is absolutely doing this, at least in Silver Spring. I stumbled on the daveblog's post via a facebook friend and was amazed because if finally explains what has happened to my home FIOS connection over the last few weeks, even down to the exact 40 kbps speed. ..."
Let's say Verizon was intentionally throttling all traffic to AWS at certain times of the day (which seems like an awfully blunt weapon, but whatever).
Where is the part of this where they are promoting their video services over Netflix's "inferior" experience?
Honestly, getting people fed up with Netflix (not realizing that Verizon is at fault) and then deciding to go with an actually TV package (with a discount for bundling it with internet, no less) might be the goal here.
That, or the goal is simply to hurt Netflix, which approaches the same end from the side of the provider, not the consumer.
To the people saying that you shouldn't trust some tech support guy, I counter with this: He is acting as representative on behalf of Verizon. Verizon trained them and Verizon gave them the scripts to read so that this tech will properly represent Verizon.
The industry as a whole will eventually just charge a per GB cost. During peak hours its more expensive...etc, just like electricity. The interesting part will be if they still have speed tiers. Its really in both parties best interest.
I'm guessing there will be speed tiers at first. The telecom industry doesn't leave money on the table without pressure from the government or fierce competition.
The parent question was asking whether or not speed tiers would go away if they charged for data by the GB. I'm saying they wont go away, at least not immediately.
Verizon may not purposely throttle content but they are certainly uncooperative because they want to be paid from the big bandwidth consumers. This is why Google is trying to make it more transparent to users who is to fault when you can barely watch YouTube in 360P on FIOS during some periods when speed tests to other locations yield 50Mbit. http://www.google.com/get/videoqualityreport/#how_video_gets...
File a compliant with the FCC. Yes, I know, but you have to start somewhere to let them know how their action/inaction/decisions affect consumers. The more complaints, the better.
Has anyone else tried duplicating some simple tests?
Ping google and netflix while connected to a verizon network.
Repeat on a network with a different provider, see if the speeds are proportional, or if there is a larger % decrease for netflix than for google when on the verizon network.
I know this doesn't explicitly prove anything, but if several people all get this result that starts to compile more evidence.
This is probably only the beginning. On demand video directly threatens the television business model of these companies. They will use every technology available to them to throttle users like crazy until they are compelled not to, either by the market (consumers en masse) or by regulation.
If companies are serious in battling ISP throttling, they should explain to users that their content will load slowly due to their ISP and they should consider switching.
I'm pretty sure the chats get logged, they can search through that. There are no privacy regulations in the US like there are in Europe about not spying on employees, and even here (Europe) most have an automated message saying "This [chat/call] may be recorded for training purposes."
AFAIK Netflix doesn't stream directly from AWS, but (while using it for their transcoding and so) uses a CDN. So blocking AWS wouldn't help in changing the quality of netflix, right? It certainly could be related though.
As long as there is at least one ISP provider who is not throttling traffic, we should reward that ISP that does not discriminate on web traffic, and punish the ones that do. You might have to show some differences across different regions and cities, but this is critical. I don't know enough about monitoring web traffic to figure this out but it sounds like an awesome project to spearhead. How do we get started?
It's just that I'm just a little sick of the finger pointing, and would prefer to do whatever I could to fix the problem or use the markets to promote the good companies and punish the bad ones. If there is a market where all the ISP providers are throttling then you have a better case for government regulation. Who's with me?