The one thing missing from all the articles about OpenConnect is any way for Netflix subscribers to see which ISPs are participating.
I'd like to know if my ISP participates and, if not, be able to call in to suggest they do. If they still don't join in a reasonable time frame, then I'd at least have the option to switch providers to someone that participates.
A program like this will be more successful if they allow for consumers to pressure ISPs to participate.
Unfortunately, as we've discovered from the whole net neutrality debate -- netflix has offered these boxes for free to all major ISP's... yet the Big5 ISP's have all turned them down.
A consumer broadband network often distributes out over a very large area and condenses into micro POPs. In the Bay Area think Pleasanton, Concord, Marin, etc. The ISP must then backhaul traffic from these micro POPs to a major POP where the whole internet interconnects (Santa Clara/San Jose).
If Netflix were to install a million terabits a second of peering and a hundred thousand caching appliances in San Jose, it wouldn't alleviate the ISPs pain point at all, which is the backhaul to the micro POPs.
Putting OpenConnect appliances into micro depolyments "out in the field" solves the problem, because it moves the content and caching out to the last mile. However the ISPs are reluctant to do this because then what used to be a half rack of telcom gear in a small shed somewhere has now become a defacto colocation facility. You have Netflix, Hulu, Amazon, Vudu, YouTube, Microsoft, eBay, and two dozen others each with 4u of gear in hundreds of locations that you are paying for power and rolling out to service when they fail (because you can't let competitors have physical access to each others gear, and a node failure can saturate the uplinks from the micro POP).
The physical box is free, yes. Imagine if every website you visited wanted to put a 1u box in your living room, which you paid to keep turned on and connected to the internet. If a box failed you either needed to unrack it, haul it down to FedEx, and rerack a replacement - or let a stranger in while you aren't home.
If you don't get a box fixed fast enough, or choose not to install the MSN or Baidu box, everyone in your house starts to complain about it.
I'm not defending the ISPs here, just simply explaining their perspective. I am a huge proponent of net neutrality, and I'm currently in the process of deploying something similar to OpenConnect for another large network.
Well, the boxes have multiple 10Gb ports on them -- so they can be as fast as the ISP will allow it to be on their network. And, Netflix will provide (free of charge) as many OpenConnect devices an ISP needs to serve all of their mutual customers.
Also, Netflix offered to change their distribution model into a P2P model as well -- the Big5 ISP's said no to that too.
That doesn't mean the hard drives can serve multiple 10Gb at once. And why would ISPs give them space, power, and cooling for free when every other colo'd box (including CDNs like Akamai) pay for that?
If the bandwidth savings generated by the box exceed the cost of hosting it, then take it. Otherwise don't. But AFAIK Netflix isn't willing to pay ISPs to take boxes that save the ISP money.
Nobody (sane) is trying to force ISPs to accept the boxes. But if an ISP refuses the boxes and they refuse to peer and they refuse to have uncongested transit... people start talking about regulation.
Also, a "negotiation" with a monopoly is often indistinguishable from extortion.
Citation needed. My understanding is that Netflix offers as many boxes as the ISP needs. This makes sense since each box is supposed to save money for Netflix and for the ISP.
That's ridiculous. What do you think is driving the traffic when an ISP doesn't accept the hardware. It's the same hardware just running in Netflix's data center.
The Security Now podcast had an indie ISP owner on the show back in June (10, I believe). Most of the show was Leo Laporte calling him as asshat, but one thing he said that was compelling was that his customers require Netflix -- if he doesn't give them Netflix, they don't sign up. The bandwidth was killing him, so he told Netflix he wanted the OpenConnect, and that he'd pay for installation and maintenance. They told him no.
Brett Glass is a bit of a crank, though. His ISP is unviable but rather than just going out of business honestly he's looking for others to subsidize it. It was discussed extensively on NANOG: http://mailman.nanog.org/pipermail/nanog/2014-July/068290.ht...
The only major ISP I know that definitively is part of OpenConnect is Optimum (Cablevision). That's why they have the fastest Netflix speeds (only after Google Fiber, and even then it is close) on the ISP Index Netflix publishes each month.
As an Optimum subscriber, I can say there is a massive difference between the Netflix experience at my apartment in Brooklyn compared to Netflix at my office in Manhattan.
They at least used to have something on their SuperHD page that would tell you whether your ISP participated ("Sorry, you can't view SuperHD content because your ISP has not opted in to our OpenConnect program"), but I think since they opened that up, there's no obvious list.
Hmm, do you know which spot in the video they discuss that (it's a long video)?
Seeing that they are using just controller cards instead of raid cards, and FreeBSD, it seems even if they were not using specific features of ZFS, the performance and other automatic things ZFS does to provide superior (compared to most (all?) other filesystems) data integrity and protection... they would be using it by default...
For all the hand-wringing about ISPs creating an "internet fast lane", OpenConnect sure seems like a step in that direction. If Netflix is allowed to colocate their server in an ISP's facilities (effectively eliminating the cost of their egress bandwidth), how does that not give them an unfair advantage against other internet video operators who lack the leverage necessary to broker this kind of deal?
OpenConnect isn't the first attempt at this sort of business model, either. Back in the 90s, Akami tried this same thing with their "FreeFlow ISP" service model - putting their servers in the ISP's point of presence in order to eliminate their bandwidth costs. Then, as now, major ISPs refused.
If you're a net neutrality advocate you should seriously consider whether this sort of peering arrangement should be tolerated. This kind of peering is a de-facto QoS arrangement that gives Netflix a performance and cost advantage over other video providers that aren't big enough or rich enough to justify and pay for direct peering with ISPs. It also gives bigger ISPs (that do more than 2Gb/s of traffic to Netflix and agree to peer, apparently) the same advantage over their smaller competitors. Isn't that exactly what we're trying to avoid with net neutrality?
This is not a new practice and I'm surprised it doesn't come up more frequently in the net neutrality debate. It's been going on for years -- broadcast.com did it, YouTube and Google do it, and I'm sure others do as well. We've never really had net neutrality -- peering agreements have been happening forever in shady back room deals at NANOG. At least the more recent shenanigans are happening out in the open.
Alternatively, if you're a network neutrality advocate, you should consider whether this is really the kind of thing you're advocating against. It's one thing to complain about ISPs intentionally throttling or dropping traffic, or shaking down services to get paid twice for the same traffic. It's something else entirely to complain about services going out of their way to establish CDNs near their customers and reduce overall network load.
Comcast is throttling / dropping traffic because they want Netflix to pay for bandwidth. You're right that they'd be getting paid twice for the same traffic, but ISPs already do this -- if Comcast and Netflix are connected via Level 3 they both have to pay Level 3 for the same traffic.
What if Comcast offered to host Open Connect devices for a fee -- would that be a violation of net neutrality? To me that just seems like the free market doing what free markets do (I'm ignoring for a moment that Comcast is basically a monopoly which is a legitimate problem).
What if Comcast throttled everyone _except_ Netflix. Is that really different than Comcast allowing _only_ Netflix to peer?
I am a net neutrality advocate in the sense that I'm against artificial traffic shaping. Netflix offering peering and cacheing solutions to offload massive amounts of traffic is not the same thing, and, in fact, is an attempt to prevent ISPs from shaping its traffic.
What? No.
This is the same situation as the company hosted in a DC that grows and prefers to pay for a 1Gbps connection than the previous 100MBps.
Net neutrality means treat all the traffic the same, doesn't matter where it comes from or what protocol it is. That's why "this" doesn't come up more often in the net neutrality debate, because it's got nothing to do with it.
Net neutrality was NEVER about making all content equally available to every consumer regardless of geography and network topology. If you're European or American serve customers in Australia, you should put a server there at your own cost, or your Australian users will suffer, net neutrality or not.
Netflix OpenConnect is just doing that on a much larger scale.
It's no different than if Netflix was paying Comcast (or whover ISP) to colo a server(s) in their data centers (you can do this as a normal citizen too).
The only difference, the idea is Comcast wouldn't charge Netflix for colo space since Netflix is "earning" it via cost reduction for the ISP. So they are still technically paying for the colo space.
Perhaps I missed your point though -- because this seems to be a much better way to a neutral net than having ISP's charging both sides of the pipe (which the small guys can't afford even more)
Is colo space at the ISP an unlimited resource? If not, then it will end up priced out of the reach of the small guys.
I suppose the ISP could offer some kind of "virtual colo", where instead of putting your servers in the ISP data center for fast access they offer some kind of faster connection to your data center. This would have two big advantages over physical colo. First, it would not have the space limitations so it could be more widely available. Second, the price could be based on how many customers you are serving at the ISP so that even small content providers can afford it.
This now sounds similar to "fast lanes", which have been the major focus of opposition to the proposed approach to net neutrality of trying to regulate under section 706.
The problem I see is that in both the company specific CDN approach and the "fast lane" approach it comes down to a content provider is making an arrangement with an ISP to make that content provider's content better for its customers on that ISP.
Is that against net neutrality? If the answer is "yes", then I don't see why it should matter if it is done via physical boxes placed on the ISP's network or by prioritizing traffic from that content provider or by purchasing a higher capacity interconnect to the backbone providers that content provider's traffic comes through. This seems to me to be one of those things where it should be looked at as a black box, and net neutrality should be concerned with the externally visible behavior of the box, not the mechanisms within the box.
The small guys can use Akamai who already has many cache boxes in place.
My understanding is that private peering and caches are primarily a cost optimization and are not about performance (as long as the "normal" Internet isn't deliberately congested). This way of thinking makes CDNs compatible with net neutrality.
Unless I'm reading this incorrectly this is a peering arrangement, which is qualitatively different than just hosting a box in a DC run by Comcast (which doesn't exist, as far as I know, so I can't do it as a normal citizen). I could colo a box "near" Comcast, but I'll be more hops away (worse performance) and I'll have to purchase transit from some ISP that does peer with Comcast (more expensive).
Most (all) major ISP's do have DC"s that you can colo in (including comast). Just call up their business services line and ask.
So, yes, you can colo in one of your ISP's DC's. Netflix is doing just that (which is why it's not a peering agreement, it's a colo agreement). They just don't get charged because they are saving the ISP the money they would normally charge for the colo service (sometimes more).
Even if you can colo with Comcast (which I can't find any information about online) you're still not direct-peering with their AS, which is what Netflix is asking for with Open Connect. I'm sure they'll also charge you for bandwidth. This is definitely different than putting a box in a colo facility. This is putting a box in a peering location and eliminating the transit cost and performance hit.
But charging some and not others is pretty clear discrimination. Colocation should definitely be allowed, but Netflix should pay the same rate a start up would.
Wanted to add a little bit more to my previous comment since I've been thinking about this for a while. The story I heard about broadcast was that they pursued this sort of peering arrangement with smaller ISPs so they didn't have to pay for transit -- basically to reduce their bandwidth bill. It was really a brilliant move on their part. Since then lots of other large bandwidth consumers have done the same.
I'm not against net neutrality but I do think the issue is more complicated than people make it out to be. Once you understand that this is common practice the position of Comcast, et al. does make more sense. If Netflix is saving a shit-ton of money and providing a better service to their customers by peering, and Comcast has less upside, why shouldn't Comcast charge Netflix for the privilege?
If Netflix is saving a shit-ton of money and providing a better service to their customers by peering, and Comcast has less upside, why shouldn't Comcast charge Netflix for the privilege?
I don't know if we can accurately estimate who's saving how much, and I definitely don't believe anything coming from Comcast. Given that there is no way to reach Comcast customers other than Comcast and both parties have an incentive to lie about their costs, I think zero is the fairest price for such peering.
I don't think your conclusion logically follows. The free market argument would be that the fairest price is whatever Comcast can get netflix to pay (or vice-versa). From a policy perspective I don't see a reason to intervene except that both parties have unfair advantages relative to their competitors (and Comcast basically has a monopoly). However, if that's the reason to intervene on traffic shaping then I think it's a reason to intervene on peering relationships too. I'd love to be able to peer with Comcast, but I can't. That's not fair.
I worked at MSN in WAN-OPS in the 90s. We did exactly that: find out what ma-and-pa ISPs we were paying transit for, offer to run a T1 directly to them for all MSCOM/MSN traffic. Easiest sales call ever.
Because Netflix uses something like 45% of North American internet traffic, they need to do something like this in order to efficiently deliver video. Of course this is going to give Netflix better ability to stream, but I don't think it is unfair.
It would be unfair if Netflix was paying to "speed up its traffic" (read as: slow down competitor's traffic).
The entire point of OpenConnect was to help ISP's reduce the inbound traffic burden into their networks.
Normally, Netflix has a CDN and when a customer views a video, that CDN serves the content --> which comes in through the ISP's providers --> costing the ISP bandwidth.
With OpenConnect, Netflix offers to give any major ISP that requests it, a free (or multiple free) OpenConnect devices. They are hands-off zero maintenance for the ISP, and they put them in the ISP's data centers.
This means, when a Netflix customers goes to view a video, it gets streamed off the OpenConnect appliance which is already inside the ISP's network -- and in-network traffic costs the ISP's almost nothing.
So, with OpenConnect, the ISP downloads a video once, then serves it up locally. Netflix customers get lower latency videos, and sometimes even higher HD content (overall better service), and ISP's get to reduce their traffic burden significantly, which saves them money.
This should be/is a win-win for everyone involved, the ISP, the mutual customer, and Netflix.
(the problem as we've seen through the net neutrality debate, the Big5 ISP's here in the US have all refused the OpenConnect appliances)
I work at an ISP with local caches from a few of the big players.
We've been contemplating getting a NetFlix OpenConnect device, but as yet it doesn't make sense for us.
The way it works is that once per day, it will sync the ENTIRE NetFlix catalogue to the OpenConnect device, which is an enormous amount of traffic (for us). Even if it's a movie or TV series none of our customers have ever watched, it still gets synced.
Because we only have ~25k internet subscribers, it would actually be more traffic to sync the OpenConnect device daily than we're currently pulling from NetFlix.
Obviously this would be different with vastly more customers, and we're watching the numbers as more and more of our internet customers subscribe to NetFlix.
We wish NetFlix's caching worked like some of the other big players, in that "popular" content is cached, as is anything watched by someone. If an asset is not popular and never watched, it will never move into our network.
Each appliance stores a portion of the catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles being added to the service, and re-encoded movies are all part of the up to 7.5TB of nightly updates each cache must download to
remain current. We recommend setting up a download window during off-peak hours when there is sufficient free capacity.
You can work with Netflix on this. Your peak usage time as an ISP should be 9 AM to 5 PM or 6 PM to 11 PM depending on if you serve businesses or residential. You then schedule the sync to be at 3 or 4 AM when normal traffic is at the lowest, resulting in minimal impact.
You can even get a cross connect (I believe Netflix will even reimburse your side of the CC fees, but don't quote me on that) to a Netflix hub so loading will not hit your 95th billing with your upstreams.
it seems to me that calling this an "internet fast lane" is a false equivalency. my understanding was that the internet fast lane concept was something that users would pay more to use, which leaves the isps free to render the "regular lane" unusable so as to bump the pricing. additionally, there was the whole notion of the "bundles" of websites available at different cost tiering.
openconnect appears, at least to me, like it's a collaboration between providers of different services to enhance the value for their shared customers.
I read that entire OpenConnect Deployment guide PDF, and I have to say that it looks like a really great, well-thought-out program. I'm impressed with how many moving parts there are and the time it must have taken to develop a system like this; they don't even touch on the details behind the Cloud Control Plane, and I'm sure that must have taken a while to develop. Netflix has said in the past that they only want the most skilled employees possible, and it certainly shows here.
What's to stop Netflix from later on asking the ISPs to pay for the cost of OpenConnect? The way that satellite TV networks charge a subscriber fee to the cable operators. Netflix has stated that they want to become like HBO. I don't imagine they're unfamiliar with how HBO become so successful.
What's to stop Netflix from later on asking the ISPs to pay for the cost of OpenConnect?
The fact that ISPs will unplug it five seconds later.
The way that satellite TV networks charge a subscriber fee to the cable operators.
HBO only charges each customer once (via the cable company). If Netflix tried to double-dip by charging the customer directly and also charging the ISP they'd get rightly flamed.
> The fact that ISPs will unplug it five seconds later.
At the risk of angering the many people who watch Netflix.
I don't get the double-dipping argument either. We already have that with television, newspapers, and radio being paid by advertisers yet still charging a subscription to viewers.
Google uses standard off-the-shelf sata drives too.
Too many disk failures to be worth spending money on the expensive "enterprise" drives when a standard SATA drive will due the job just fine (especially in arrays like this -- plenty of spindles to get performance)
Backblaze also uses off-the-shelf HDDs (including, when HDDs were in short supply after the flood in Thailand, buying USB drives and tearing the case off). They publish a lot of reliability numbers on drives as well (which match up quite well with my own experiences), as well as cost/GB numbers.
I remember that! I think "This Developer's Life" or a similar podcast interviewed them about trying to game the walmart hdd rationing (2 per customer) and stuff.
> For the operating system, we use FreeBSD version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise.
...or due to the less restrictive licensing, it's ok netflix everybody does it...
AFAIK Netflix contributes back all their changes to FreeBSD, so I don't see how the license matters.
My impression is that companies that don't like the GPL either deal with it (Android) or just violate it (almost all embedded devices). Nobody is actually willing to switch to BSD for licensing reasons.
Sony switched over from Linux (ps3) to kfreebsd (ps4) due to licensing. Sure they (netflix) contribute back to FreeBSD, I don't doubt it, but don't act like licensing doesn't have anything to do with it.
I believe the only portions of the development kits which used Linux was the communication processor, an isolated computing platform bolted on to the hardware to facilitate certain debugging functions.
FreeBSD is pretty standard for most storage servers, be it CDN's, file hosting, etc. When they say "BSD powers the internet", it's not really that much of an exaggeration.
I'd like to know if my ISP participates and, if not, be able to call in to suggest they do. If they still don't join in a reasonable time frame, then I'd at least have the option to switch providers to someone that participates.
A program like this will be more successful if they allow for consumers to pressure ISPs to participate.