> “We monitor all the connections and if someone is using a lot of bandwidth for a long period of time, we talk to them and figure out what they are doing,” Sutton said. “Often times it's people watching Netflix and then falling asleep and then it keep auto-playing things all night long.”
This is the kind of statement that makes me long for a faceless, impersonal ISP. As much as I'm impressed by a neighborhood banding together and coming up with a solution, the idea of a neighbor trying to "figure out" usage or "talk to" customers about it is horrifying.
Hopefully they have a mechanism in place to allow everyone to burst to available bandwidth, but to throttle people to a sustainable proportion otherwise. Such a mechanism seems both necessary and sufficient, and given the professional equipment they're using, it seems quite likely to be available.
When Comcast sends bandwidth usage nastygrams, people get up in arms about it; we talk about "network neutrality", and that ISPs should remain "dumb pipes".
While the ability to introspect traffic at all is a bug that needs fixing in client and server software, to combat surveillance, at least with an ISP I'm reasonably confident that only 1) a trusted subset of ISP staff and 2) the government (hopefully with a warrant) have access.
By contrast, would you want your neighborhood association looking at your ISP logs? Or anyone you know personally? And making it their business how much bandwidth you use and for what purpose?
(On a separate note, I wonder how much Netflix's and YouTube's CDN boxes cost, given that Netflix and YouTube tend to subsidize them.)
> On a separate note, I wonder how much Netflix's and YouTube's CDN boxes cost, given that Netflix and YouTube tend to subsidize them
I worked at a large ISP for 4 years, and for us, both the YouTube and Netflix CDN boxes were free.
However, the boxes consume a massive amount of traffic to keep themselves in sync. For us, YouTube wasn't worth having until we had over 15,000 customers, and we still don't have a NetFlix box because it syncs the entire NetFlix catalog and all changes, every 24 hours, so they actually use less traffic not having one.
> By contrast, would you want your neighborhood association looking at your ISP logs?
Maybe you misinterpreted? He said:
> " ... we talk to them and figure out what they are doing."
He didn't say, "we transparently force all traffic through a proxy and log what sites everyone is visiting".
I work for an ISP and we have some customers (both business and residential) that are completely clueless about what's going across their network and out to the Internet.
"We monitor all the connections" could mean many things. More to the point, they have the capability to do so; you don't need a proxy for that, just root on the gateway box and a mapping from IP addresses to users. Or, for that matter, DNS logs if they have a local DNS server. It's clear that they could have the capability; it's not obvious from the article what they're doing with it.
I'm the "Senior Network Engineer" and I have full administrative control over every device (of ours) in our network. Thus, I have the capability to monitor any traffic passing through any point of our network. That doesn't mean I do, indiscriminately, 24/7.
Perhaps it's because of my role but I interpreted the statement, "We monitor all the connections", a little differently than you, I think. Replace "connections" with "links" and, like the man in the article, I also monitor all the connections. When a problem occurs, I want to know about it before we hear about it from our customers.
I think that by "connections", you assumed, for example, TCP connections. Unless they have a bunch of extra money laying around -- and it didn't sound like they do -- I'd be willing to bet they are NOT recording every connection that passes through their network.
Not every ISP is evil and trying to track everything you do -- and the small community-based ones like this are probably the last ones you should worry about -- and you shouldn't automatically assume that they are.
The quotes in the article seem ambiguous. Monitoring all the links, and monitoring aggregate bandwidth usage, seems perfectly fine.
I wouldn't expect them to keep logs, but I also wouldn't find it surprising if, when they observed a problem, they ran tcpdump on the gateway.
> the small community-based ones like this are probably the last ones you should worry about
I certainly wouldn't expect "evil" from people who have gone to these lengths to do something awesome. But I would expect a well-paved road of good intentions. A small community-based ISP like this would 1) not have an army of people telling them what will get them in legal trouble or create a PR incident, and 2) have the kind of "play nice" mindset that makes them not automatically think to treat customer log data and other personal information as radioactive. Unless the designers and maintainers of the network are specifically privacy-minded folks, the thought might not even have occurred to them.
Probably because their wireless equipment is the bottleneck and they are utilizing a mesh network topography which means bottlenecks impact other customers. The article stated that their backhaul uplink was on 70Mbps for the whole island.
Seeing as they're using Ubiquiti equipment, I'm not sure why they are not using of their Gbps setups for the backhaul at least.
Probably because they can't afford the backhaul. They pay $900 for 100M.
According to the network diagram in the article they do not appear to run a mesh topology. They run a hub and spoke topology. Each repeater has multiple radios.
Even so, they have serious problems if they can't sustain their 30 Mbps peak usage with their wireless equipment.
> I'm not sure why they are not using of their Gbps setups for the backhaul at least.
Considering they received a loan to purchase the necessary equipment to start this operation up, I'm guessing it's because they don't have a need for it -- yet, at least.
Bandwidth isn't cheap. But they should impose a traffic shaping policy to bring their users' transit levels in line with the fees they pay, rather than poking around into who is doing what on the network.
> Bandwidth isn't cheap. But they should impose a traffic shaping policy to bring their users' transit levels in line with the fees they pay, rather than poking around into who is doing what on the network.
That's the curious thing. The users are paying more than enough to cover all costs, so there is no need for all this tomfoolery. I guess they have either have mentality or technical issues.
Sure, bandwidth is cheap -- when you're buying tens of gigabits at a well-connected facility. The further from one of those you go, the more the price tends to go up. They're buying, tops, 100 Mbps, and having it sent to them via microwaves from a tower on the mainland.
Indeed it is, especially when you consider their fees and costs. Based on the information in the article their monthly free cash flow is $2150, so they certainly can afford their current bandwidth usage be it at $9 or $30 per Mbps without playing silly games and spying on their users.
A bit of an inconsistency in the article, perhaps?
First:
> The monthly fees also cover the $900 a month DBIUA pays StarTouch for bandwidth.
Then:
> The StarTouch link uses burstable billing, with prices going up the more they use.
I don't think I've ever had any links that were, basically, a flat rate for n Mbps with the ability to burst over that for an additional amount, although that certainly doesn't mean they don't exist.
> "The Netflix example is really wierd, as usage is at it's lowest during the night."
Netflix usage or this particular network's usage? How can you know?
Right, I understand how it works. I would have expected, in this particular case, though, that they have a fixed amount of bandwidth available at a fixed dollar amount per month (just for the consistent/predictable monthly expenditure). It may very well be that burstable billing was their best option, though.
> at peak usage times, total bandwidth usage across the entire network is 30Mbps or so.
The microwave backhaul is connected to a 100 Mbps port which would come out to $9/Mbit/month. From what I've seen, that's higher than what you'll pay if you're connecting to an upstream in a well-connected facility. If we then assume 95% billing and peak usage at 30 Mbps, that's an obscene per-megabit rate.
It's quite possible that you're right, of course. I'm making some assumptions based on my experiences which very likely are different from those on a small island in the PNW.
> I would have expected, in this particular case, though, that they have a fixed amount of bandwidth available at a fixed dollar amount per month (just for the consistent/predictable monthly expenditure).
Wholesale bandwidth is sold with a commit and burst. The commit is your fixed monthly charge (and bandwidth) you pay for each month regardless of how much you use. Overages, calculated with the 95th percentile formula, are paid at the burst rate.
> The microwave backhaul is connected to a 100 Mbps port which would come out to $9/Mbit/month. From what I've seen, that's higher than what you'll pay if you're connecting to an upstream in a well-connected facility. If we then assume 95% billing and peak usage at 30 Mbps, that's an obscene per-megabit rate.
Well, yes. I remember Hurricane Electric running a promotion for 1 Gbps of IP transit for $360 a month, but then again they are not present at a HE PoP.
As to pricing, it sounds like run of the mill rural charges. Also we don't know their commit, so can't really calculate with Mbps rate. Perhaps they just decided to go with a zero commit and $30 burst rate instead of paying a flat fee of $1500 for the 100M port. Would make sense at their very low usage rates.
>> "The Netflix example is really wierd, as usage is at it's lowest during the night."
> Netflix usage or this particular network's usage? How can you know?
Both. Because that's how ISP bandwidth usage universally is. Peak usage starts after people get home and peak usage ends as they go to sleep. Few stay up during the night and even fewer schedule nightly downloads.
In many places in the country, especially in rural areas, backhaul bandwidth is actually both expensive and metered. "Bandwidth is cheap" is a rule of thumb that really only applies to places close to the major fiber routes.
This particular location is quite well connected, despite being an island. They have a local fiber co-op, a subsea fiber cable and line of sight to several mainland backhaul sites.
Are you familiar with physical internet infrastructure pricing, or are you just assuming? I don't think access is priced as bandwidth. I used to work at a small business isp and that was not how physical access was priced, particularly over relatively bandwidth limited physical connections. Though perhaps you work in the industry?
It's going to differ around the world but in Australia for business connections I'm used to paying a separate fee for the physical connection (which will have a maximum throughput) and a separate fee for the bandwidth allocation and a third for the monthly data cap.
Australia is a bit of an outlier, mainly due to Telstra and the Gang of Four. I believe the polite expression for their actions is "maximizing shareholder value".
Netflix has no auto play that I'm aware of. When the movie is finished, it stops. Good point about neighborhood watchdogs. I don't want my neighbors knowing what I do online.
It's really never made sense to me. Sure, play two in a row, but why would Netflix keep playing episodes any incur the royalty costs when people fall asleep watching a show, don't turn off the service when they turn off the TV, just plain walk out with the show playing, kids shows keep playing with no one watching?
I assume they incur royalty costs, or is there some incentive for them to somehow boost the number of hours of content played? Is this one of those cases where they are trying to inflate their numbers to look good to investors?
It will play 2-3 in a row without any user interaction, but if you haven't taken any action in a while it'll stop and give a modal "Continue Playing?" message
Yeah, I thought this was really weird, too. I'm actually in the process of building out a small WISP outside of Georgetown, TX to serve an area that's right now /only/ being served by cellular, and I've gone with Ubiquiti gear. Not only does it allow traffic shaping but the practice is encouraged. There shouldn't be any need in a PTMP Ubiquiti setup to call people and tell them to lay off the Netflix.
> This is the kind of statement that makes me long for a faceless, impersonal ISP.
Slightly OT, but this statement reminded me of a question I've wondered about before and researched without finding a definitive answer:
Your average ISP probably has a lot of data which, in the wrong hands, would be excellent blackmail material. What regulations (if any) prevent the sale of that data to whoever wants it?
If you are in the EU there are pretty extensive data privacy rules. If you are in the US, it is a bit more complicated. It would mostly depend on what the US ISP claimed it was doing. If it did something beyond what it claimed it was doing, it could get in trouble. However, most ISPs are pretty broad with their claimed rights. Still a big problem with the bad ISP idea is that it would be hard to exclude children, and that could cause issues.
I think that you forget that most of the people on Orcas are retired, and don't have the same fears that we have regarding snooping of data. My parents retired to Orcas over a decade ago, and all of their neighbors are 60+. Also, their internet habits are very different than my household - they check emails once per day, and that is really it.
> the idea of a neighbor trying to "figure out" usage or "talk to" customers about it is horrifying
I sympathize (and would therefore avoid this size community in the first place, I think) but I'm curious: what would you have the customer-neighbors do, if they're being impacted by one person's selfishness?
> I sympathize (and would therefore avoid this size community in the first place, I think) but I'm curious: what would you have the customer-neighbors do
Throttle traffic, such that if there's more traffic than the total upstream bandwidth, everyone active at any given time gets a proportional share of the bandwidth.
To take an example from another domain, much of California's Central Valley municipal water infrastructure is unmetered. A consequence of this is that problems with water infrastructure go unaddressed. In most of the cases I'm aware in which meters have been installed, even where there's no actual usage service billing, a near-immediate result is to find many previously undiscovered leaks, mains breaks, or service pipe breaks. While I'm not an uncritical fan of "you cannot manage what you don't measure", there are times when having some sanity checks on usage help.
In the case of online usage, it may be that you're a spam hub. Or that, as at one hosting provider I worked discovered, the interaction of a client's autoplay audio and a particular version of MSIE meant that we were continuously streaming what should have been a 5 second audio clip. We overshot out 95%ile bandwidth cap that month.
They still won't do any harm to anyone else. (Or if they do, then either the traffic shaping or the billing is broken.) Still not a good reason to track where the traffic is going.
However, it would make sense to give everyone fully-automated metrics on their bill, letting them know their total transfer, along the same lines as what the upstream providers provide. Make it clear that it's perfectly acceptable to use what they pay for, and that by design there's no information about the type or destination of traffic, but inform them that if the usage looks unexpectedly high compared to their own known usage (provide equivalencies in "hours of HD video" or similar so they have a baseline), they may want to investigate for themselves.
In the video, there's a shot from what looks like Cacti, an SNMP monitoring program typically used to monitor network utilization on routers (among other things). That's probably what they mean when they mention "connections".
It makes me wish that Netflix, et al. provided a bit more in the way of configuration/control options. In this case, the ability to turn auto-play off. One checkbox, with a bit of code behind it.
Realize that not everyone is in the same boat, connection-wise. And with data caps continuing to loom, some would like a simple setting to help guard against accidentally running over.
Also, believe it or not, some of us like the credits. And with outfits like Marvel stuffing teasers and jokes in after they've run, that interest may be increasing.
Hmm. Wonder how I missed these. I recall going looking for such a setting, at least once. Thanks!
P.S. Found the described setting in the web client. Those settings pages look a bit different than I recall. I think they have added options to them since I last looked (months ago). Or, I am simply clueless -- wouldn't be the first time.
I looked for this on my Nexus 7 2013 running what I imagine is the current version of the Android Netflix app (no reason on my part that it wouldn't be), and I didn't find it.
Under the hamburger menu, I can only choose which user profile and what category of content I want to view. There is no "Account Settings".
The "three dot" menu does have a "Settings" item, but it only offers a very limited set of settings, not including the one you mention.
I did find a setting on the Netflix web page interface, and I changed that. But it appears to have had no effect on the behavior I experience under the same account and profile on the Nexus tablet.
(The web page Settings page offers many more choices than the Android app Settings page.)
Thanks, I never thought to log in to the website and look for a setting like this. We -- well, the girlfriend -- only use Netflix on a Roku and I wasn't able to find any settings for it there.
To be fair, "Netflix played all night" is not actually a thing that happens, as far as I can tell. When autoplaying, they pause every few episodes in case you're not watching. I'm sure they don't want to pay bandwidth/licensing for shows playing to an empty room.
I may be wrong on this, but I believe they are legally required to stop auto-playing every so often, so that it still counts as on-demand streaming and not simply a TV channel.
In fact, that's probably not correct, not sure where I picked that up.
Netflix has to pay for every "view" to whomever they're licensing content from, just as Spotify has to pay for every listen (although, the rules are much different for internet "radio" than they are for video streaming agreements Netflix has).
All services check if you're still around so they're not spending licensing money they don't have to.
I haven't allowed myself to run into it, but I seem to recall having read that it is after circa 3 episodes.
Nonetheless, if you are in the habit of dosing off, many nights, that can still result in a lot of "wasted" bandwidth, particularly with some of the more miserly caps being described in the (U.S.) press as having been imposed or "coming soon."
This is the kind of statement that makes me long for a faceless, impersonal ISP. As much as I'm impressed by a neighborhood banding together and coming up with a solution, the idea of a neighbor trying to "figure out" usage or "talk to" customers about it is horrifying.
Hopefully they have a mechanism in place to allow everyone to burst to available bandwidth, but to throttle people to a sustainable proportion otherwise. Such a mechanism seems both necessary and sufficient, and given the professional equipment they're using, it seems quite likely to be available.
When Comcast sends bandwidth usage nastygrams, people get up in arms about it; we talk about "network neutrality", and that ISPs should remain "dumb pipes".
While the ability to introspect traffic at all is a bug that needs fixing in client and server software, to combat surveillance, at least with an ISP I'm reasonably confident that only 1) a trusted subset of ISP staff and 2) the government (hopefully with a warrant) have access.
By contrast, would you want your neighborhood association looking at your ISP logs? Or anyone you know personally? And making it their business how much bandwidth you use and for what purpose?
(On a separate note, I wonder how much Netflix's and YouTube's CDN boxes cost, given that Netflix and YouTube tend to subsidize them.)