On a meta level, this is one of the reasons why I tell every junior/entry level person I encounter in the ISP business the following: Ethics is important in network engineering. You can and should refuse to do things that cause measurable harm to the Internet. You should understand why certain things are bad, and should make a conscious choice not to aid and abet them.
It is regretful that organizations like NANOG, RIPE, ARIN, APNIC and others do not take a stronger stance (on a global geopolitical level) against censorship of the Internet, and attempts to create walled gardens and national firewalls.
The people who possess the equivalent of 'enable'/'configure' on the core routers of gigantic ASNs have real power to refuse to further harm the Internet.
We have seen a number of discussions on HN about ethics in software engineering. A quick search for posts with 'ethics' in the subject line turns up a number of things. In my opinion, ethics in the ISP, telecom and network engineering industry is equally important.
It'd be nice if we could address some things like BCP38 (anti spoofing), RPKI, route filtering and folks who knowingly support infrastructure that's used for outbound ddos (c2s and regular hosts), spam and malware phishing. Plenty of hosting shops in US and Canada have these problems. That seems a bit more within our reach whereas an ISP in India is more than happy to pay a vendor to implement middlebox packet molesters.
I've been dealing with a ban evader/forum shock image spammer for months now, and the place he is buying proxies from is actively doing BGP hijacking on resources owned by AT&T, Windstream, hospitals and universities - for the primary purpose of carding and fraud. I haven't managed to get anyone knowledgeable at those companies to figure out how to pressure the small upstreams (that are not those T1s) to stop it.
Good luck. When I wrote forum software, moderation controls, what people might call shadow banning these days, and other filtering took up 70% development time. Retaliation was DDOS. I was one of Cloudflare’s first customers.
I'm about to start down a road that might lead to where you are, but my target audience is a bit more mature and laid back so it might not be an issue.
But what you said reminded me of a conversation we had here a month ago. I think it may be that I reserve image upload functionality for users who have proven their humanness (and their humanity).
In my case image quality matters much more than quantity, so I can afford to make that choice.
That depends on perspective, and what you consider trolling. Yeah...the 5 dollars is a great filter for the vast majority of the Greater Internet Fuckwad Theory[1] posters, but that just narrowed posters and moderation down to a particular echo chamber (not unlike HN, fwiw). If you are down with the content/tone of the majority of MF posts, you prolly think it's great. But just like HN, there are posters with 'cred' who can say most anything (things I would definitely consider a troll), the 'little people' who can say things, including trolling, as long as they don't deviate to far from orthodoxy, and the unclean who get moderated away (hey...they paid 5 bucks so can't just punt them). They have done a remarkable job at maintaining the illusion that they're some moderation utopia for a long time.
Sounds like you have an axe to grind with some of the political or ideological positions of the people who run Metafilter. I never said it was run as a user-administered democracy, it's run by a core group of about five people who also own the servers. It's their pet project.
It's their pet project with their pet axe to grind. I said as much. If you disagree specifically with the points I made about moderation, which was all I commented on, then cite where I'm wrong. Otherwise, accept it's just as much an echo chamber as all the others and not some magical moderation utopia is made out to be by it's uncritical fanbois. But "you just don't agree with their politics", which mostly I do, is just sad, lame apologist crap.
Could you drop some IP prefixes you believe are hijacked and timestamps? I can probably help out and bring this to folks who can take action.
(I have to deal with hijacks frequently and part of our investigation is beating on folks who have permissive filters and pressuring their peers to improve things)
Absolutely agreed. It is really easy to be a shitty, lazy colocation/hosting/VPS hosting company. It is somewhat more effort and more difficult to be a proper one. Margins are so thin in the hosting business that it would be own version of a personal hell... I feel a lot better about operating symmetric gigabit last mile residential services.
Yes. Many of these shops follow the same design patterns. Ports for hosts at L2 that they bill on that are part of a big Vlan/L3/SVI interface that has tons of customers. I've seen these configs where folks have hundreds of "secondary" ip addresses setup where any customer can steal other customer IPs in addition to lack of anti spoofing. It's slop and it's tolerated.
All well and good, but are these kind of 'middleboxes' unequivocally unethical? For example, some ISPs might want to block highly illegal content - let's use the typical examples, e.g. child porn sites, malware domains, and so on. It's not inherently unethical (or, at least, there are plenty of reasonable people who would say it is ethical) to install a middlebox that will make it more difficult for users to access these sites.
So now your company has got a content blocker installed. What exactly are your network engineers meant to do now? Demand personal refusal over any additions to the sites that these boxes will block? That seems highly unlikely to happen, and how could that even work in practice? Are all the engineers meant to vote on blocks, and only those unopposed sites get added to the list?
Can ethical network engineers usefully oppose content blocking?
Whack a mole can be a highly successful strategy if there is a cost to having the mole appear somewhere else and the whacker has more resources than the one controlling the mole.
Successful does not imply sustainable. There is an argument to be made for doing manual strategies like whack-a-mole until you have a generic solution, but if you don't have the generic solution coming down the pike, it's time to go back to the drawing board.
Devil's advocate response: If these content blockers are just 'futile games of whack-a-mole', then why are you getting up-in-arms about their existence? Should be easy to avoid them if you truly believe what you say.
Stuff like what the UK is trying to do with a DNS based "black list" of bad things on the internet? Futile game of whack a mole.
Authoritarian regime that forces all ISPs in a country to run networks funnelling all traffic through a government run central point where they do DPI and flow analysis on it (Chinese GFW for instance)? More of a real threat.
For instance there is one ASN in Iran that has transit connections to the outside world. All ISPs are forced to be downstream of it. https://bgp.he.net/AS12880
Well, yes, the original article is about funneling traffic through a centralised DPI content blocker, and not a trivial DNS blacklist.
Agreed, the UK's DNS filtering is definitely a simple to defeat by anybody whack-a-mole (e.g. thepiratebay.org is blocked? Oh no! Let's just google for Pirate Bay and pick one of the many, many unblocked mirrors)
But the kind of DPI, forced blocking utilised by these middleboxes is certainly a step above that, to the point that most people will not be (say) using measures like a VPN to bypass the block.
There are easily measures that will cause the closure of your small business if used against you personally that would not stop A drug dealer let alone all drug dealing.
Because whacking even 5% of moles looking for ways around government[0] censorship is a unconscionable travesty.
0: eg, China, Iran, etc; if you don't think people getting caught is a problem then I question your basic human decency; if you don't think 5% of people getting caught is a problem then I question your sense of scale and/or ability to multiply numbers by other numbers.
> some ISPs might want to block highly illegal content
There is no way - not even a theoretical way - to allow blocking of illegal content (for any definition of illegal) that won't allow for blocking of any other arbitrary content. Censorship is binary. You can accept either none of it, or all of it.
At some level, everywhere has some form of censorship. For any country you could name, there are, or could easily be, content in any kind of media - books, audio, video, games, whatever, that is so abhorrent that it would either not be published, or would be shut down as soon as possible.
So if you say censorship is binary, it's already here, and has been here for ever. But I would guess that few believe that censorship is truly binary like you say.
Censorship is fine if it's opt in. I don't use facebook and censor myself from it. I opt in to use a pihole and adblocker. It filters many things I otherwise would see.
You can't often choose your ISP so this makes it extra important for censorship of any kind of to be opt in rather than forced.
> There is no way - not even a theoretical way - to allow blocking of illegal content (for any definition of illegal) that won't allow for blocking of any other arbitrary content.
Well, there is: take the person or body that ultimately determines whether content is illegal, and have them review each request and proposed response and decide whether to allow the content through to the requester.
For slightly better scalability, have that body review all content outside of any request-response cycle before it can be published and sign any approved content, then block any content they haven’t signed.
Somewhat more generally, as long as the specific blocking methodology is itself part of the definition of what content is legal, any blocking method can meet the standard of “allows blocking illegal content without allowing blocking of other content”, since any content blocked by the method is, ipso facto, illegal.
How do you propose to determine that a new, never-seen-before URL hosts child porn? In the US, viewing child porn is a strict-liability offense, meaning you are guilty of a serious felony just by looking at a page with the image(s) on it.
There are also civil and criminal liability concerns at the corporate level by assuming the responsibility for constructing and/or maintaining these filters.
ISPs should help LEAs find illegal content all day long, but developing this sort of blocking technology is not the same thing, and neither is using it. There are legitimate uses of it though, namely in employers' networks, for example.
This goes for any position with responsibility for and access to end user data. For instance: administering a mail server or hosted mail account for the company, dealing with the fall-out of security incidents and so on.
Tor Project closely cooperates with The Open Observatory of Network Interference. OONI provides a standardized testsuit that automatically probes a wide range of websites for evaluating the censorship status in different parts of the world, and volunteers can contribute data by running an instance.
But the objective of the project is only to identify governmental and ISP-wide censorships. And its capabilities for checking protocol-level censorship techniques are rather limited [0]. Perhaps we need a similar tool or fork for probing middleboxes and censorship in private networks.
This is pretty clever! The reset within airtel_103.224.212.222_fullhd720.com.pcap arrives with IP time-to-live of fifty-seven while the segment carrying synchronize | acknowledge flags arrived with a time-to-live of forty-four. So without any active probing, and some educated guesses around default IP time-to-live values @ 1<<[6..8] you could could conclude that the reset originated fourteen hops closer to the capture than other packets in same five-tuple defined "flow".
Agreed, it's flimsy. Certainly a bit more effort for them spoof it correctly though. Would need to watch traffic on the path back per flow to isolate the number of prior decrements to the TTL leading up to MitM, and then store that value until such time that it sees an SNI it cares about / it's time to generate a reset.
What reason could they have for blocking DDG? Is it easier to find pirated content there than on Google or something? That's my best guess. I can't imagine they'd block on behalf of a competitor or something.
You never know. Could be a mistake where they were trying to block a certain path due to some search result of copyright infringement, but ended up banning the domain itself. One can only guess.
Reddit and Github have previously been temporarily banned in India due to similar "mistakes"
Unless the site uses certificate pinning its possible to do a downgrade attack that forces browser off of HTTPS. The extension HTTPS Everywhere is a stopgap against this
things that do stuff like this can't, they try whatever tricks are possible to push javascript or redirects to send the client browser to something non https, on port 80
Heads up, that site was injected with a ton of malware/adware and redirects. Possibly their ad network got hosed, but that site doesn't seem safe unless you are locked down.
This is wrong. The ClientHello message is not encrypted in TLS 1.3, so, the client has to announce any extensions in plaintext. Thus the Great Firewall blocks connections which say they want to do encrypted SNI.
TLS 1.3 works fine in China, but if you use TLS 1.3 with the earlier proposed encrypted SNI draft it is blocked. The Great Firewall can't tell which name you actually wanted, but it can tell you're encrypting the SNI and block that.
With the currently proposed Encrypted Client Hello with a GREASE-style dummy ECH on all connections (so the "real" Hello is sometimes in an encrypted block and sometimes that encrypted block was just noise), China would still be able to choose to block all ECH-enabled connections since their presence is detectable. This would break everything, but China can choose to do that. What happens next is a policy question.
If you want to sneak past nation state snooping you need something else, that's not what TLS is for. The TOR project does not directly offer this either, but they can help you find out how to connect to TOR in a sneaky way if that's necessary for you.
If they block ESNI, it's likely all ESNI requests would be blocked, because they can't tell if its a blocked site or not. (it'd be possible they only apply ESNI-blocking against IPs associated with blocked sites, but that seems unlikely)
Agreed and since TLS 1.3 is still work in progress the chances are slim that you will find a blocked website that meets your criteria. Great article, very accessible. Thanks!
That RFC is marked as "PROPOSED STANDARD" which is why I saw it as work in progress but you're right, that seems to be the end of the road for RFC's (for example RFC 6455 December 2011 (websockets) is also marked as proposed standard but this is what everyone has implemented)
The IETF deliberately has no power whatsoever. Whether an IETF standards track proposal in fact becomes a standard everybody implements is entirely up to the implementers. This is in contrast to many standards development organisations (and indeed whether the IETF is even an organisation is open to doubt) which are government functions and can produce de jure standards you're obliged to implement or in the worst case force may be exercised against you by those with a monopoly on its use.
As a result IETF standards are only proposed and that is as you say "the end of the road".
> As a result IETF standards are only proposed and that is as you say "the end of the road".
Not really, after PROPOSED STANDARD there's INTERNET STANDARD, the STD series. For instance, IPv4/ICMPv4 (RFC 791/792) is STD 5, UDP (RFC 768) is STD 6, TCP (RFC 793) is STD 7, DNS is STD 13, and so on (STD 1 has the full list).
However, an IETF standard only reaches that level after it's been in use for a while; according to RFC 2026 (BCP 9), "A specification for which significant implementation and successful operational experience has been obtained may be elevated to the Internet Standard level. An Internet Standard (which may simply be referred to as a Standard) is characterized by a high degree of technical maturity and by a generally held belief that the specified protocol or service provides significant benefit to the Internet community."
They openly state that they supply ISPs with DPI hardware, I talked to them in person 15 years ago and they had no problem to admit that they also supply government institutions.
My job overlaps with microwave and millimeter wave RF engineering somewhat. R&S also has no qualms about selling high-end spectrum analysis equipment to authoritarian regimes. Probably done through middlemen. For instance, you can find the Iranian government using their equipment in Tehran to hunt down things they don't like.
To be fair, there's probably less than ten manufacturers of their category of spectrum analysis gear (for commercial/non-military use) in the world, and their stuff is top quality.
Interesting, so triangulating people forwarding "open" internet over consumer-grade microwave (like Ubiquiti or similar)?
I assume they can't do much about Toosheh since it's "read only", multiplexed with legitimate TV channels on the same transponder, and uplinked from the UAE.
Things that transmit generally, in all sorts of bands, lots of countries where the government holds an armed monopoly on connections to the outside world.
Try setting up an independent two-way satellite based C or Ku band earth station in Ethiopia, offer service to your neighbours and armed men will come to dismantle it.
Commercial spectrum analysis tools are an essential and important things in the hands of network engineers, but also a tool to crack down on anything that transmits that an authoritarian regime doesn't like.
Not with perfect accuracy, but the local oscillator of a receiver (which is mixed with the signal from the antenna to produce an intermediate frequency which is further processed) leaks back out the antenna a little bit.
In the case of an FM radio, this means you can tell what station a given receiver is tuned to, if you make certain assumptions about common LO/IF frequencies. There was a company monetizing this called "Mobiltrak", but I haven't seen much about them lately.
In the satellite case, there's the IF of the LNB itself, which leaks fairly loudly out the feedhorn, but it only tells you which band they're tuned to. There's likely a second IF used in the IRD, which should be much fainter from outside, but if you could recover that, you could tell which transponder is being demodulated.
That still wouldn't tell you if the Toosheh packets are being saved, but if the program they're muxed with isn't particularly popular, it would be a strong hint.
This is not about the block a month ago. This is a new block.
When it was blocked a month ago, there was no notice. Now, there is a notice that it is a TRAI order. usually seen on sites that the govt themselves ask to block (piratebay, torrentz.eu, etc)
I guess its only limited to some regions. I'm using Airtel for my home connection as well mobile and get redirected to https everytime I visit the http version of ddg on both connections.
Airtel is so big they use roaming for their customers. If you have a SIM from Bengaluru and go to Delhi you'll see the little R indicator. That suggests the Airtel business in each state manages at least parts of their network independently. And so the MitMs could be deployed non-uniformly.
India used to be and still is split into several telecom regions with different spectrum leasing, operations and governance.
Until around 2009-10, when you are traveling out of state, you had to pay roaming charges. Worse used to be metro cities within their own states as they used to be different telecom circles. I used to pay roaming charges when going to Chennai from rest of Tamil Nadu. Even the operators were different sometimes. E.g. there was no Hutch (now Vodafone) originally in rest of Tamil Nadu and they operated only in Chennai. Similarly RPG (later Aircel which went bankrupt couple of years back) had 2 networks - RPG in Chennai and Aircel in rest of Tamil Nadu. It used to be a mess.
No operator had pan-India operation as every small operator had their own fiefdoms and the big operators like Airtel used to pay roaming charges to those operators for their subscribers to get signal.
I wish the world would adopt tcpcrypt. SSL provides two services, encryption, and authentication.
tcpcrypt just does the encryption part. Once the connection is established, userspace on both sides can invoke an ioctl that provides a session nonce. If the nonce matches on both sides, the connection is not man-in-the-middled. It’s easy to confirm the nonce matches: Both sides sign it and send it to the other party.
This has two big advantages:
(1) operating systems can opportunistically encrypt traffic for unmodified legacy applications and network protocols, allowing the endpoints to detect mass surveillance without requiring any certificates.
(2) Like newer versions of SSL, it encrypts the information currently sent in cleartext in the SNI, preventing the type of censorship in the article.
A) some kind of secret that only the server knows that the client can verify in order to ensure it's not trading nonces with the MITM.
B) A way for the client to ensure that the nonce isn't being passed through a second tcpcrypt session between the MITM and the server with the connection being in cleartext between the 2 tcpcrypt streams.
Currently the best supported method of implementing both A and B is certificates, which means you may as well use TLS.
Client establishes a tcpcrypt session with what it thinks is Real Web Server but is actually Evil Middlebox replaying the request to the server and the response back to the client.
Yeah, I'm not sure what the parent was getting at separating them out since from the clients perspective they're the same. I guess they mean that getting a tcpcrypt connection on your server isn't a guarantee that there isn't a middlebox either.
We have SNI because it's needed to support virtual hosting, which we really only need because IPv4 addresses are scarce. If we could ever get to IPv6, SNI could be retired completely.
I wonder what they intend to do (other than just blocking entire IP ranges of non-Indian hosting providers, which I would not be surprised by) when things increasingly move to TLS1.3 with ESNI.
The Great Firewall does not block TLS 1.3. You may have seen headlines which claim it does, but they're based on a report that actually says it doesn't. Remember journalists probably know even less than you do about most things they write about!
In this case the report says the Great Firewall was determined to block the following specific combination:
* A ClientHello for TLS 1.3 that
* Includes the 0xffce extension value (used for experimenting with an earlier SNI draft)
If you add a 0xffce extension full of random noise, the Great Firewall blocks it. If you use the same random noise but pick a different extension value (do not do this in production code - those aren't for your meddling!) the Great Firewall doesn't interfere at all.
We have yet to discover what happens if a big bang release of Encrypted Client Hello (the current iteration of the encrypted SNI work) just deluges the Great Firewall with ECH connections. But we do know TLS 1.3 has been used successfully for years from China.
You also mention this idea that it would "force hosts to drop down to 1.2 for connections".
It is hard to tell what you intended here, it would of course be possible to force the humans using a computer to downgrade, or to disable encryption, or to cease using a computer altogether, perhaps you could put a gun to their heads for example.
But TLS 1.3 has an anti-downgrade design. A [edited to add] modern TLS 1.3 capable web browser which connects to a TLS 1.3 capable web site but finds that the connection has been negotiated as TLS 1.2 instead will reject the connection as clearly under attack, you cannot reach that site until the problem is remedied. I think you would notice if all TLS 1.3 capable sites (about a third of popular sites) suddenly did not work from China, even the Chinese government might struggle to silence such confusion and dismay from their people.
While all of the above is correct, it doesn't stop the GFW from implementing per flow based DPI that drops traffic, or throttles it to a throughput that is so slow as to be unusable, based on detection of consistent encrypted flows between an IP that is outside of China, and domestically within China.
The one thing TLS1.3 with ESNI is not is hard to detect. It's a consistent traffic pattern if you throw a sufficient amount of CPU and RAM resources at doing DPI on each and every user's flows.
In an ordinary non censored ISP environment the ratio at which you export netflow data to a collector adjacent to the router is quite low. And not a great deal of CPU and RAM resources are put into doing detailed analysis of it, other than for basic things like figuring out who you should be peering with that you aren't peering already, and identifying percentages of traffic patterns (eg: at 10pm every night we see this much traffic from our on-net locally hosted netflix cache boxes going towards the residential GPON customers).
If you are a Chinese entity with access to the router-design people at Huawei and ZTE, and sufficient motivation to do so, there's no reason why you couldn't crank up the ratio greatly and (on a middle mile and per POP basis) export netflow by a dedicated 100Gbps link to a set of directly-adjacent high performance x86-64 servers, running custom flow analysis and DPI inspection software.
They're already doing massive flow sampling at the GFW complexes today. They can police down to individual flows if they want to. They scale wide by distributing out traffic based upon src or dst IP.
Theoretically, lets say am in India, using a tor browser, or an opera browser wit inbuilt VPN, would I see different results? my point is to see whether a VPN of sort can circumvent, in that case, what if all those browsers decides to create inbuilt vpn (for connection origins)..
A VPN would prevent SNI snooping since traffic over it is encrypted. Of course, the ISP could block the TLS handshake between you and the VPN server if that's how your VPN functions.
I think they're talking about the blog itself not being configured to use HTTPS. I had to add an exception in my HTTPS everywhere extension to read the article.
This is completely off-topic but the strike-through on links had me confused for a good few minutes. I was not clicking on those because I thought the link were not valid today and hence it has strike-through.
Using the TTL, we figured out that the censor kicks in at the kth hop. This kth box belonged to Airtel.
If a box was censoring after Airtel, we would have received a clean response (ICMP timeout) at hop k as well. Of course, the TTL itself can change during the run, but that wouldn't happen for so many cases :)
On the scale of a personal blog, its approximately 0 minutes per year to maintain HTTPS certificates using Lets Encrypt with something like certbot (or use Caddy, which handles it on it own).
Yes, you gotta automate a whole bunch of things if you need HTTPS, you have to update the protocols every few years, certificates as often as every few months, OpenSSL versions on a moment's notice.
Or, you could decide to just go HTTP-only for your blog, and never bother doing any of the above, never worry about any automation failing for any reason, never worry about any expired or revoked certificates, never worry about the extra compatibility issues that TLS brings. There's no benefit for HTTPS for a personal blog. It's only there to restrict the access, increase attack surface, and cause compatibility issues.
It is regretful that organizations like NANOG, RIPE, ARIN, APNIC and others do not take a stronger stance (on a global geopolitical level) against censorship of the Internet, and attempts to create walled gardens and national firewalls.
The people who possess the equivalent of 'enable'/'configure' on the core routers of gigantic ASNs have real power to refuse to further harm the Internet.
We have seen a number of discussions on HN about ethics in software engineering. A quick search for posts with 'ethics' in the subject line turns up a number of things. In my opinion, ethics in the ISP, telecom and network engineering industry is equally important.