Shadowsocks being the most well-known example. From a cryptographic perspective, it was a big step backwards. Since Diffie-Hellman handshakes were subjected to traffic classifiers, these tools only used symmetric encryption with fixed keys. Their encryption protocols were ad-hoc, and not cryptographically robust. While it was a matter of fact that nobody could break a simple AES-CBC encryption, nobody would trust these tools for one's confidential data as well (for example, AEAD was unsupported for many years). But since the goal was bypassing censorship, not secrecy, they became extremely popular. It was not seen as an major issue, since the widespread use of HTTPS offered robust secrecy. DNS encryption was still essential (usually the SOCKS-5 interface was provided by these tools, SOCKS-5 can be configured to pass the original domain name to the proxy, the proxy can resolve the names inside its encrypted connection), but became less useful when used on its own, since the IP blocklist was huge by the time.

The landscape of the Internet has changed dramatically since 2013 as well. The universal adoption of HTTPS eventually rendered all keyword-based inspection useless. A few sites were considered too large to block, including Amazon AWS and GitHub. One side of the battle started becoming a mutual assured destruction game - either allowing people to exploit a large platform to publish uncensored material, or blocking the platform altogether and creating economic damages. I am confident that the MAD game will continue to play out, however, Russia's response to AWS domain fronting showed this strategy could fail if major platforms don't want to cooperate, it was a bit worrying, at least. But anyway, encrypting SNIs should be the next step.

But I digressed, back to Shadowsocks, et al, since the state was eliminated (pun intended), all one could see was encrypted raw TCP packets, there was no reliable way for the firewall to classify Shadowsocks-like tools for many years (until recently, possibly by exploiting cryptographic-related issues, but we are not sure how successful it is). But the censorship system started getting weirder and weirder - sometimes, connections break without any apparent reason at all, sometimes data rate was extremely low, sometimes a few IPs were blocked mysteriously, and so on, but life kept going on. There were several possible hypotheses, one was that the traffic classifiers were getting more and more functionalities, and occasionally they could hit something. Another was that the TCP RST was sent in a probabilistic manner to suspected endpoints to degrade reliability. The only thing that could be confirmed was the significantly increased use of QoS by the ISPs, so that all unknown protocols would be classified as "low priority", degrading the reliability of all anti-censorship tools. At this point, bad connectivity and censorship was indistinguishable.

It's safe to say, that at this point, nobody ever understands how the Great Firewall of China work anymore. This is the end of our story.

For simplicity, I skipped many less used techniques, such as Tor's domain fronting, or CDN-based circumvention, or obfsproxy4 that featured Diffie-Hellman keys indistinguishable from random strings, and possibly others. I'm well-aware of them. But it's expected that, unless everything is encrypted and all infoleak is plugged (then, we will start playing the mutual assured destruction game), all these tools are doing is an endless cat-and-mouse game.

Developers of anti-censorship tools need to consider countermeasures based on what China is currently doing. So that when the same techniques used by China are implemented by their own ISPs in the future, they are always prepared to act.

Fantastic breakdown on the recent history of censorship in China, thanks for sharing it.

You mentioned that for many of these efforts bypassing censorship trumped secrecy concerns. Is this still the case?

If I were a citizen regularly bypassing censorship of an authoritarian government, I’d be concerned for my safety if it was well documented that I regularly accessed censored material.

From what I gather, the regime doesn't really intend to arrest anybody who simply regularly accesses western websites. Some big corps also have their special VPN channels to access foreign websites so that they can do business normally. Hell, even the foreign ministry spokesperson posts regularly on Twitter. What they want is to stop this floodgate of information being opened to the common mass, that's when things could get problematic.

People are arrested for producing things that are deemed potentially destabilizing for the regime/country, but nobody as far as I know ever got arrested for accessing blocked materials.

Of course, if you are also actively producing content it would be much wiser to camouflage your identity much better, if you can. That's when the secrecy becomes a major concern.

> You mentioned that for many of these efforts bypassing censorship trumped secrecy concerns. Is this still the case?

Yes, it's still the case, but how bad is a matter of debate.

To make it specific, we can use two criteria to evaluate anti-censorship circumvention tools: (a) How cryptographically robust it is? (secrecy) and (b) How well they can avoid detection? (visibility) The situation is complicated, since they are related but independent.

First, OpenVPN has good secrecy, but high visibility, since it's handshake is obvious, and it even led to a complete block. Second, everything that exploits a bug in the DPI system will have circumvention capabilities, but bad secrecy and high visibility - ultimately, the fact that a TCP connection has been created cannot be hidden, and the fact that you are bypassing censorship will be clear - on the other hand, high visibility doesn't necessarily mean it can be blocked (fixing such a bug can be difficult). [0] Third, a protocol with cryptographic flaws (such as not providing good protection against ciphertext modification) can otherwise have low (or high) visibility, but allows attackers to compromise infosec in some ways. Finally, Tor has circumvention capabilities, excellent secrecy, but high visibility - it's anonymity depends on its large anonymity set, not hiding the fact that someone is using it (which is unpractical), and its network is completely open.

Primarily, my personal concern is whether the circumvention tools are cryptographically robust, so that my secrecy won't be compromised when I browse a HTTP website (The NSA can always wiretap at the exit node, but at least it should not be vulnerable at the entry point). I don't trust these tools, if cryptographers kept discovering implementation flaws from established protocols, why should I trust a tool with ad-hoc crypto? For example, Shadowsocks did not have any forms of forward secrecy, if someone is recording all the outgoing traffic, and later take control over my computer, using a single key allows the decryption of everything. On the other hand, some people argue that flaws may exist, but exploitable ones are rare. But still, I think it's a bad practice to lower the standard of secrecy. If I have to use them, I'll run an additional layer of TLS on top of these tools, so that my connection will always be as secure as TLS, while the outer layer provides circumvention. Fortunately, most people are protected anyway by HTTPS.

> If I were a citizen regularly bypassing censorship of an authoritarian government, I’d be concerned for my safety if it was well documented that I regularly accessed censored material.

If your goal is totally avoid detection of using any circumvention tool at all, it's going to be much harder. Many privacy tools are developed to exercise one's rights to privacy, but they are not designed to avoid detection. On the other hand, the same tools are usually promoted for citizens in oppressive regimes. This can be dangerous. For example, a full-disk encryption software that includes clickable links to its official website, with automatic update, what can possibly go wrong? If the regime is authoritarian enough, the regime can simply make a list of all users that have accessible to these servers before and hunt them down.

A huge amount of work needs to be done to fix this problem. However, if you are in China, it's not that dangerous. In the authoritarianism spectrum, there are Kazakhstan, Iran, China, Russia, and others. However, China is nowhere close to the extreme. Being economically open at large, the censorship of information in China cannot, and was never meant to prevent all forms of access. The purpose is merely to increase the costs from doing so. In fact, criticisms of the government in domestic social media are sometimes tolerated, often the censorship only kicks in when it became popular.

By installing an Internet censorship system in China, the consequences are: (a) Most people are not interested in accessing block websites, at not in a regular basis, even if methods are available. (b) Accessing information doesn't necessarily mean a change of point-of-view, especially when the opinions are completely different from one's education, personal experience, or worldview. (c) Foreign platforms cannot gain any significant influence, even if they are accessed to many. For example, the Chinese Twitter community is an interesting place (if one digs deeper below the political flamewars at the surface), you could see people coming from the entire political spectrum. There are even jokes, such as "Twitter - the future of governance in China", but they are irrelevant in the big picture. (d) IT workers are required to use Google and other blocked sites for doing one's job.

Under this background, regularly bypassing censorship in China just for web browsing is perfectly safe [1]. If you want the best invisibility, I recommend you to use the most popular VPN service used by the highest number of people, and run your own encrypted tunnel inside that. The downside is that these services are too popular to be stable, most IT workers still prefer to use a personal hosting service.

[0] Due to the increased centralization of the web, changes are expected. With SNI encryption, if all the censor can see is a connection to an unknown website on CloudFlare's server, it's less of a threat. But different opinions exist, one says the pressure of censoring everything vs. not to censor can lead to an decrease of censorship or a faster overthrow of the censorship system , but others say the censorship/anti-censorship forces are in a dynamic balance, the introduction of centralized services with SNI encryption can actually break the balance. What used to be a slow censorship progression that needed in 5 years can speed up to 2 years, creating an accelerated and more aggressive censorship, and ends up to be a net negative everyone. Whether it is the case is yet to be seen.

[1] Unless you are in regions like Xinjiang, where separatist conflicts are seen as a threat, and that the censorship has extra objectives.

Thanks for this summary. The firewall has been a lot stricter recently and it's been a real pain in the ass, even for legitimate things. I can only speculate they are using deep learning type tools now to do their blocking

> I can only speculate they are using deep learning type tools now to do their blocking

It needs careful justification before making such a statement - the censorship system has a serious constraint on computational costs - it needs to operate on the stream of the entire outgoing international traffic, and to make a decision in real-time (or for back-analysis). We are talking about many terabytes per seconds of traffic, any censorship tools that have a high computational costs cannot be deployed for such a purpose, even if it runs okay on a single PC. Also, a high false-positive rate is not acceptable, as it will create massive service disruption and practically useless.

Unlike the case for SOCKS5-over-SSH, HTTPS-over-HTTPS, or VPN handshakes, which can be detected by relatively simpler rules, most deep learning tools required excessive CPU time, so it's unlikely that complex deep learning algorithms are being used, at least not the category that costs the highest CPU time (anything with "AI").

Given these constraints, the algorithms available to the censorship system is rather limited, it seems. What types of algorithms are being used, then? Unfortunately, nobody can answer this question. This is the fundamental question people are facing today. 10 years ago, every sysadmin in China knew the censorship system works, but today, the system has became completely opaque.

Great post, thank you.

I agree with the other posters -- fascinating, detailed info. These posts should be promoted to their own HN article/discussion...

Were they doing that full page text matching in an ASIC too?! Doesn't that basically involve writing a simple parser also? Else what prevents things like usage of Google analytics/fonts etc from triggering a match and blocking?

> Else what prevents things like usage of Google analytics/fonts etc from triggering a match and blocking?

The blocking was/is complementary. Usually, domain names themselves were blocked by DNS poisoning (or IP blocking if it escalated), domains themselves (or the names of the websites) did not appear in the keyword blocklist. A link to Google Analytics or Facebook button could stuck the webpage from loading properly until a timeout, but merely mentioning or linking a domain name would not trigger a keyword match of the page itself.

The intention of keyword matching was to allow partial access while still blocking unwanted content. Usually, only the most politically unwanted keyword entered the keyword list. For example, Wikipedia could be accessed normally, but as soon as "a word that should not be named" appeared in the webpage, the connection would be reset immediately. An interesting phenomenon was, sometimes the page could partially load and stopped exactly before the forbidden word. And since the censorship system worded on mirrored traffic, sometimes a slight processing delay allowed the full page to load before the RST was received, it would be a "I'm feeling lucky moment".

Anyway, there was how the system worked before 2010. The extensive use of HTTPS rendered it useless, and it appeared that some forms of keyword filtering has already been lifted, since it's already a pointless exercise.

For quite some time after keyword matching became ineffectively, DNS poisoning remained the only form of censorship for many unwanted but not significant websites, for example, Hacker News. But recently, SNI matching was implemented.