How does XTLS REALITY break through the whitelist? REALITY source code analysis

Beyond the security of normal TLS for anti-censorship? No need to buy domain names and configure TLS? How does the "Magic Tech" REALITY achieve these?

Table of contents

(All the content below is translated from Chinese by Google Translate ↓)
(Please feel free to make your own comments at the translation.)

ℹ️Foreword

XTLS/Xray-core, also known as Project X is an open source censorship circumvention tool. It is well-known in the Chinese anti-censorship field for its novel, avant-garde and practical conceptual technologies ~~ and the project creator & maintainer RPRX who once mysteriously disappeared~~. These technologies include VLESS, XTLS-Vision, XUDP… There is always one you have heard of or used.

Since some areas in mainland China began to deploy a new censorship strategy on a large scale – SNI whitelist, all TLS-based circumvention tools before the appearance of REALITY and ShadowTLS, whether directly connected or connected through transit or CDN, became unavailable in these areas overnight.

(It is known that China Mobile in Quanzhou, Fujian Province, mainland China is the first ISP in mainland China to deploy this censorship strategy)

The circumvention tool ShadowTLS developed by ihciah previously received widespread attention. However, ShadowTLS was still in version v1 at that time, with an incomplete code base and weak censorship resistance. Later, REALITY developed by RPRX also had the ability to circumvent the SNI whitelist censorship strategy, and its high integration with the mature circumvention tool Xray-core, ~~ perhaps there was also the surprise brought to users by the return of RPRX~~, which attracted a lot of attention in the field of Chinese anti-censorship.

So how does REALITY circumvent this censorship strategy? How to understand its details from a technical perspective? These two questions will be the focus of this article. By interpreting REALITY’s source code, readers can sort out the specific implementation of REALITY.

(PS: ShadowTLS has three incompatible versions: v1, v2, and v3. v2 fixes the active detection vulnerability in v1, see the paper Chasing Shadows: A security analysis of the ShadowTLS proxy; and v3 is designed to make the minimum necessary modifications to one of the popular TLS library implementations, Rustls, so that the server has the ability to respond normally to TLS Alert (to deal with MITM traffic tampering), which is more hidden than v2; of course, this is all later, and there may be an article about ShadowTLS in the future, so I won’t go into details about it here)

👀 What is “SNI whitelist”? What is the relationship between SNI and TLS?

You may know that the widely used application layer security protocol, the cornerstone of HTTPS, the TLS protocol, has its own “handshake process” when initiating a connection. What? You don’t know what a “handshake” is? You can take a look at this paragraph in my previous article “Traffic classification and identification of encryption proxy? A preliminary exploration of TLS in Any”.

TLS has been a “hybrid encryption system” since its first version was designed. This means that TLS uses both asymmetric and symmetric encryption algorithms. Symmetric encryption algorithms require both parties to hold exactly the same key, and the encryption and decryption overhead is low; while asymmetric encryption only requires both parties to exchange public keys in their respective key pairs, but asymmetric encryption requires verification that the public key has not been replaced or tampered with when exchanging public keys, which gave rise to the digital certificate mechanism. And asymmetric encryption and decryption overhead is large. Therefore, TLS uses asymmetric encryption to transmit keys used for symmetric encryption, and in order to exchange public keys used for asymmetric encryption, the TLS handshake mechanism was born.

Before understanding the SNI whitelist censorship strategy, let’s take a look at the handshake process of a common TLS1.3 connection without ECH:

(Here is a picture from Cloudflare Blog, copyright belongs to Cloudflare lnc..)

First, the client initiates a TLS connection. After correctly completing the TCP handshake, the client generates a key pair and sends a TLS Client Hello message to the server through the open TCP connection: All handshake parameters of the client, including various extension fields (extensions) and key_share (the public key in the key pair just generated by the client). These parameters whether sensitive or not are sent in the TLS Client Hello.

Secondly, the server uses the key exchange algorithm allowed in TLS Client Hello to generate a key pair and sends TLS Server Hello: The Server Hello message of TLS1.3 is different from TLS1.2. It only contains all insensitive handshake parameters of the server, such as key_share (the public key in the key pair just generated by the server).

After receiving the TLS Server Hello, the client extracts the key_share (the public key in the key pair just generated by the server) and inputs it into the Diffle-Hellman key exchange function together with the private key in the key pair originally generated by the client.

(The Diffle-Hellman key exchange algorithm is referred to as the DH algorithm. The DHE algorithm is a DH variant that achieves forward security by rotating keys. The ECDHE algorithm is a DHE variant based on elliptic curves. As of the time of this article, the latest version of crypto/tls only supports the ECDHE key exchange algorithm when processing TLS1.3 public key exchange.)

DH and its derivative algorithms have a common property: exchanging the public or private keys of two key pairs that meet the requirements of the algorithm, and then inputting the two exchanged key pairs into the algorithm, you will definitely get exactly the same value. That is: generate key pair A (including public key pub_A and private key sec_A) and key pair B (including public key pub_B and private key sec_B) that meet the requirements of the algorithm, and input (pub_A, sec_B) into the algorithm. The value obtained must be exactly the same as the value obtained by inputting (pub_B, sec_A) into the algorithm.

Here, the client uses the public key from the server and its own private key to generate a key by inputting the DH algorithm. The generated key must be the same as the key calculated by the server using the public key from the client and its own private key. The key generated here is called preMasterKey, which is only used to encrypt and decrypt the next handshake message.

Next, the server uses preMasterKey to encrypt sensitive handshake parameters that have not yet been sent, including the digital certificate used to verify the server’s identity (because the digital certificate contains the corresponding domain name/IP information), encapsulates it into a TLS Application Data message (also called Encrypted Extensions, but this name is only visible after decryption), and attaches it to the Change Cipher Spec before sending it to the client. After receiving the Change Cipher Spec, the client uses preMasterKey to decrypt the TLS Application Data attached to it, verifies the server’s identity (that is, proves that the server holds a specific domain name/IP) through the digital certificate in it, extracts the handshake parameters, constructs a TLS Finished message, and encrypts it with preMasterKey and sends it to the server to indicate that the TLS handshake is successfully completed. At the same time, the client inputs the complete handshake parameters extracted from the additional messages after TLS Server Hello and Change Cipher Spec into the DH algorithm to calculate the MasterKey, which is used to encrypt and decrypt all messages (i.e. the following TLS Application Data packets). The server also calculates the MasterKey with the same logic after receiving the TLS Finished message.

(In the actual TLS library implementation, the client often sends the first TLS Application Data containing application data before the server uses the TCP ACK packet to respond to TLS Finished to reduce latency. The last “GET /index.html HTTP/1.1” in the diagram above refers to this situation.)

Among them, at the beginning of the TLS handshake, the client uses the SNI extension (Server Name Indicator) in the TLS Client Hello message to indicate to the server the website it wants to visit. This is critical for the modern internet, as it is now common for many origin servers to sit behind a single TLS server, such as a content delivery network (CDN). The server uses SNI to determine who will authenticate the connection: without it, there is no way to know which website’s TLS certificate to present to the client.

The key to censors implementing SNI-based censorship strategies is that SNI is plaintext visible to any route in the network (the endpoints that traffic passes through when it is transmitted), and therefore will reveal the origin server that the client wants to connect to. At the same time, censors block TLS traffic that is not trusted by the censor by controlling all traffic egress routes and limiting the values ​​of the SNI extension in the TLS Client Hello that are allowed to pass. We call this censorship strategy “SNI whitelisting”.

Questions: Why is TLS1.3 chosen as an example instead of TLS1.2? Why is the ECH feature of TLS1.3 not enabled here?

Answers:

  1. TLS1.3 is largely a simplified version of TLS1.2. It fixes many security vulnerabilities in the design of the previous generation of protocols, is designed to be more concise and easy to use, and has been deployed on an ever-increasing scale since the first official version was released in 2018. Secondly, the REALITY server implemented by XTLS does not support REALITY clients to use TLS versions other than TLS1.3 to connect to transmit evasion traffic.

  2. As of the time of this article, the REALITY implemented by XTLS does not enable the ECH feature of TLS1.3 by default. Well, there is actually no relevant option in the configuration file. Moreover, the core of REALITY’s design is to “perform” a valid TLS handshake using allowed SNI extended values to the censor (middleman), and transmit evasion traffic through the TLS channel opened by the handshake.

🤔 Let’s think about how to change SNI

  1. When the client of the circumvention tool initiates a TLS handshake to the server, directly modify the value of the SNI extension of TLS Client Hello.

This method is simple and crude, but it is obvious that it does not have good censorship resistance and will destroy the TLS authentication mechanism. The digital certificate used to verify the identity of the server during the TLS handshake needs to be signed by the upper CA (certificate authority) trusted by the client using a private key, including certificate attachment information and domain name information, so it is impossible to modify a valid certificate from another website while maintaining its validity without the upper CA signing the modified certificate. In fact, in this case, even the owner of the certificate cannot modify the valid digital certificate he holds while maintaining its validity.

The value of the SNI extension of TLS Client Hello must be the same as the domain name in the extension information of the valid digital certificate in the TLS Server Hello from the server.

The domain name in the extension information of the valid digital certificate in the TLS Server Hello of the self-server is consistent. (Not necessarily exactly the same. There is a type of “wildcard certificate” in the digital certificate used for TLS server authentication. This type of certificate can verify the identity of all subdomains under a domain name without issuing certificates for each subdomain.)

Similarly, the reviewer can record the value of the SNI extension of the TLS Client Hello in the TLS traffic passing through, use this value to construct the TLS Client Hello, and send it to the corresponding server to check the validity of the digital certificate in the returned TLS Server Hello, and whether the domain name in the certificate extension information is consistent with the original SNI value. If any of them does not match, the original traffic can be considered as circumvention traffic.

  1. From 1., we know that if you want to modify the SNI of TLS Client Hello, you may need to modify the digital certificate.

Of course, this is just a joke. The mathematical properties of digital signatures determine that it cannot be tampered with while maintaining its validity under current computing power conditions.

Wait, did we mention something in the previous section? Let’s look at the diagram again… The server’s digital certificate in the TLS1.3 handshake is sent encrypted, right? Therefore, in 1., the censor cannot passively observe and collect the digital certificate in the TLS1.3 handshake initiated by the user. The censor must construct the TLS Client Hello using the original SNI value to obtain the previous digital certificate to achieve the purpose of censorship. Although digital certificates cannot be tampered with, if we can distinguish between circumventing clients and censors, can we provide them with different digital certificates?

 🔍 Try to dive into the code ocean of REALITY?

In the previous section, we proposed the circumvention strategy of “providing different digital certificates by distinguishing TLS client types”, which is one of the key design of REALITY. But how to distinguish? How to negotiate the key preMasterKey used for encrypted handshake? In this section, we try to go deep into the source code of REALITY, combine the knowledge of the first section, and analyze the technical details of REALITY’s implementation of this circumvention strategy.

Please don’t leave yet! Whether you are a programming novice or a developer, this section will try to explain the functions of all the source codes in this section in easy-to-understand natural language. Please always remember: Understanding the program is to understand the design ideas, not to understand the programming language knowledge.

All server source codes appearing here are based on the 079d0bd commit version of the main branch in the Github code library XTLS/REALITY; All client source codes appearing here are based on the 4c9e4b9 commit version of the main branch in the Github code library XTLS/Xray-core.

💻Client

Since the design of Xray-core is that the client initiates the proxy connection, let’s start with the client’s reality package 1: (1 package, package, is a unit for organizing program functions. In Go, only one package can exist in the same directory (excluding subdirectories))

In Xray-core, all packages responsible for encapsulating and transmitting proxy traffic are placed in the transport/internet directory, and the reality package is no exception:

1xray-core/
2|-- transport/
3| |-- internet/
4| | |-- reality/
5| | | |-- config.go
6| | | |-- config.pb.go
7| | | |-- config.proto
8| | | |-- reality.go

The three files config.proto, config.pb.go, and config.go in this directory start from The main functions are to define the protobuf3 field format passed into the package, define the public methods of struct2 reality.Config3,4 for storing configurations, and fill the public fields5 of reality.Config completely. reality.Config mainly stores configurations for REALITY. (2 struct, a unit for organizing data in Go, an object in a broad sense; 3 method, refers to a function bound to a specific struct for reading, writing/processing all fields of the struct; 4 public method, refers to a method that can be called by other packages; 5 public field, refers to a field that can be accessed and modified by other packages)

The key to initiating a REALITY connection is in reality.go, where func6 UClient is called when other packages initiate a REALITY connection, and is the most important functional entry point for the client part of the package:

1// Xray-core/transport/internet/reality/reality.go#L106
2func UClient(c net.Conn, config *Config, ctx context.Context, dest net.Destination) (net.Conn, error)

(6 func, function, a unit that implements a specific subdivision function)

As shown above, func UClient accepts the readable and writable network stream interface c from the net package, config for storing REALITY configuration, notification channel ctx for timeout control, and structure dest indicating the destination.

One of the key functions of func UClient is to initialize the struct UConn defined in this package:

 1// Xray-core/transport/internet/reality/reality.go#L64-L69
 2// Struct fields definition of UConn
 3type UConn struct {
 4	*utls.UConn
 5	ServerName string
 6	AuthKey    []byte
 7	Verified   bool
 8}
 9
10// Xray-core/transport/internet/reality/reality.go#L108-L115
11// uConn is assigned to the UConn instance, utlsConfig is assigned to the utls.Config instance (configuration for utls package)
12uConn := &UConn{}
13utlsConfig := &utls.Config{
14	VerifyPeerCertificate:  uConn.VerifyPeerCertificate,
15	ServerName:             config.ServerName,
16	InsecureSkipVerify:     true,
17	SessionTicketsDisabled: true,
18	KeyLogWriter:           KeyLogWriterFromConfig(config),
19}
20
21// Xray-core/transport/internet/reality/reality.go#L124
22// Initialize utls.UClient and assign it to the first field of uConn
23uConn.UConn = utls.UClient(c, utlsConfig, *fingerprint)

The first field of UConn is an anonymous field 7, of type pointer to the UConn instance from the utls8 package. This code configures and initializes the uConn used to initiate and process the TLS handshake based on parameters such as Servername(SNI), TLS Client Hello fingerprint identification countermeasure, and func VerifyPeerCertificate for verifying the validity of the server certificate and the server identity.

(7Anonymous fields, i.e. fields with the same name as the corresponding type by default; 8 utls, a variant of the go standard library “crypto/tls”, provides TLS Client Hello fingerprinting, full access to TLS handshakes, Fake Session Tickets, etc. for anti-censorship purposes)

Then we come to the key point: REALITY clients use the Session ID field space in TLS Client Hello to covertly mark the client so that the server can distinguish between censors and legitimate REALITY clients. The Session ID field was originally used for the 0-RTT session resumption mechanism of TLS1.2. Although TLS1.3 switched to the session resumption mechanism based on PSK (Pre-shared Key), in order to maintain compatibility with TLS1.2 as much as possible, the Session ID field was retained while TLS1.3 was deprecated. Therefore, the Session ID used for each TLS1.3 connection should be randomly generated.

(Xray-core also defines a block-level scope for the following client code.)

 1// Xray-core/transport/internet/reality/reality.go#L126-L135
 2// Generate the default ClientHello and provide it to uConn
 3// In this step, the client's TLS key pair is also generated
 4uConn.BuildHandshakeState()
 5// Assign the pointer of the generated default ClientHello to hello
 6// (Assigning a pointer to hello means that modifying hello means modifying the original data)
 7hello := uConn.HandshakeState.Hello
 8// Assign a 32-byte empty slice (dynamic array) to SessionId
 9// And fill it to the 40th-71th bytes of ClientHello, occupying the unit position
10hello.SessionId = make([]byte, 32)
11copy(hello.Raw[39:], hello.SessionId)
12// Fill the version number x.y.z of Xray-core into the 1st-3rd bytes of SessionId
13hello.SessionId[0] = core.Version_x
14hello.SessionId[1] = core.Version_y
15hello.SessionId[2] = core.Version_z
16hello.SessionId[3] = 0 // Fill 0 to the 4th byte
17// Fill the current Unix timestamp into the 5th-8th bytes of SessionId
18binary.BigEndian.PutUint32(hello.SessionId[4:], uint32(time.Now().Unix()))
19// Fill shortId from the 9th byte of SessionId
20copy(hello.SessionId[8:], config.ShortId)
21
22// Xray-core/transport/internet/reality/reality.go#L139
23// Convert the REALITY public key to a usable public key object
24publicKey, err := ecdh.X25519().NewPublicKey(config.PublicKey)
25
26// Xray-core/transport/internet/reality/reality.go#L143
27// Use the x25519 private key and REALITY public key in the client TLS key pair generated by BuildHandshakeState() to enter the ECDH algorithm to calculate the shared key.
28// Input the key into HKDF (key derivation function based on HMAC) to calculate preMasterKey.
29uConn.AuthKey, _ = uConn.HandshakeState.State13.EcdheKey.ECDH(publicKey)
30// Xray-core/transport/internet/reality/reality.go#L147-L149
31if _, err := hkdf.New(sha256.New, uConn.AuthKey, hello.Random[:20], []byte("REALITY")).Read(uConn.AuthKey); err != nil {
32			return nil, err
33}
34
35// Xray-core/transport/internet/reality/reality.go#L150-L156
36// Select AEAD algorithm: AES-GCM or Chacha20-Poly1305.
37// AEAD algorithm ensures the security, integrity and anti-replay of ciphertext data,
38// and also provides integrity guarantee for additional data.
39var aead cipher.AEAD
40if aesgcmPreferred(hello.CipherSuites) {
41	block, _ := aes.NewCipher(uConn.AuthKey)
42			aead, _ = cipher.NewGCM(block)
43} else {
44	aead, _ = chacha20poly1305.New(uConn.AuthKey)
45}
46
47// Xray-core/transport/internet/reality/reality.go#L160-L161
48// Encrypt SessionId using AEAD algorithm.
49// Here preMasterKey is used as key and ClientHello is used as additional data.
50// Note: hello.SessionId[:0] refers to reuse of its memory space.
51aead.Seal(hello.SessionId[:0], hello.Random[20:], hello.SessionId[:16], hello.Raw)
52// Copy the final SessionId to bytes 40-72 of ClientHello
53// copy() is used to copy data. Usage: copy(destination, source)
54copy(hello.Raw[39:], hello.SessionId)

At this point, the REALITY client has completed its own hidden marking. Next, the client initiates a TLS connection to the REALITY server:

1if err := uConn.HandshakeContext(ctx); err != nil {
2	return nil, err
3}

🐧 Server

👆I use this emoji not because I prefer Linux servers, but because the performance and maintenance experience of Windows Server is too poor

Coming to the source code of the REALITY server, it is actually a variant of the server part of the crypto/tls package in the Go 1.20 standard library. Since the REALITY server modifies the crypto/tls package with the principle of minimal modification, there are many files in the directory that are not directly related to the REALITY protocol, and the directory tree is not listed here.

The key to the REALITY server processing TLS handshake is func Server in the file tls.go. In fact, after carefully reading the client source code in Xray-core, readers should have a general understanding of the key logic of the server to distinguish legitimate REALITY clients. In view of this, the following source code comments will be relatively concise, and readers are welcome to forgive me.

So how does the REALITY server inform the legitimate REALITY client that it can transmit? How does the server handle the handshake initiated by the illegal client (censor)? For the convenience of description, here is a conclusion:

  1. The REALITY server forwards the ClientHello to the TLS server dest (mask server) with a valid certificate, makes minimal modifications to the ServerHello and Change Cipher Spec from dest and its attached encrypted information, and then forwards it to the REALITY client. This approach can complete the TLS handshake in the normal TLS server way, avoiding the generation of server-side TLS fingerprints.

  2. When the REALITY server modifies the encryption information attached after Change Cipher Spec, it replaces all digital certificates with “temporary certificates” and modifies the signature value of the “temporary certificate” so that the REALITY client can compare the signature calculated by using preMasterKey to inform the client that the transmission can be carried out.

  3. The REALITY server forwards all traffic except that from the legitimate REALITY client to dest. The benefits of this approach are the same as 1.

  1// REALITY/blob/main/tls.go#L113
  2// Here is the function definition, where conn is the TCP connection accepted by the server
  3func Server(ctx context.Context, conn net.Conn, config *Config) (*Conn, error)
  4
  5// REALITY/blob/main/tls.go#L119
  6// Open a TCP connection to dest (mask server) in REALITY configuration
  7target, err := config.DialContext(ctx, config.Type, config.Dest)
  8
  9// REALITY/blob/main/tls.go#L139-L156
 10// Initialize the mutex (allows to monopolize a task)
 11mutex := new(sync.Mutex)
 12
 13// Initialize the serverHandshakeStateTLS13 instance
 14// which is defined in handshake_server_tls13.go to store the handshake information
 15hs := serverHandshakeStateTLS13{
 16		c: &Conn{
 17			conn: &MirrorConn{
 18				Mutex:  mutex,
 19				Conn:   conn,
 20				Target: target,
 21			},
 22			config: config,
 23		},
 24		ctx: context.Background(),
 25	}
 26
 27// Indicates whether the client sends irrelevant data before completing the handshake
 28copying := false
 29
 30// Waiting group, so that the main coroutine can wait for the sub-coroutine to finish running
 31// Coroutine: lightweight thread in Go language
 32waitGroup := new(sync.WaitGroup)
 33waitGroup.Add(2) // Add two sub-coroutines
 34
 35// REALITY/blob/main/tls.go#L158-L223
 36// Run the coroutine to distinguish clients
 37go func() {
 38	for {
 39		mutex.Lock() // Lock, monopolize current task
 40		// Read ClientHello, context.Background() generates an empty ctx placeholder
 41		hs.clientHello, err = hs.c.readClientHello(context.Background())
 42		// Determine 1. Whether the client sends data before completing the handshake; 2. Read error;
 43		// 3. The version is less than TLS1.3; 4. The SNI in ClientHello is not in the configuration
 44		// If any condition is met, end the loop (break no longer enters the next loop)
 45		// (Go uses unconditional for to represent an infinite loop)
 46		if copying || err != nil || hs.c.vers != VersionTLS13 || !config.ServerNames[hs.clientHello.serverName] {
 47			break
 48		}
 49		// for loop to get the client x25519 public key
 50		// TLS1.3 ClientHello contains as many public keys and digital certificates as possible
 51		// for different algorithms to avoid
 52		// asking for supported algorithms adds 1 round trip delay in TLS1.2
 53		for i, keyShare := range hs.clientHello.keyShares {
 54			// Determine whether the key type is x25519 and the length is equal to 32 bytes
 55			// This is the length and type of the public key used by the REALITY client
 56			// If any condition is not met, countinue enters the next loop
 57			if keyShare.group != X25519 || len(keyShare.data) != 32 {
 58				continue
 59			}
 60			// Enter the REALITY private key and the client public key into the ECDH algorithm to calculate the shared key.
 61			if hs.c.AuthKey, err = curve25519.X25519(config.PrivateKey, keyShare.data); err != nil {
 62				break
 63			}
 64			// The key is input into HKDF (HMAC-based key derivation function) to calculate preMasterKey.
 65			if _, err = hkdf.New(sha256.New, hs.c.AuthKey, hs.clientHello.random[:20], []byte("REALITY")).Read(hs.c.AuthKey); err != nil {
 66				break
 67			}
 68			// Choose AEAD Algorithm。
 69			var aead cipher.AEAD
 70			if aesgcmPreferred(hs.clientHello.cipherSuites) {
 71				block, _ := aes.NewCipher(hs.c.AuthKey)
 72				aead, _ = cipher.NewGCM(block)
 73			} else {
 74				aead, _ = chacha20poly1305.New(hs.c.AuthKey)
 75			}
 76			if config.Show {
 77				fmt.Printf("REALITY remoteAddr: %v\ths.c.AuthKey[:16]: %v\tAEAD: %T\n", remoteAddr, hs.c.AuthKey[:16], aead) //for debug
 78			}
 79			// Initialize two 32-byte slices
 80			// Used to store ciphertext and plaintext respectively.
 81			ciphertext := make([]byte, 32)
 82			plainText := make([]byte, 32)
 83			copy(ciphertext, hs.clientHello.sessionId)
 84			copy(hs.clientHello.sessionId, plainText)
 85			// Decrypt SessionId using AEAD algorithm.
 86			if _, err = aead.Open(plainText[:0], hs.clientHello.random[20:], ciphertext, hs.clientHello.raw); err != nil {
 87				break
 88			}
 89			// Parse the Xray-core version contained in SessionId,
 90			// Unix timestamp, and ShortId
 91			copy(hs.clientHello.sessionId, ciphertext)
 92			copy(hs.c.ClientVer[:], plainText)
 93			hs.c.ClientTime = time.Unix(int64(binary.BigEndian.Uint32(plainText[4:])), 0)
 94			copy(hs.c.ClientShortId[:], plainText[8:])
 95			if config.Show {
 96				fmt.Printf("REALITY remoteAddr: %v\ths.c.ClientVer: %v\n", remoteAddr, hs.c.ClientVer)
 97				fmt.Printf("REALITY remoteAddr: %v\ths.c.ClientTime: %v\n", remoteAddr, hs.c.ClientTime)
 98				fmt.Printf("REALITY remoteAddr: %v\ths.c.ClientShortId: %v\n", remoteAddr, hs.c.ClientShortId) //for debug
 99			}
100			// Determine whether Xray-core version, time delay, ShortId is allowed
101			if (config.MinClientVer == nil || Value(hs.c.ClientVer[:]...) >= Value(config.MinClientVer...)) &&
102				(config.MaxClientVer == nil || Value(hs.c.ClientVer[:]...) <= Value(config.MaxClientVer...)) &&
103				(config.MaxTimeDiff == 0 || time.Since(hs.c.ClientTime).Abs() <= config.MaxTimeDiff) &&
104				(config.ShortIds[hs.c.ClientShortId]) {
105				hs.c.conn = conn
106			}
107			// Identifies the type of public key used by the client for subsequent use
108			hs.clientHello.keyShares[0].group = CurveID(i)
109			break
110		}
111		if config.Show {
112			fmt.Printf("REALITY remoteAddr: %v\ths.c.conn == conn: %v\n", remoteAddr, hs.c.conn == conn) //for debug
113		}
114		break
115	}
116	// Unlock, cancel monopolization of current task
117	mutex.Unlock()
118	// Determine whether the client sends irrelevant data without completing the handshake
119	if hs.c.conn != conn {
120		if config.Show && hs.clientHello != nil {
121			fmt.Printf("REALITY remoteAddr: %v\tforwarded SNI: %v\n", remoteAddr, hs.clientHello.serverName) //for debug
122		}
123		// Forward the connection to dest in the configuration (i.e. the masquerade server)
124		io.Copy(target, underlying)
125	}
126	// Notify the waiting group that the coroutine has completed running
127	waitGroup.Done()
128}()

At this time, the server has completed the task of distinguishing clients. But the TLS handshake is not yet completed?

In the following part, the REALITY server forwards the ClientHello to dest, modifies the ServerHello returned by dest, replaces all digital certificates with “temporary certificates”, modifies the signature value of the “temporary certificate”, and then sends the modified ServerHello back to the legitimate REALITY client, thereby completing the TLS handshake and informing the client that it can transmit evasion traffic. Since the “temporary certificate” is not signed by the client’s trusted CA, the REALITY client confirms the server’s identity by verifying the signature value of the digital certificate.

  1// REALITY/blob/main/tls.go#L225-L349
  2// Run the coroutine for REALITY server handshake and communication with dest
  3go func() {
  4	// Initialize two slices of size = 8192 long
  5	// s2cSaved is used to store all received data from dest
  6	// buf is the buffer from REALITY server to dest
  7	s2cSaved := make([]byte, 0, size)
  8	buf := make([]byte, size)
  9	handshakeLen := 0
 10f:
 11	for {
 12		// Temporarily give up CPU time slices to optimize program performance
 13		runtime.Gosched()
 14		// Read the received data from dest into buf (buf will be cleared)
 15		n, err := target.Read(buf)
 16		// Determine whether data from dest has been received
 17		if n == 0 {
 18			if err != nil {
 19				conn.Close()
 20				waitGroup.Done()
 21				return
 22			}
 23			continue
 24		}
 25		// Lock, monopolize current task
 26		mutex.Lock()
 27		// Append the data in buf to the existing s2cSaved data
 28		s2cSaved = append(s2cSaved, buf[:n]...)
 29		// Determine whether the client sends irrelevant data without completing the handshake
 30		if hs.c.conn != conn {
 31			// Indicates that the client sends irrelevant data without completing the handshake
 32			copying = true
 33			break
 34		}
 35		// Determine whether the length of s2cSaved is too large
 36		if len(s2cSaved) > size {
 37			break
 38		}
 39		// Traverse type to determine the type of the packet returned by dest
 40		// REALITY/blob/main/tls.go#L91-L99
 41		// types = [7]string{
 42		// "Server Hello",
 43		// "Change Cipher Spec",
 44		// "Encrypted Extensions",
 45		// "Certificate",
 46		// "Certificate Verify",
 47		// "Finished",
 48		// "New Session Ticket",
 49		// }
 50		for i, t := range types {
 51			// Determine whether ServerHello has been sent
 52			// Enter the next loop if it matches
 53			if hs.c.out.handshakeLen[i] != 0 {
 54				continue
 55			}
 56			// Determine whether the type of this loop is
 57			// "New Session Ticket" and the length of s2cSaved is 0
 58			if i == 6 && len(s2cSaved) == 0 {
 59				break
 60			}
 61			// handshakeLen records the length of the handshake packet from dest.
 62			// Here, it is judged that its length is 0 and the length of the handshake packet from dest is greater than 5.
 63			// REALITY/blob/main/common.go#L63
 64			// recordHeaderLen = 5
 65			if handshakeLen == 0 && len(s2cSaved) > recordHeaderLen {
 66				// [1:3] points to the version information of the 2nd-3rd bytes of the TLS handshake packet
 67				// For compatibility reasons, the 2nd-3rd bytes of the ClientHello of TLS1.3, 
 68				// have the same value as TLS1.2, and the version mark of 1.3 exists in the extension
 69				// Here we determine whether the data packet from dest is ServerHello
 70				// or ChangeCipherSpec or ApplicationData,
 71				// These three types of data packets should appear in the handshake
 72				if Value(s2cSaved[1:3]...) != VersionTLS12 ||
 73					(i == 0 && (recordType(s2cSaved[0]) != recordTypeHandshake || s2cSaved[5] != typeServerHello)) ||
 74					(i == 1 && (recordType(s2cSaved[0]) != recordTypeChangeCipherSpec || s2cSaved[5] != 1)) ||
 75					(i > 1 && recordType(s2cSaved[0]) != recordTypeApplicationData) {
 76					break f
 77				}
 78				// [3:5] points to the handshake packet length information of the 4th and 5th bytes of the TLS handshake packet
 79				handshakeLen = recordHeaderLen + Value(s2cSaved[3:5]...)
 80			}
 81			if config.Show {
 82				fmt.Printf("REALITY remoteAddr: %v\tlen(s2cSaved): %v\t%v: %v\n", remoteAddr, len(s2cSaved), t, handshakeLen) //for debug
 83			}
 84			// Determine whether the handshake packet is too long
 85			if handshakeLen > size {
 86				break f
 87			}
 88			// Determine whether the handshake packet type is Change Cipher Spec
 89			if i == 1 && handshakeLen > 0 && handshakeLen != 6 {
 90				break f
 91			}
 92			// Determine whether the handshake packet type is Encrypted Extensions,
 93			// which is the encrypted part attached after the Change Cipher Spec.
 94			if i == 2 && handshakeLen > 512 {
 95				hs.c.out.handshakeLen[i] = handshakeLen
 96				hs.c.out.handshakeBuf = buf[:0]
 97				break
 98			}
 99			// Determine whether the handshake packet type is New Session Ticket
100			if i == 6 && handshakeLen > 0 {
101				hs.c.out.handshakeLen[i] = handshakeLen
102				break
103			}
104			// Determine whether the length of the handshake packet is 0 or greater than the s2cSaved length
105			if handshakeLen == 0 || len(s2cSaved) < handshakeLen {
106				// Unlock, cancel monopolization of current task
107				mutex.Unlock()
108				continue f
109			}
110			// Determine whether the handshake packet type is Server Hello
111			if i == 0 {
112				// Construct a new Server Hello
113				hs.hello = new(serverHelloMsg)
114				// unmarshal is used to parse the ServerHello from dest and fill it into the newly constructed handshake packet
115				// Here we check whether unmarshal reports an error and whether the handshake packet after parsing and filling is legal
116				if !hs.hello.unmarshal(s2cSaved[recordHeaderLen:handshakeLen]) ||
117					hs.hello.vers != VersionTLS12 || hs.hello.supportedVersion != VersionTLS13 ||
118					cipherSuiteTLS13ByID(hs.hello.cipherSuite) == nil ||
119					hs.hello.serverShare.group != X25519 || len(hs.hello.serverShare.data) != 32 {
120					break f
121				}
122			}
123			// Modify the sent length of the corresponding type of handshake packet
124			// then reset s2cSaved and handshakeLen
125			hs.c.out.handshakeLen[i] = handshakeLen
126			s2cSaved = s2cSaved[handshakeLen:]
127			handshakeLen = 0
128		} // End of traversal
129		// Mark the start time of the run
130		start := time.Now()
131		// Handshake with REALITY client
132		// That means sending modified Server Hello,
133		// Change Cipher Spec and Encrypted Extensions
134		err = hs.handshake()
135		if config.Show {
136			fmt.Printf("REALITY remoteAddr: %v\ths.handshake() err: %v\n", remoteAddr, err) //for debug
137		}
138		if err != nil {
139			break
140		}
141		// Run the coroutine to handle the connection with dest. At this time, the REALITY handshake has already ended.
142		// The connection to dest is no longer needed.
143		// The purpose of using coroutines is to avoid blocking communication with the client.
144		go func() {
145			if handshakeLen-len(s2cSaved) > 0 {
146				io.ReadFull(target, buf[:handshakeLen-len(s2cSaved)])
147			}
148			if n, err := target.Read(buf); !hs.c.isHandshakeComplete.Load() {
149				if err != nil {
150					conn.Close()
151				}
152				if config.Show {
153					fmt.Printf("REALITY remoteAddr: %v\ttime.Since(start): %v\tn: %v\terr: %v\n", remoteAddr, time.Since(start), n, err)
154				}
155			}
156		}()
157		// Wait for the client to return TLS Finished information
158		err = hs.readClientFinished()
159		if config.Show {
160			fmt.Printf("REALITY remoteAddr: %v\ths.readClientFinished() err: %v\n", remoteAddr, err)
161		}
162		if err != nil {
163			break
164		}
165		hs.c.isHandshakeComplete.Store(true)
166		break
167	}
168	// Unlock, cancel monopolization of current task
169	mutex.Unlock()
170	// Check if the REALITY server did not send a ServerHello.
171	// This is usually triggered when dest returns an invalid ServerHello
172	// or before dest returns ServerHello.
173	if hs.c.out.handshakeLen[0] == 0 {
174		// Determine whether dest does not process ClientHello correctly
175		if hs.c.conn == conn {
176			waitGroup.Add(1)
177			go func() {
178				// Forward all traffic to dest directly
179				io.Copy(target, underlying)
180				waitGroup.Done()
181			}()
182		}
183		// Write client connection content to s2cSaved
184		conn.Write(s2cSaved)
185		// Forward all traffic to dest directly
186		io.Copy(underlying, target)
187		// When io.Copy() returns, it means that dest has closed the connection. 
188		// The forwarding channel to the client should also be closed at this time
189		underlying.CloseWrite()
190	}
191	// Notify the waiting group that the coroutine has completed running
192	waitGroup.Done()
193}()

Next, the server randomly generates a “temporary certificate” in ed25519 format (actually it is done when the reality package is imported), and replaces the signature part of the digital certificate with the value obtained by inputting the public key of the “temporary trusted certificate” into the HMAC algorithm using preMasterKey as the key.

 1// REALITY/blob/main/handshake_server_tls13.go#L55-L59
 2// func init is executed when the package it is in is imported
 3func init() {
 4    // Defines the x509 certificate template
 5	certificate := x509.Certificate{SerialNumber: &big.Int{}}
 6    // Generate a 64-byte ed25519 private key
 7	// _ means ignore and discard the value
 8	_, ed25519Priv, _ = ed25519.GenerateKey(rand.Reader)
 9    // Generate a temporary x509 certificate using the previous template
10	// I don't know why the REALITY server intercepted
11	// Bytes 33-64 of the private key ed25519Priv
12	// Public key for the temporary certificate
13	signedCert, _ = x509.CreateCertificate(rand.Reader, &certificate, &certificate, ed25519.PublicKey(ed25519Priv[32:]), ed25519Priv)
14}
15
16// REALITY/blob/main/handshake_server_tls13.go#L74-L85
17// (Again) calculate preMasterKey
18// The key value calculated this time
19// is assigned to hs.sharedKey
20{
21	hs.suite = cipherSuiteTLS13ByID(hs.hello.cipherSuite)
22	c.cipherSuite = hs.suite.id
23	hs.transcript = hs.suite.hash.New()
24	
25	key, _ := generateECDHEKey(c.config.rand(), X25519)
26	copy(hs.hello.serverShare.data, key.PublicKey().Bytes())
27	peerKey, _ := key.Curve().NewPublicKey(hs.clientHello.keyShares[hs.clientHello.keyShares[0].group].data)
28	hs.sharedKey, _ = key.ECDH(peerKey)
29
30	c.serverName = hs.clientHello.serverName
31}
32
33// REALITY/blob/main/handshake_server_tls13.go#L94-L106
34// Modify the value of the temporary certificate signature
35{
36	// Store the temporary certificate in the certificate array
37	signedCert := append([]byte{}, signedCert...)
38
39	// Initialize the hmac calculation object h
40	// Use preMasterKey as the key
41	h := hmac.New(sha512.New, c.AuthKey)
42	// Write bytes 33-64 of the ed25519 private key to the buffer h
43	h.Write(ed25519Priv[32:])
44	// Calculate the hmac value of the buffer data and write it
45	// starting from byte 65 of the temporary certificate
46	h.Sum(signedCert[:len(signedCert)-64])
47
48	// Constructs a complete certificate object
49	hs.cert = &Certificate{
50		Certificate: [][]byte{signedCert},
51		PrivateKey:  ed25519Priv,
52	}
53	// Identifies the signature algorithm as ed25519
54	hs.sigAlg = Ed25519
55}

At this point, the REALITY server has completed the handshake with the client. The following is a short snippet of the final processing code:

 1// Block until all coroutines in the waiting group have completed running
 2waitGroup.Wait()
 3// Close the connection with dest
 4target.Close()
 5if config.Show {
 6	fmt.Printf("REALITY remoteAddr: %v\ths.c.handshakeStatus: %v\n", remoteAddr, hs.c.isHandshakeComplete.Load()) //for debug
 7}
 8// Determine whether the handshake with the REALITY client has ended
 9if hs.c.isHandshakeComplete.Load() {
10	// Return the connection to the caller
11	return hs.c, nil
12}
13// If the handshake with the REALITY client has not ended normally,
14// close the connection with the client
15conn.Close()
16// Return the error to the caller
17return nil, errors.New("REALITY: processed invalid connection")

After the REALITY server returns the connection, the caller (usually an upper-level proxy protocol such as VLESS) can transmit evasion traffic through the same public API provided by the reality package as crypto/tls. The subsequent traffic transmission is exactly the same as the behavior of crypto/tls.

🔐Verify the server identity

Normally, after the REALITY client verifies the server identity, it sends TLS Finished, thus ending the TLS handshake with the REALITY server and starting to circumvent the transmission of traffic. Here is a brief explanation of the key logic of the client verifying the signature value of the server certificate.

 1// Xray-core/transport/internet/reality/reality.go#L82-L104
 2func (c *UConn) VerifyPeerCertificate(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
 3	// Since the utls package does not provide low-level access and modification to certificates
 4	// Here, the reflect package in Go is used to obtain the memory address of the certificate array
 5	// And the original data is converted to a Go array by operating pointers and type assertions
 6	p, _ := reflect.TypeOf(c.Conn).Elem().FieldByName("peerCertificates")
 7	certs := *(*([]*x509.Certificate))(unsafe.Pointer(uintptr(unsafe.Pointer(c.Conn)) + p.Offset))
 8	// Convert the public key in the first certificate in the certificate array to
 9	// ed25519.PublicKey type
10	// Confirm the public key type by checking whether an error is reported
11	if pub, ok := certs[0].PublicKey.(ed25519.PublicKey); ok {
12		// Initialize hmac calculation object h
13		// Use preMasterKey as key
14		h := hmac.New(sha512.New, c.AuthKey)
15		// Write the certificate public key to h's buffer
16		h.Write(pub)
17		// h.Sum calculates and returns the hmac value of the buffer data
18		// Determine whether the hmac value is
19		// completely consistent with the certificate signature
20		if bytes.Equal(h.Sum(nil), certs[0].Signature) {
21			// Indicates that the REALITY server authentication has passed
22			c.Verified = true
23			return nil
24		}
25	}
26	... // The original verification logic of the crypto/tls package is omitted here
27}

🚀Conclusion

In this article, we have learned about the normal handshake process of the TLS1.3 protocol without ECH enabled. Based on this, we have analyzed the source code of the REALITY client and server in depth, and have gained insight into the specific implementation of the REALITY protocol to circumvent the SNI-based censorship strategy.

This article has been written non-stop since I established the blog and published the article on July 30, and it has been published today for exactly one week. I hope to provide readers with a comprehensive understanding of the REALITY protocol in the most understandable way without losing rigor, and I hope you who are reading this can gain something from it.

Finally, thanks to RPRX, the creator of XTLS/Xray-core, and all the contributors to the projects under ProjectX (XTLS). Since there are many projects under ProjectX, only the contributors of Xray-core are listed here:

(Made with contrib.rocks. Access from mainland China and Iran may be slow.)

(Recommended related reading: )

  1. A Detailed Look at RFC 8446 (a.k.a. TLS 1.3) - Cloudflare Blog
  2. Good-bye ESNI, hello ECH! - Cloudflare Blog