sb-vs-xray/README.md

sing-box vs xray

An attempt on benchmarking these two multi-protocol proxying frameworks.

In a nutshell

Results are inaccurate, they look like a statistical error, made by, for example, different ISP's network load, that's why I said "an attempt on benchmarking". It would be great if someone with a powerful CPU and 1 Gbit at home could do the benchmarks.

Both proxies are almost the same speed. Xray was sometimes a bit faster. But in the iperf3 benchmark it very often behaved weird: from 2nd packet speed was dropping to zero or I even got iperf3: error - control socket has closed unexpectedly.

So, use what you want / to what you already get used / what works better in your case. As for me, I found sing-box' JSON config more convenient than Xray's. SB supports many protocols and platforms, even can setup a TUN interface (like a VPN app). On the other hand, Xray provides more "stealth" features to hide proxy traffic, that is important, I guess (?), in China and Iran.

To reproduce

• Get sing-box and xray binaries by downloading from "Releases" or compiling by yourself
• Compile iperf3 fork with socks5 support by cloning git repo (don't forget that you need branch issue-1095-socks5-support, not master!) and running ./configure && make -- you'll get a built iperf in ./src/iperf3
• Generate your own TLS cert (cert.pem and key.pem included in the repo are for dc09.ru domain name) with sing-box generate tls-keypair <insert domain here> OR xray tls cert --domain=<domain>
• Replace dc09.ru in all configs to match your domain name instead of mine
• Upload sing-box and xray to your server, install iperf3 from a package manager or upload compiled previously (on a server, you won't need socks support), upload server.json, server_xray.json, cert.pem and key.pem
• Run ./sing-box run --config server.json & on your server, then launch iperf3 -s
• In config.json edit the line "final": "vless-out" to default to socks-out, the same for config_xray.json: edit "outboundTag": "vless-out" in the 2nd routing rule.
• Run ./sing-box run --config config.json and ./xray run -c config_xray.json on client, sing-box will open port 2080 for a SOCKSv5 inbound, xray will open port 2081 for its inbound.
• Make tests with iperf3: repo_with_iperf_fork/src/iperf3 -c <address of your server or domain> --bidir --socks5 127.0.0.1:2080 for sing-box client and ... --socks5 127.0.0.1:2081 for xray client.
• Change "final": "socks-out" and "outboundTag": "socks-out" to trojan-out to test with Trojan, restart sing-box and xray on client, peform iperf3 tests, then change default outbound back to vless-out, restart proxy clients again, peform tests
• Stop iperf3 server by hitting Ctrl-C, stop sing-box server proxy by bringing the task to foreground with fg command and hitting Ctrl-C
• Run ./xray run -c server_xray.json & on the server, then launch iperf3 -s
• Repeat the tests
• Stop iperf3 with Ctrl-C, stop xray with fg and Ctrl-C

Software versions used

sing-box built from dev-next branch, 26f092d

sing-box version unknown

Environment: go1.23.2 linux/amd64
Tags: with_gvisor,with_dhcp,with_wireguard,with_reality_server,with_clash_api,with_quic,with_utls,with_ech
Revision: d97a7569507816bf2ac1a355e19d26b521fb046e
CGO: enabled

Xray-core built from main branch, 5a96ef6

Xray 24.11.11 (Xray, Penetrates Everything.) 5a96ef6 (go1.23.2 linux/amd64)
A unified platform for anti-censorship.

Fork of iperf3 with socks support, run on client:

iperf 3.16+ (cJSON 1.7.15)
Linux dc09void 6.6.60_1 #1 SMP PREEMPT_DYNAMIC Mon Nov 11 21:45:58 UTC 2024 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads

Regular iperf3 from Alpine repo, run on server:

iperf 3.17.1 (cJSON 1.7.15)
Linux mx1.dc09.ru 6.6.61-0-virt #1-Alpine SMP PREEMPT_DYNAMIC 2024-11-14 20:10:08 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads

Benchmark 1: hyperfine, curl, direct outbound

File: bench_curl_direct.txt

Measurement of execution time of curl. Shows overhead of a proxying software.

Proxies are set up to accept requests by SOCKSv5, sing-box on port 2080, xray on 2081, and forward directly to net.

Units: ms (less is better)

~1.5M binary file over https from dc09.ru

proxy	min	avg	max
no proxy	383.3	477.2	697.7
sing-box	376.4	478.6	681.2
xray	374.2	467.7	662.8

162 bytes HTML over plain http from dc09.ru

proxy	min	avg	max
no proxy	51.0	60.5	81.7
sing-box	49.1	62.2	69.9
xray	51.2	61.6	76.4

~150K HTML over https from github.com

proxy	min	avg	max
no proxy	365.3	402.1	449.8
sing-box	338.6	390.5	445.5
xray	342.3	390.8	426.0

Benchmark 2: iperf3, proxied outbound

Measurement of bandwidth with iperf3. Shows processing speed of a proxying software.

Client proxy (sing-box or xray, specified in a table column) is set up to accept requests by SOCKSv5 on 2080 or 2081 and to connect to a SOCKSv5, Trojan over uTLS or VLESS over Reality inbound on dc09.ru; server proxy on dc09.ru is either sing-box or xray (specified in a caption before a table), accepts requests on all 3 inbounds on ports 2220, 2221 and 2222; iperf3 server is running on the same host as a server proxy.

Units: Mbit/s (more is better)

no proxy

File: bench_iperf_noproxy.txt

sender 93.7 Mbit/s (receiver 91.4 Mbit/s)

server is sing-box

File: bench_sb_*.txt

protocol	sing-box	xray client
SOCKSv5	102.0 (91.3)	102.0 (91.3)
Trojan	101.3 (91.0)	100.5 (90.6)
VLESS	101.3 (91.3)	101.5 (90.7)

server is xray

File: bench_xray_*.txt

protocol	sing-box	xray client
SOCKSv5	101.5 (91.3)	103.0 (91.4)
Trojan	100.1 (90.0)	100.5 (90.6)
VLESS	99.6 (91.1)	102.0 (91.1)