DarkCat09/sb-vs-xray
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sb-vs-xray/README.md

7.4 KiB
Raw Blame History

sing-box vs xray

An attempt on benchmarking these two multi-protocol proxying frameworks.

In a nutshell

Results are inaccurate, they look like a statistical error, made by, for example, different ISP's network load, that's why I said "an attempt on benchmarking". It would be great if someone with a powerful CPU and 1 Gbit at home could do the benchmarks.

Both proxies are almost the same speed. Xray was sometimes a bit faster. But in the iperf3 benchmark it very often behaved weird: from 2nd packet speed was dropping to zero or I even got iperf3: error - control socket has closed unexpectedly.

So, use what you want / to what you already get used / what works better in your case. As for me, I found sing-box' JSON config more convenient than Xray's. SB supports many protocols and platforms, even can setup a TUN interface (like a VPN app). On the other hand, Xray provides more "stealth" features to hide proxy traffic, that is important, I guess (?), in China and Iran.

Software versions used

sing-box built from dev-next branch, 26f092d

sing-box version unknown

Environment: go1.23.2 linux/amd64
Tags: with_gvisor,with_dhcp,with_wireguard,with_reality_server,with_clash_api,with_quic,with_utls,with_ech
Revision: d97a7569507816bf2ac1a355e19d26b521fb046e
CGO: enabled

Xray-core built from main branch, 5a96ef6

Xray 24.11.11 (Xray, Penetrates Everything.) 5a96ef6 (go1.23.2 linux/amd64)
A unified platform for anti-censorship.

Fork of iperf3 with socks support, run on client:

iperf 3.16+ (cJSON 1.7.15)
Linux dc09void 6.6.60_1 #1 SMP PREEMPT_DYNAMIC Mon Nov 11 21:45:58 UTC 2024 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads

Regular iperf3 from Alpine repo, run on server:

iperf 3.17.1 (cJSON 1.7.15)
Linux mx1.dc09.ru 6.6.61-0-virt #1-Alpine SMP PREEMPT_DYNAMIC 2024-11-14 20:10:08 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads

Benchmark 1: hyperfine, curl, direct outbound

File: bench_curl_direct.txt

Measurement of execution time of curl. Shows overhead of a proxying software.

Proxies are set up to accept requests by SOCKSv5, sing-box on port 2080, xray on 2081, and forward directly to net.

Units: ms (less is better)

~1.5M binary file over https from dc09.ru

proxy min avg max
no proxy 383.3 477.2 697.7
sing-box 376.4 478.6 681.2
xray 374.2 467.7 662.8

162 bytes HTML over plain http from dc09.ru

proxy min avg max
no proxy 51.0 60.5 81.7
sing-box 49.1 62.2 69.9
xray 51.2 61.6 76.4

~150K HTML over https from github.com

proxy min avg max
no proxy 365.3 402.1 449.8
sing-box 338.6 390.5 445.5
xray 342.3 390.8 426.0

Benchmark 2: iperf3, proxied outbound

Measurement of bandwidth with iperf3. Shows processing speed of a proxying software.

Client proxy (sing-box or xray, specified in a table column) is set up to accept requests by SOCKSv5 on 2080 or 2081 and to connect to a SOCKSv5, Trojan over uTLS or VLESS over Reality inbound on dc09.ru; server proxy on dc09.ru is either sing-box or xray (specified in a caption before a table), accepts requests on all 3 inbounds on ports 2220, 2221 and 2222; iperf3 server is running on the same host as a server proxy.

Units: Mbit/s (more is better)

no proxy

File: bench_iperf_noproxy.txt

sender 93.7 Mbit/s (receiver 91.4 Mbit/s)

server is sing-box

File: bench_sb_*.txt

protocol sing-box xray client
SOCKSv5 102.0 (91.3) 102.0 (91.3)
Trojan 101.3 (91.0) 100.5 (90.6)
VLESS 101.3 (91.3) 101.5 (90.7)

server is xray

File: bench_xray_*.txt

protocol sing-box xray client
SOCKSv5 101.5 (91.3) 103.0 (91.4)
Trojan 100.1 (90.0) 100.5 (90.6)
VLESS 99.6 (91.1) 102.0 (91.1)

To reproduce

  • Get sing-box and xray binaries by downloading from "Releases" or compiling by yourself

Benchmark 1:

  • Install hyperfine
  • Run ./sing-box run --config direct.json and ./xray run -c direct_xray.json on client (use different terminal tabs/windows, or put one proxy to background with &)
  • Find some binary file with size about 1.5M and upload it to your server (because I can not guarantee that I won't delete Marisa.m4a used in tests; it's an audio stream of some video downloaded from YT)
  • Run hyperfine -m 50 'curl https://...link to the file...' 'curl --proxy socks5://127.0.0.1:2080 https://link' 'curl --proxy socks5://127.0.0.1:2081 https://link'
  • Choose some web site with a small load and no ratelimits that gives an https-redirect with a small HTML response when accessing it by plain http, like http://dc09.ru, repeat the test replacing the link
  • Choose some web site with a heavy load and relatively small HTML landing, like https://github.com, repeat the test replacing the link

Benchmark 2:

  • Compile iperf3 fork with socks5 support by cloning git repo (don't forget that you need branch issue-1095-socks5-support, not master!) and running ./configure && make -- you'll get a built iperf in ./src/iperf3
  • Generate your own TLS cert (cert.pem and key.pem included in the repo are for dc09.ru domain name) with sing-box generate tls-keypair <insert domain here> OR xray tls cert --domain=<domain>
  • Replace dc09.ru in all configs to match your domain name instead of mine
  • Upload sing-box and xray to your server, install iperf3 from a package manager or upload compiled previously (on a server, you won't need socks support), upload server.json, server_xray.json, cert.pem and key.pem
  • Run ./sing-box run --config server.json & on your server, then launch iperf3 -s
  • In config.json edit the line "final": "vless-out" to default to socks-out, the same for config_xray.json: edit "outboundTag": "vless-out" in the 2nd routing rule.
  • Run ./sing-box run --config config.json and ./xray run -c config_xray.json on client, sing-box will open port 2080 for a SOCKSv5 inbound, xray will open port 2081 for its inbound.
  • Make tests with iperf3: repo_with_iperf_fork/src/iperf3 -c <address of your server or domain> --bidir --socks5 127.0.0.1:2080 for sing-box client and ... --socks5 127.0.0.1:2081 for xray client.
  • Change "final": "socks-out" and "outboundTag": "socks-out" to trojan-out to test with Trojan, restart sing-box and xray on client, peform iperf3 tests, then change default outbound back to vless-out, restart proxy clients again, peform tests
  • Stop iperf3 server by hitting Ctrl-C, stop sing-box server proxy by bringing the task to foreground with fg command and hitting Ctrl-C
  • Run ./xray run -c server_xray.json & on the server, then launch iperf3 -s
  • Repeat the tests
  • Stop iperf3 with Ctrl-C, stop xray with fg and Ctrl-C