round1 | ||
LICENSE | ||
README.md |
sing-box vs xray
An attempt on benchmarking these two multi-protocol proxying frameworks.
In a nutshell
Results are inaccurate, they look like a statistical error, made by, for example, different ISP's network load, that's why I said "an attempt on benchmarking". It would be great if someone with a powerful CPU and 1 Gbit at home could do the benchmarks.
Both proxies are almost the same speed. Xray was sometimes a bit faster.
But in the iperf3 benchmark it very often behaved weird: from 2nd packet speed was dropping to zero
or I even got iperf3: error - control socket has closed unexpectedly
.
So, use what you want / to what you already get used / what works better in your case. As for me, I found sing-box' JSON config more convenient than Xray's. SB supports many protocols and platforms, even can setup a TUN interface (like a VPN app). On the other hand, Xray provides more "stealth" features to hide proxy traffic, that is important, I guess (?), in China and Iran.
Software versions used
sing-box built from dev-next branch, 26f092d
sing-box version unknown
Environment: go1.23.2 linux/amd64
Tags: with_gvisor,with_dhcp,with_wireguard,with_reality_server,with_clash_api,with_quic,with_utls,with_ech
Revision: d97a7569507816bf2ac1a355e19d26b521fb046e
CGO: enabled
Xray-core built from main branch, 5a96ef6
Xray 24.11.11 (Xray, Penetrates Everything.) 5a96ef6 (go1.23.2 linux/amd64)
A unified platform for anti-censorship.
Fork of iperf3 with socks support, run on client:
iperf 3.16+ (cJSON 1.7.15)
Linux dc09void 6.6.60_1 #1 SMP PREEMPT_DYNAMIC Mon Nov 11 21:45:58 UTC 2024 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads
Regular iperf3 from Alpine repo, run on server:
iperf 3.17.1 (cJSON 1.7.15)
Linux mx1.dc09.ru 6.6.61-0-virt #1-Alpine SMP PREEMPT_DYNAMIC 2024-11-14 20:10:08 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads
Benchmark 1: hyperfine, curl, direct outbound
File: bench_curl_direct.txt
Measurement of execution time of curl. Shows overhead of a proxying software.
Proxies are set up to accept requests by SOCKSv5, sing-box on port 2080, xray on 2081, and forward directly to net.
Units: ms (less is better)
~1.5M binary file over https from dc09.ru
proxy | min | avg | max |
---|---|---|---|
no proxy | 383.3 | 477.2 | 697.7 |
sing-box | 376.4 | 478.6 | 681.2 |
xray | 374.2 | 467.7 | 662.8 |
162 bytes HTML over plain http from dc09.ru
proxy | min | avg | max |
---|---|---|---|
no proxy | 51.0 | 60.5 | 81.7 |
sing-box | 49.1 | 62.2 | 69.9 |
xray | 51.2 | 61.6 | 76.4 |
~150K HTML over https from github.com
proxy | min | avg | max |
---|---|---|---|
no proxy | 365.3 | 402.1 | 449.8 |
sing-box | 338.6 | 390.5 | 445.5 |
xray | 342.3 | 390.8 | 426.0 |
Benchmark 2: iperf3, proxied outbound
Measurement of bandwidth with iperf3. Shows processing speed of a proxying software.
Client proxy (sing-box or xray, specified in a table column) is set up to accept requests by SOCKSv5 on 2080 or 2081 and to connect to a SOCKSv5, Trojan over uTLS or VLESS over Reality inbound on dc09.ru; server proxy on dc09.ru is either sing-box or xray (specified in a caption before a table), accepts requests on all 3 inbounds on ports 2220, 2221 and 2222; iperf3 server is running on the same host as a server proxy.
Units: Mbit/s (more is better)
no proxy
File: bench_iperf_noproxy.txt
sender 93.7 Mbit/s (receiver 91.4 Mbit/s)
server is sing-box
File: bench_sb_*.txt
protocol | sing-box | xray client |
---|---|---|
SOCKSv5 | 102.0 (91.3) | 102.0 (91.3) |
Trojan | 101.3 (91.0) | 100.5 (90.6) |
VLESS | 101.3 (91.3) | 101.5 (90.7) |
server is xray
File: bench_xray_*.txt
protocol | sing-box | xray client |
---|---|---|
SOCKSv5 | 101.5 (91.3) | 103.0 (91.4) |
Trojan | 100.1 (90.0) | 100.5 (90.6) |
VLESS | 99.6 (91.1) | 102.0 (91.1) |
To reproduce
Benchmark 1:
- Install hyperfine
- Run
./sing-box run --config direct.json
and./xray run -c direct_xray.json
on client (use different terminal tabs/windows, or put one proxy to background with&
) - Find some binary file with size about 1.5M and upload it to your server
(because I can not guarantee that I won't delete
Marisa.m4a
used in tests; it's an audio stream of some video downloaded from YT) - Run
hyperfine -m 50 'curl https://...link to the file...' 'curl --proxy socks5://127.0.0.1:2080 https://link' 'curl --proxy socks5://127.0.0.1:2081 https://link'
- Choose some web site with a small load and no ratelimits that gives an https-redirect
with a small HTML response when accessing it by plain http, like
http://dc09.ru
, repeat the test replacing the link - Choose some web site with a heavy load and relatively small HTML landing, like
https://github.com
, repeat the test replacing the link
Benchmark 2:
- Compile iperf3 fork with socks5 support
by cloning git repo (don't forget that you need branch
issue-1095-socks5-support
, not master!) and running./configure && make
-- you'll get a built iperf in./src/iperf3
- Generate your own TLS cert (
cert.pem
andkey.pem
included in the repo are fordc09.ru
domain name) withsing-box generate tls-keypair <insert domain here>
ORxray tls cert --domain=<domain>
- Replace
dc09.ru
in all configs to match your domain name instead of mine - Upload sing-box and xray to your server, install iperf3 from a package manager
or upload compiled previously (on a server, you won't need socks support),
upload
server.json
,server_xray.json
,cert.pem
andkey.pem
- Run
./sing-box run --config server.json &
on your server, then launchiperf3 -s
- In
config.json
edit the line"final": "vless-out"
to default tosocks-out
, the same forconfig_xray.json
: edit"outboundTag": "vless-out"
in the 2nd routing rule. - Run
./sing-box run --config config.json
and./xray run -c config_xray.json
on client, sing-box will open port 2080 for a SOCKSv5 inbound, xray will open port 2081 for its inbound. - Make tests with iperf3:
repo_with_iperf_fork/src/iperf3 -c <address of your server or domain> --bidir --socks5 127.0.0.1:2080
for sing-box client and... --socks5 127.0.0.1:2081
for xray client. - Change
"final": "socks-out"
and"outboundTag": "socks-out"
totrojan-out
to test with Trojan, restart sing-box and xray on client, peform iperf3 tests, then change default outbound back tovless-out
, restart proxy clients again, peform tests - Stop iperf3 server by hitting Ctrl-C, stop sing-box server proxy by bringing the task to foreground with
fg
command and hitting Ctrl-C - Run
./xray run -c server_xray.json &
on the server, then launchiperf3 -s
- Repeat the tests
- Stop iperf3 with Ctrl-C, stop xray with
fg
and Ctrl-C