sb-vs-xray/README.md

# sing-box vs xray

An attempt on benchmarking these two multi-protocol proxying frameworks.

## In a nutshell

Results are inaccurate, they look like a statistical error, made by,
for example, different ISP's network load, that's why I said "an attempt on benchmarking".
It would be great if someone with a powerful CPU and 1 Gbit at home could do the benchmarks.

Both proxies are almost the same speed. *Xray was sometimes a bit faster.*
But in the iperf3 benchmark it very often behaved weird: from 2nd packet speed was dropping to zero
or I even got `iperf3: error - control socket has closed unexpectedly`.

So, use what you want / to what you already get used / what works better in your case.
As for me, I found sing-box' JSON config more convenient than Xray's.
SB supports many protocols and platforms, even can setup a TUN interface (like a VPN app).
On the other hand, Xray provides more "stealth" features to hide proxy traffic,
that is important, I guess (?), in China and Iran.

## Software versions used

sing-box built from dev-next branch, [26f092d](https://github.com/SagerNet/sing-box/commit/26f092da6fb0801b11c91fd5c8468e9949312e02)
```
sing-box version unknown

Environment: go1.23.2 linux/amd64
Tags: with_gvisor,with_dhcp,with_wireguard,with_reality_server,with_clash_api,with_quic,with_utls,with_ech
Revision: d97a7569507816bf2ac1a355e19d26b521fb046e
CGO: enabled
```

Xray-core built from main branch, [5a96ef6](https://github.com/XTLS/Xray-core/commit/5a96ef632d65b8e68c4f337e0f918a55d1925396)
```
Xray 24.11.11 (Xray, Penetrates Everything.) 5a96ef6 (go1.23.2 linux/amd64)
A unified platform for anti-censorship.
```

Fork of iperf3 with socks support, run on client:
```
iperf 3.16+ (cJSON 1.7.15)
Linux dc09void 6.6.60_1 #1 SMP PREEMPT_DYNAMIC Mon Nov 11 21:45:58 UTC 2024 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads
```

Regular iperf3 from Alpine repo, run on server:
```
iperf 3.17.1 (cJSON 1.7.15)
Linux mx1.dc09.ru 6.6.61-0-virt #1-Alpine SMP PREEMPT_DYNAMIC 2024-11-14 20:10:08 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads
```

## Benchmark 1: hyperfine, curl, direct outbound

File: `bench_curl_direct.txt`

Measurement of execution time of curl.
Shows overhead of a proxying software.

Proxies are set up to accept requests by SOCKSv5,
sing-box on port 2080, xray on 2081,
and forward directly to net.

Units: ms (less is better)

### ~1.5M binary file over https from dc09.ru

|proxy|min|avg|max|
|:----|:-:|:-:|:-:|
|no proxy|383.3|477.2|697.7|
|sing-box|376.4|478.6|681.2|
|xray|374.2|467.7|662.8|

### 162 bytes HTML over plain http from dc09.ru

|proxy|min|avg|max|
|:----|:-:|:-:|:-:|
|no proxy|51.0|60.5|81.7|
|sing-box|49.1|62.2|69.9|
|xray|51.2|61.6|76.4|

### ~150K HTML over https from github.com

|proxy|min|avg|max|
|:----|:-:|:-:|:-:|
|no proxy|365.3|402.1|449.8|
|sing-box|338.6|390.5|445.5|
|xray|342.3|390.8|426.0|

## Benchmark 2: iperf3, proxied outbound

Measurement of bandwidth with iperf3.
Shows processing speed of a proxying software.

Client proxy (sing-box or xray, specified in a table column) is set up
to accept requests by SOCKSv5 on 2080 or 2081 and to connect to
a SOCKSv5, Trojan over uTLS or VLESS over Reality inbound on dc09.ru;
server proxy on dc09.ru is either sing-box or xray (specified in a caption before a table),
accepts requests on all 3 inbounds on ports 2220, 2221 and 2222;
iperf3 server is running on the same host as a server proxy.

Units: Mbit/s (more is better)

### no proxy

File: `bench_iperf_noproxy.txt`

*sender* 93.7 Mbit/s (*receiver* 91.4 Mbit/s)

### server is sing-box

File: `bench_sb_*.txt`

|protocol|sing-box|xray client|
|:-------|:------:|:---------:|
|SOCKSv5|102.0 (91.3)|102.0 (91.3)|
|Trojan|101.3 (91.0)|100.5 (90.6)|
|VLESS|101.3 (91.3)|101.5 (90.7)|

### server is xray

File: `bench_xray_*.txt`

|protocol|sing-box|xray client|
|:-------|:------:|:---------:|
|SOCKSv5|101.5 (91.3)|103.0 (91.4)|
|Trojan|100.1 (90.0)|100.5 (90.6)|
|VLESS|99.6 (91.1)|102.0 (91.1)|

## To reproduce

- Get [sing-box](https://github.com/SagerNet/sing-box/releases/latest)
  and [xray](https://github.com/XTLS/Xray-core/releases/latest) binaries
  by downloading from "Releases" or compiling by yourself

Benchmark 1:
- Install [hyperfine](https://github.com/sharkdp/hyperfine)
- Run `./sing-box run --config direct.json` and `./xray run -c direct_xray.json` on client
  (use different terminal tabs/windows, or put one proxy to background with `&`)
- Find some binary file with size about 1.5M and upload it to your server
  (because I can not guarantee that I won't delete `Marisa.m4a` used in tests;
  it's an audio stream of some video downloaded from YT)
- Run
  `hyperfine -m 50 'curl https://...link to the file...' 'curl --proxy socks5://127.0.0.1:2080 https://link' 'curl --proxy socks5://127.0.0.1:2081 https://link'`
- Choose some web site with a small load and no ratelimits that gives an https-redirect
  with a small HTML response when accessing it by plain http, like `http://dc09.ru`,
  repeat the test replacing the link
- Choose some web site with a heavy load and relatively small HTML landing, like `https://github.com`,
  repeat the test replacing the link

Benchmark 2:
- Compile [iperf3 fork with socks5 support](https://github.com/davidBar-On/iperf/tree/issue-1095-socks5-support)
  by cloning git repo (don't forget that you need branch `issue-1095-socks5-support`, not master!)
  and running `./configure && make` -- you'll get a built iperf in `./src/iperf3`
- Generate your own TLS cert (`cert.pem` and `key.pem` included in the repo are for `dc09.ru` domain name)
  with `sing-box generate tls-keypair <insert domain here>` OR `xray tls cert --domain=<domain>`
- Replace `dc09.ru` in all configs to match your domain name instead of mine
- Upload sing-box and xray to your server, install iperf3 from a package manager
  or upload compiled previously (on a server, you won't need socks support),
  upload `server.json`, `server_xray.json`, `cert.pem` and `key.pem`
- Run `./sing-box run --config server.json &` on your server, then launch `iperf3 -s`
- In `config.json` edit the line `"final": "vless-out"` to default to `socks-out`,
  the same for `config_xray.json`: edit `"outboundTag": "vless-out"` in the 2nd routing rule.
- Run `./sing-box run --config config.json` and `./xray run -c config_xray.json` on client,
  sing-box will open port 2080 for a SOCKSv5 inbound, xray will open port 2081 for its inbound.
- Make tests with iperf3:
  `repo_with_iperf_fork/src/iperf3 -c <address of your server or domain> --bidir --socks5 127.0.0.1:2080`
  for sing-box client and `... --socks5 127.0.0.1:2081` for xray client.
- Change `"final": "socks-out"` and `"outboundTag": "socks-out"` to `trojan-out` to test with Trojan,
  restart sing-box and xray on client, peform iperf3 tests,
  then change default outbound back to `vless-out`, restart proxy clients again, peform tests
- Stop iperf3 server by hitting Ctrl-C, stop sing-box server proxy by bringing the task to foreground with `fg` command and hitting Ctrl-C
- Run `./xray run -c server_xray.json &` on the server, then launch `iperf3 -s`
- Repeat the tests
- Stop iperf3 with Ctrl-C, stop xray with `fg` and Ctrl-C