I compared the performance of Jester on FreeBSD with that on Linux, then the result showed req/sec of Jester on FreeBSD is 1/10 of that on Linux. I changed kernel parameters of FreeBSD; kern.ipc.somaxconn=4096, net.inet.icmp.icmplim=40000, but the result did not change much. Load tester(wrk) was running on Linux. Did I overlook something?
Load test to Jester on Amazon Linux 2 (EC2 m5.4xlarge)
$ wrk -t 32 -c 256 http://172.31.41.171:5000/
Running 10s test @ http://172.31.41.171:5000/
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 519.69us 2.73ms 208.48ms 99.29%
Req/Sec 20.21k 1.33k 48.62k 89.05%
6499276 requests in 10.10s, 0.93GB read
Requests/sec: 643534.47
Transfer/sec: 94.51MB
Load test to Jester on FreeBSD 12 (EC2 m5.4xlarge)
$ wrk -t 32 -c 256 http://172.31.40.197:5000/
Running 10s test @ http://172.31.40.197:5000/
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.44ms 23.96ms 160.38ms 92.12%
Req/Sec 2.03k 576.70 10.56k 85.57%
638425 requests in 10.10s, 93.76MB read
Requests/sec: 63213.33
Transfer/sec: 9.28MB
No big difference. FreeBSD was 1/8 of Linux. (BTW, httpbeast is incredibly fast. 1M req/sec!)
Load test against httpbeast on Linux
$ wrk --latency -t 16 -c 1024 http://13.112.244.135:8080/
Running 10s test @ http://13.112.244.135:8080/
16 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.94ms 3.27ms 210.96ms 99.60%
Req/Sec 70.04k 4.72k 117.41k 88.28%
Latency Distribution
50% 769.00us
75% 0.97ms
90% 1.22ms
99% 2.87ms
11235338 requests in 10.10s, 1.11GB read
Socket errors: connect 19, read 0, write 0, timeout 0
Requests/sec: 1112420.06
Transfer/sec: 112.45MB
Load test against httpbeast on FreeBSD
$ wrk --latency -t 16 -c 1024 http://3.112.192.88:8080/
Running 10s test @ http://3.112.192.88:8080/
16 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.17ms 35.64ms 594.11ms 98.16%
Req/Sec 8.85k 2.02k 44.48k 92.77%
Latency Distribution
50% 7.16ms
75% 7.24ms
90% 7.33ms
99% 199.25ms
1413518 requests in 10.10s, 142.89MB read
Socket errors: connect 19, read 154423, write 8091, timeout 0
Requests/sec: 139962.78
Transfer/sec: 14.15MB
Cool, thanks for testing. I can't imagine what could cause this difference in httpbeast, perhaps Nim's kqueue implementation is significantly worse than its epoll implementation.
Maybe Amazon's EC2 has worse performance on FreeBSD? Have you tried comparing other http servers?
OK. nginx on FreeBSD has worse performance as well. It looks a performance problem in FreeBSD on EC2.
nginx on FreeBSD
$ wrk --latency -t 16 -c 1024 http://172.31.40.197/
Running 10s test @ http://172.31.40.197/
16 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 18.35ms 2.27ms 43.38ms 96.47%
Req/Sec 3.44k 640.96 18.06k 98.57%
Latency Distribution
50% 18.44ms
75% 18.52ms
90% 18.62ms
99% 26.91ms
548455 requests in 10.07s, 444.57MB read
Socket errors: connect 19, read 14064, write 1023, timeout 0
Requests/sec: 54448.59
Transfer/sec: 44.13MB
nginx on Linux
$ wrk --latency -t 16 -c 1024 http://172.31.41.171/
Running 10s test @ http://172.31.41.171/
16 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.18ms 5.73ms 212.48ms 99.76%
Req/Sec 20.84k 3.78k 51.04k 79.81%
Latency Distribution
50% 3.33ms
75% 4.35ms
90% 4.98ms
99% 5.97ms
3348919 requests in 10.10s, 11.72GB read
Socket errors: connect 19, read 0, write 0, timeout 0
Requests/sec: 331580.83
Transfer/sec: 1.16GB