jasey Nginx/Tengine Non-Persistent Connections

How Many Non-Persistent Connections Can Nginx/Tengine Support Concurrently

來源 http://jaseywang.me/2015/06/04/how-many-non-persistent-connections-can-nginxtengine-support-concurrently/

碰巧看到一篇文章https://blog.cloudflare.com/how-to-receive-a-million-packets/

比這篇還前，厲害了

我總結用到的知識點

Fortunately, these Nginx servers all ship with 10G network cards, with mode 4 bonding, some link layer and lower level are also pre-optimizated like TSO/GSO/GRO/LRO, ring buffer size

TCP accept queue

CPU and memory

the networking goes wild cuz packets(包轉發率)

interrupts

drop/overrun

tcpdump  incoming packagea are less than 75Byte

the 3-way handshake TCP sengment HTTP POST request, and exist with 4-way handshake

resend the same requests every 10 minutes or longer,

More than 80% of the traffic are purely small TCP packages, which will have significant impact on network card and CPU

quite expensive to set up a TCP connection

the new request connected per second is qute small compare to the normal

需要12個知識點

包轉發率

packet size barckets for interface bond0

結論

總之，如果你想要一個完美的性能，你需要做下面這些：

確定流量均勻分布在許多RX隊列和SO_REUSEPORT程序上。在實踐中，隻要有大量的連接配接(或流動)，負載通常是分布式的。

需要有足夠的CPU容量去從核心上擷取資料包。

為了使事情更加穩定，RX隊列和接收程序都應該在單個NUMA節點上。

RX隊列和接收程序

(ethtool -S eth2 |grep rx   watch 'netstat -s --udp' )

packet size(bytes) count packet size(bytes) count

反推

what導緻TCP accept queue，nginx處理的能力問題

情況

a client packet's lifetime

The first one is the packet capture between load balance IPVS and Nginx, the second one is the communications between Nginx and upstream server

發現the 3-way handshake TCP sengment HTTP POST request, and exist with 4-way handshake

<a href="https://s1.51cto.com/wyfs02/M02/96/42/wKiom1kepnPDFuqGAADP5IuxrsY715.png" target="_blank"></a>

<a href="https://s2.51cto.com/wyfs02/M02/96/42/wKioL1kepnSBvSUrAAD9LCLxRMg175.png" target="_blank"></a>

最後的解決辦法

Nginx will becomes bottleneck one day,

I need to run some real traffic benchmarks to get the accurate result.

recovering the product 

加到9台後

Each Nginx gets about 10K qps, with 300ms response time, zero TCP queue, and 10% CPU util.

測試

The benchmark process is not complex, remove one Nginx server(real server) from IPVS at a time, and monitor its metrics like qps, response time, TCP queue, and CPU/memory/networking/disk utils. When qps or similar metric don't goes up anymore and begins to turn round, that's usually the maximum power the server can support.

1.Before kicking off, make sure some key parameters or directives are setting correct, including Nginx worker_processes/worker_connections, CPU affinity to distrubute interrupts, and kernel parameter(tcp_max_syn_backlog/file-max/netdev_max_backlog/somaxconn, etc.). 

2.Keep an eye on the rmem_max/wmem_max, during my benchmark, I notice quite different results with different values.

3.Turning off your access log to cut down the IO and timestamps in TCP stack may achieve better performance, I haven't tested.

4. In this case, you can decrease TCP keepalive parameter to workround.

測試結論

Here are the results:

The best performance for a single server is 25K qps, however during that period, it's not so stable, I observe a almost full queue size in TCP and some connection failures during requesting the URI, except that, everything seems normal. A conservative value is about 23K qps. That comes with 300K TCP established connections, 200Kpps, 500K interrupts and 40% CPU util.

During that time, the total resource consumed from the IPVS perspective is 900K current connection, 600Kpps, 800Mbps, and 100K qps.

The above benchmark was tested during 10:30PM ~ 11:30PM, the peak time usually falls between 10:00PM to 10:30PM.

單個伺服器的最佳性能是25K qps，但是在這段時間内，它并不那麼穩定，在TCP請求時，我觀察到幾乎完全的TCP隊列大小以及一些連接配接失敗，除了這個,其他似乎都是正常的。 保守值約為23K qps。 這與300K TCP建立連接配接，200Kpps，500K中斷和40％CPU util。

在此期間，從IPVS角度看，消耗的資源總量為900K目前連接配接，600Kpps，800Mbps和100K qps。

上述基準測試在10:30 PM11:30 PM進行測試，高峰時間通常在下午10:00至10:30之間。

本文轉自 liqius 51CTO部落格，原文連結：http://blog.51cto.com/szgb17/1927566，如需轉載請自行聯系原作者