What are the recommendations for 10 Gigabit network adapter tuning parameters?

​

What are the recommendations for 10 Gigabit network adapter tuning parameters?

环境

• Red Hat Enterprise Linux 7
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 5
• 10 Gigabit Ethernet network interface adapters (10GbE)

问题

• 10Gbit NIC with MTU of 9000. Testing using ​

  ​iperf​

  ​ between two systems only results in 6-7 gigabits/s transfer speed.
• What are the expected and recommended tuning parameters to configure to achieve 10Gbps connection wirespeed for streaming bulk transfers?

决议

The following parameters are very general recommendations to achieve faster streaming transfers over 10 Gigabit Ethernet.

Depending on the actual production use of the network, these settings may not be optimal for your usage, however they will certainly make artificial benchmarks like ​

​iperf​

​ run faster.

Interrupt Balancing

Ensure the ​

​irqbalance​

​ service is installed and running.

Ring Buffer

It is recommended to make the NIC receive and transmit ring buffers as large as possible to prevent packet drops.

The procedure to increase the buffer size is described at:

• ​​System dropping packets due to rx_fw_discards​​

The change can be persisted across reboots by following:

• ​​How do I run a script or program immediately after my network interface goes up?​​

Socket Buffers

Increasing the maximum socket buffer size to 16MiB allows buffers to grow larger if required.

The following changes can be applied in ​

​/etc/sysctl.conf​

​ and applied immediately with ​

​sysctl -p​

​:

​​Raw​​

# set min/default/max TCP read buffer
net.ipv4.tcp_rmem = 4096 87380 16777216
# set min/default/max TCP write buffer
net.ipv4.tcp_wmem = 4096 16384 16777216

# set maximum non-TCP receive socket buffer size
net.core.rmem_max = 16777216
# set maximum non-TCP write socket buffer size
net.core.wmem_max = 16777216
      

Increasing the default socket buffer sizes is not required as this basic level of tuning as it may harm efficiency of non-streaming transfers.

Network Budget

If the third column of ​

​/proc/net/softnet_stat​

​ is increasing, eg:

# cat /proc/net/softnet_stat
0073d76b 00000000 000049ae 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
      

Then there could be benefit to allowing the kernel packet receive handler to run for a slightly longer time.

The default value is ​

​300​

​ meaning 300 packets are received before the handler ends. This can be increased, though there is unlikely to be much improvement increasing this past several thousand.

The following change can be applied in ​

​/etc/sysctl.conf​

​sysctl -p​

# set number of packets softirq receives off NIC before exiting
net.core.netdev_budget = 1000