引言

SR-IOV簡介

網卡支援

配置步驟

1. 修改Bios enable SR-IOV

2. 修改啟動參數

3. 設定vf網卡mac位址，權限

4. 切換網卡驅動

5.啟動VPP

小結

引言

在隻有一張網卡的情況下，這個網卡不能既被VPP(DPDK)接管，同時被作業系統所使用。這個時候一般隻能通過建立虛拟機，在虛拟機中運作VPP(DPDK)接管virtio的虛拟網卡。最近才發現了SR-IOV技術，可以在PCI層面虛拟網卡，不需要建立虛拟機，這個虛拟網卡直接就可以被DPDK接管，而原來的實體網卡還可以正常使用

SR-IOV簡介

SR-IOV全稱 single root input/output virtualization，是一個硬體虛拟化的規範和标準。

SR-IOV 标準将一個PCIe的網絡控制器虛拟化成多個PCIe裝置，即多個PCI虛拟網卡，這些虛拟網卡不僅可以給虛拟機使用，也可以直接給作業系統使用，也可以給實體機上的DPDK使用。

SR-IOV的優點：效率高，速度快；缺點:特定硬體支援，主機闆，CPU，網卡都必須同時支援該規範。

PF （Phycics function） ——實體網卡

VF （Virutal Funtion）——虛拟網卡

網卡支援

https://www.intel.com/content/www/us/en/support/articles/000005722/network-and-io/ethernet-products.html

Intel® Ethernet Network Adapter X722 Series
- Intel® Ethernet Network Adapter X722-DA2
- Intel® Ethernet Network Adapter X722-DA4
Intel® Ethernet Converged Network Adapter XL710 Series
- Intel® Ethernet Converged Network Adapter XL710-QDA1
- Intel® Ethernet Converged Network Adapter XL710-QDA2
- Intel® Ethernet Converged Network Adapter XL710-QDA1 for OCP
- Intel® Ethernet Converged Network Adapter XL710-QDA2 for OCP
Intel® Ethernet Network Adapter XXV710 Series
- Intel® Ethernet Network Adapter XXV710-DA1
- Intel® Ethernet Network Adapter XXV710-DA2
- Intel® Ethernet Network Adapter XXV710-DA1 for OCP
- Intel® Ethernet Network Adapter XXV710-DA2 for OCP
Intel® Ethernet Converged Network Adapter X710 Series
- Intel® Ethernet Converged Network Adapter X710-DA2
- Intel® Ethernet Converged Network Adapter X710-DA4
- Intel® Ethernet Converged Network Adapter X710-T4
- Intel® Ethernet Controller X710/X557-AT 10GBASE-T
Intel® Ethernet Connection X722
- Intel® Ethernet Connection X722 for 10GBASE-T
- Intel® Ethernet Connection X722 for 10GbE backplane
- Intel® Ethernet Connection X722 for 10GbE QSFP+
- Intel® Ethernet Connection X722 for 10GbE SFP+
Intel® Ethernet Converged Network Adapter X550
- Intel® Ethernet Converged Network Adapter X550-T1
- Intel® Ethernet Converged Network Adapter X550-T2
Intel® Ethernet Converged Network Adapter X540
- Intel® Ethernet Converged Network Adapter X540-T1
- Intel® Ethernet Converged Network Adapter X540-T2
Intel® 82599 10 Gigabit Ethernet Controller
- Intel® Ethernet 82599EB 10 Gigabit Ethernet Controller
- Intel® Ethernet 82599ES 10 Gigabit Ethernet Controller
- Intel® Ethernet 82599EN 10 Gigabit Ethernet Controller
Intel® Ethernet Converged Network Adapter X520
- Intel® Ethernet Converged Network Adapter X520-DA2
- Intel® Ethernet Converged Network Adapter X520-SR1
- Intel® Ethernet Converged Network Adapter X520-SR2
- Intel® Ethernet Converged Network Adapter X520-LR1
- Intel® Ethernet Converged Network Adapter X520-T2
Intel® Ethernet Controller I350
- Intel® Ethernet Controller I350-AM4
- Intel® Ethernet Controller I350-AM2
- Intel® Ethernet Controller I350-BT2
Intel® Ethernet Server Adapter I350
- Intel® Ethernet Server Adapter I350-T2
- Intel® Ethernet Server Adapter I350-T4
- Intel® Ethernet Server Adapter I350-F2
- Intel® Ethernet Server Adapter I350-F4

配置步驟

1. 修改Bios enable SR-IOV

SR-IOV虛拟出來的網卡如何被VPP(DPDK)接管引言SR-IOV簡介網卡支援配置步驟小結

2. 修改啟動參數

添加intel_iommu=on iommu=pt igb.max_vfs=1，然後重新開機裝置

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg00/Root rhgb quiet intel_iommu=on iommu=pt igb.max_vfs=1"

Igb.max_vfs 指igb驅動的網卡建立VF的個數。如果是其他驅動一樣修改，如ixgbe.max_vfs。

檢視驅動的方式可以用DPDK工具dpdk-devbind.sh，根據具體的驅動來修改啟動參數

[[email protected] ~]# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:05:10.0 'I350 Ethernet Controller Virtual Function' drv=vfio-pci unused=igbvf,uio_pci_generic
0000:05:10.1 'I350 Ethernet Controller Virtual Function' drv=vfio-pci unused=igbvf,uio_pci_generic

Network devices using kernel driver
===================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0 drv=ixgbe unused=vfio-pci,uio_pci_generic 
0000:05:00.0 'I350 Gigabit Network Connection' if=enp5s0f0 drv=igb unused=vfio-pci,uio_pci_generic 
0000:05:00.1 'I350 Gigabit Network Connection' if=enp5s0f1 drv=igb unused=vfio-pci,uio_pci_generic 
0000:0a:00.0 'I210 Gigabit Network Connection' if=eno1 drv=igb unused=vfio-pci,uio_pci_generic 
0000:0b:00.0 'I210 Gigabit Network Connection' if=eno2 drv=igb unused=vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>

3. 設定vf網卡mac位址，權限

ip link set enp5s0f1 vf 0 mac a0:36:9f:aa:64:9d
ip link set dev enp5s0f1 vf 0 trust on
ip link set dev enp5s0f1 vf 0 spoof off
ip link set enp5s0f1 allmulticast on

這裡有一個坑點，VF預設是拒絕multicast多點傳播封包的。以至于ipv6 neighbor solicitation封包被過濾，VF網卡無法被鄰居發現。查詢了一圈資料，更換了多次驅動未果，終于發現PF開啟allmulticast，VF才能收到左右多點傳播封包，當然ipv6 nd也就成功了。

[[email protected] ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:bb:3d:00 brd ff:ff:ff:ff:ff:ff
3: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:09:63:9c brd ff:ff:ff:ff:ff:ff
    vf 0 MAC b2:6e:0e:68:0f:2b, spoof checking on, link-state auto, trust off
4: enp5s0f1: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:09:63:9d brd ff:ff:ff:ff:ff:ff
    vf 0 MAC a0:36:9f:aa:64:9d, spoof checking off, link-state auto, trust on

4. 切換網卡驅動

DPDK支援SR-IOV的驅動隻有igb_uio和vfio-pci，uio_pci_generic是無法接管VF的。這裡使用vfio-pci驅動。

modprobe vfio-pci

/etc/vpp/startup.conf 中添加uio-driver vfio-pci

dpdk {
        ## Change default settings for all interfaces
        # dev default {
                ## Number of receive queues, enables RSS 
                ## Default is 1
                # num-rx-queues 3

                ## Number of transmit queues, Default is equal
                ## to number of worker threads or 1 if no workers treads
                # num-tx-queues 3

                ## Number of descriptors in transmit and receive rings
                ## increasing or reducing number can impact performance
                ## Default is 1024 for both rx and tx
                # num-rx-desc 512 
                # num-tx-desc 512 

                ## VLAN strip offload mode for interface
                ## Default is off 
                # vlan-strip-offload on
        # } 

        ## Whitelist specific interface by specifying PCI address
        # dev 0000:02:00.0

        ## Blacklist specific device type by specifying PCI vendor:device
        ## Whitelist entries take precedence
        # blacklist 8086:10fb

        ## Set interface name
        # dev 0000:02:00.1 {
        #       name eth0
        # } 

        ## Whitelist specific interface by specifying PCI address and in
        ## addition specify custom parameters for this interface
        # dev 0000:02:00.1 {
        #       num-rx-queues 2
        # } 

        ## Change UIO driver used by VPP, Options are: igb_uio, vfio-pci,
        ## uio_pci_generic or auto (default)
        uio-driver vfio-pci

5.啟動VPP

systemctl start vpp

[[email protected] system]# vppctl 
    _______    _        _   _____  ___ 
 __/ __/ _ \  (_)__    | | / / _ \/ _ \
 _/ _// // / / / _ \   | |/ / ___/ ___/
 /_/ /____(_)_/\___/   |___/_/  /_/    

vpp# 
vpp# show int
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
VirtualFunctionEthernet5/10/0     1     down         9000/0/0/0     
VirtualFunctionEthernet5/10/1     2      up          9000/0/0/0     rx packets                    12
                                                                    rx bytes                     946
                                                                    tx packets                     4
                                                                    tx bytes                     364
                                                                    drops                         12
                                                                    ip4                            9
local0                            0     down          0/0/0/0       drops                          1
vpp# 
vpp# 
vpp# show int addr
VirtualFunctionEthernet5/10/0 (dn):
VirtualFunctionEthernet5/10/1 (up):
  L3 10.2.1.96/24
  L3 240e:ff:e000:8::96/64
local0 (dn):
vpp# 






[[email protected] ~]# ping6 240e:ff:e000:8::96
PING 240e:ff:e000:8::96(240e:ff:e000:8::96) 56 data bytes
64 bytes from 240e:ff:e000:8::96: icmp_seq=1 ttl=63 time=0.215 ms
64 bytes from 240e:ff:e000:8::96: icmp_seq=2 ttl=63 time=0.111 ms
64 bytes from 240e:ff:e000:8::96: icmp_seq=3 ttl=63 time=0.110 ms
64 bytes from 240e:ff:e000:8::96: icmp_seq=4 ttl=63 time=0.110 ms
^C
--- 240e:ff:e000:8::96 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.110/0.136/0.215/0.046 ms
[[email protected] ~]#

區域網路内其他機器可以鄰居發現VF，并成功ping通。

小結

網上SR-IOV技術用于虛拟網卡給VM的例子很多，但是還沒找到一個主控端直接使用的例子。是以這裡就分享一下這個配置測試過程，也算是對自己踩過坑的一個總結吧。

SR-IOV虛拟出來的網卡如何被VPP(DPDK)接管引言SR-IOV簡介網卡支援配置步驟小結

引言

SR-IOV簡介

網卡支援

配置步驟

1. 修改Bios enable SR-IOV

2. 修改啟動參數

3. 設定vf網卡mac位址，權限

4. 切換網卡驅動

5.啟動VPP

小結

繼續閱讀

/etc/snmp/snmpd.conf

Circuit Switching and Packet Switching Networks

windows下用NET-SNMP接受并處理SNMP Trap/Notification

usb 列印機問題

2010年CIO面臨的五大迫在眉睫的任務

Cisco HDLC

TCP的Checksum計算方法[轉]

IP首部及檢驗和算法

vmware,qemu各種方式上網設定

Generic P2P Architecture, Tutorial and Example - CodeProject

配置Windows網絡同時支援DHCP和靜态IP

再談存儲-IOPS 計算方法

asp.net 該目錄不存在或因為安全設定而無法通路

What are training set, validation set and test set?

《eWEEK》：09年5大科技發展趨勢雲計算居首

「Python 網絡自動化」NETCONF —— Python 使用 NETCONF 管理配置 H3C 網絡裝置