SR-IOV功能介紹
SR-IOV 使一個單一的功能單元(比如,一個以太網端口)能看起來像多個獨立的實體裝置,即支援SR-IOV 功能的實體裝置能被配置為多個功能單元。
SR-IOV 兩種功能(function):
實體功能(Physical Functions,PF):這是完整的帶有 SR-IOV 能力的PCIe 裝置。PF 能像普通 PCI 裝置那樣被發現、管理和配置。
虛拟功能(Virtual Functions,VF):簡單的 PCIe 功能,它隻能處理I/O。每個 VF 都是從 PF 中分離出來的。每個實體硬體都有一個 VF 數目的限制。一個 PF,能被虛拟成多個 VF 用于配置設定給多個虛拟機。
Hypervisor 能将一個或者多個 VF 配置設定給一個虛機。在某一時刻,一個 VF 隻能被配置設定給一個虛機。一個虛機可以擁有多個 VF。在虛機的作業系統看來,一個 VF 網卡看起來和一個普通網卡沒有差別。SR-IOV 驅動是在核心中實作的。
網卡 SR-IOV 的例子:
![](https://img.laitimes.com/img/__Qf2AjLwojIjJCLyojI0JCLiAjM2EzLcd3LcJzLcJzdllmVldWYtl2PnVGcq5Sc1Yje5FXMzF2YvwVM5UjNzYzMtUGall3LcVmdhNXLwRHdo9CXt92YucWbpRWdvx2Yx5yazF2Lc9CX6MHc0RHaiojIsJye.jpeg)
将sriov計算節點的PF虛拟化成多個VF
sriov計算節點選擇光口1(例如 enp5s0f1)作為sriov網卡
1) 在計算節點上,設定BIOS,對于Intel使支援VT-d,可通過cat /proc/cpuinfo | grep vmx驗證
2) 配置計算節點的/etc/default/grub檔案,在GRUB_CMDLINE_LINUX中添加intel_iommu=on來激活VT-d功能,重新開機實體機(本環境采用intel 82599系列網卡)
$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=bclinux/root rd.lvm.lv=bclinux/swap intel_iommu=on rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
$ grub2-mkconfig --output=/boot/grub2/grub.cfg
$ reboot
複制
3) 本環境中82599網卡最多支援64個vf,拟在每個sriov計算節點虛拟出16個vf供使用指令如下:
# echo '16' > /sys/class/net/enp5s0f1/device/sriov_numvfs
###檢視是否生效
# lspci | grep Ethernet
05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
05:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
...
複制
# ip link show enp5s0f1
5: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 6c:92:bf:04:66:01 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
...
複制
4) 保證重新開機後vfs仍然存在,需要将設定vf的數目指令寫入/etc/rc.local,指令如下
# echo "echo '16' > /sys/class/net/enp5s0f1/device/sriov_numvfs" >> /etc/rc.local
# chmod +x /etc/rc.local /etc/rc.d/rc.local
複制
5) 修改控制節點nova.conf檔案,讓nova-schedule支援對pci passthrough的過濾
# vim /etc/nova/nova.conf
enable_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters
# systemctl restart openstack-nova-scheduler
複制
配置SRIOV虛機計算節點
1) 安裝neutron-sriov-agent
# yum install openstack-neutron-sriov-nic-agent openstack-neutron
複制
2) 配置neutron.conf
可以将控制節點neutron.conf配置拷貝過來,修改相應參數即可,主要是修改本地管理網ip參數
3) 配置sriov_agent.ini
# vim /etc/neutron/plugins/ml2/sriov_agent.ini
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver
[sriov_nic]
physical_device_mappings = default: enp5s0f1
exclude_devices =
複制
4) 啟動neutron-sriov-agent服務
# systemctl enable neutron-sriov-nic-agent
# systemctl start neutron-sriov-nic-agent
複制
5) 修改該節點nova.conf檔案,重新開機nova-compute
# vim /etc/nova/nova.conf
passthrough_whitelist = {"devname": "enp5s0f1", "physical_network": "physnet1"}
# systemctl restart openstack-nova-compute
複制
若需要配置多塊SRIOV網卡
[root@controller ~]# vim /etc/nova/nova.conf
[pci]
passthrough_whitelist = {"devname": "enp129s0f0", "physical_network": "physnet1"}
passthrough_whitelist = {"devname": "enp130s0f0", "physical_network": "physnet1"}
[root@SRIOV03 ~]# vim /etc/neutron/plugins/ml2/sriov_agent.ini
[sriov_nic]
physical_device_mappings = physnet1:enp129s0f0,physnet1:enp130s0f0
複制
測試srivo非親和功能
即:設定vm的兩個vnic在同一個非親群組,生成的sriov虛拟機的兩個vnic會分布在不同的兩個實體SRIOV網卡上
[root@controller ~]# openstack port create --vnic-type direct --binding-profile anti_affinity_group=gwgh729 --network 40fa3655-1dc6-4fa1-8a21-507ae2a92cc5 sriov-port-1
複制
[root@controller ~]# openstack port create --vnic-type direct --binding-profile anti_affinity_group=gwgh729 --network 40fa3655-1dc6-4fa1-8a21-507ae2a92cc5 sriov-port-2
+-----------------------+-----------------------------------------------------------------------------+
複制
[root@controller ~]# nova boot han-sriov-antiaffinity --image 4ce53c0e-b3f5-4463-bdcd-20ee0f74fec6 --flavor 071b6288-8087-4d2e-959a-0b3bbb7fc565 --nic port-id=cb96ed5a-9bbc-46f7-9a44-fdc45e11b25b --nic port-id=d8ae3337-b98a-4523-9665-ed7773c6da90 --availability-zone sriov:SRIOV03
複制
[root@SRIOV03 ~]# ip link show enp130s0f0|grep fa:16:3e:c8:f7:9f
[root@SRIOV03 ~]# ip link show enp129s0f0|grep fa:16:3e:c8:f7:9f
vf 0 MAC fa:16:3e:c8:f7:9f, vlan 811, spoof checking on, link-state auto, trust off, query_rss off
[root@SRIOV03 ~]# ip link show enp130s0f0|grep fa:16:3e:23:0e:45
vf 0 MAC fa:16:3e:23:0e:45, vlan 811, spoof checking on, link-state auto, trust off, query_rss off
[root@SRIOV03 ~]# ip link show enp129s0f0|grep fa:16:3e:23:0e:45
[root@SRIOV03 ~]#
複制
mac位址為fa:16:3e:c8:f7:9f的vnic落在enp129s0f0,mac位址為fa:16:3e:23:0e:45的vnic落在enp130s0f0