OpenStack網絡知識片斷(持續更新)
LinuxBridge建立虛拟機
生成裝置名:tap+portid
如果不是Xen,執行ip link show dev “tap+portid”看裝置是否已存在,若存在傳回;若不存在:
ip tuntap add “tap+portid” mode tap
ip link set “tap+portid” address <mac_address>
ip link set “tap+portid” up
LinuxBridge agent
使用pydev庫擷取本機所有以tap開頭的裝置
對每一個tap裝置(port)循環:
向Quantum擷取port的詳細資訊
ip lingk show dev <dev>
擷取裝置所在的網橋(對于新增裝置應該是空):在/sys/devices/virtual/net/目錄下找到以brq開頭的網橋名,對每一個網橋:
擷取/sys/devices/virtual/net/<bridge>/brif/目錄下所有裝置
生成network對應的網橋(brq+networkid)
擷取network對應phynet所對應的phyinterface(必須已存在)
ip link add link <phyinterface> name <phyinterface.vlanid> type vlan id <vlanid>
ip link set <phyinterface.vlanid> up
brctl addbr <brq+networkid>
brctl setfd <brq+networkid>
brctl stp <brq+networkid> off
ip link set <brq+networkid> up
brctl addif <brq+networkid> <phyinterface.vlanid>
brctl addif <brq+networkid> <dev>
循環結束
OVS run_instance(準備網絡)
建立虛拟機,例如網卡portid:1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba
ip link show dev qbr1e2b09d7-e9 #判斷
brctl addbr qbr1e2b09d7-e9 #增加Linux網橋
ip link show dev qvo1e2b09d7-e9 #判斷
ip link show dev qvb1e2b09d7-e9 #判斷
ip link add qvb1e2b09d7-e9 type veth peer name qvo1e2b09d7-e9 #增加對等裝置
ip link set qvb1e2b09d7-e9 up #激活裝置
ip link set qvb1e2b09d7-e9 promisc on #混雜模式
ip link set qvo1e2b09d7-e9 up
ip link set qvo1e2b09d7-e9 promisc on
ip link set qbr1e2b09d7-e9 up #激活網橋
brctl addif qbr1e2b09d7-e9 qvb1e2b09d7-e9 #向Linux網橋添加裝置
#下面的指令向OVS添加port
ovs-vsctl -- --may-exist add-port br-int qvo1e2b09d7-e9 -- set Interface qvo1e2b09d7-e9 external-ids:iface-id=1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:ea:ad:8d external-ids:vm-uuid=49b6d841-163f-4aab-b309-149727c227b4
OVS agent
初始化:
ovs-vsctl -- --if-exists del-port br-int patch-tun
ovs-ofctl del-flows br-int
ovs-ofctl add-flow br-int hard_timeout=0,idle_timeout=0,priority=1,actions=normal
循環主體:
1) ovs-vsctl list-ports br-int
輸出:qvo1e2b09d7-e9\nqvo2d58d5dc-db\nqvo2e505b97-bb\nqvo5739b2dc-78\nqvo69121bea-6a\nqvod58fde4e-5f\nqvoe0a0b269-53\n
2) 循環調用:ovs-vsctl get Interface qvo1e2b09d7-e9 external_ids,擷取iface-id(portid)
輸出:
{attached-mac="fa:16:3e:ea:ad:8d", iface-id="1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba", iface-status=active, vm-uuid="49b6d841-163f-4aab-b309-149727c227b4"}\n
3) 根據portid循環:
a) 向Quantum查詢資訊,調用get_device_details接口
b) ovs-vsctl -- --columns=external_ids,name,ofport find Interface external_ids:iface-id="1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba",輸出:
external_ids : {attached-mac="fa:16:3e:ea:ad:8d", iface-id="1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba", iface-status=active, vm-uuid="49b6d841-163f-4aab-b309-149727c227b4"}
name : "qvo1e2b09d7-e9"
ofport : 6
c) 給port所屬的network自動配置設定(如果已記錄過該net,直接跳到下一步)local vlan id(1-4094),且對于network對應的physical net,節點上要有一個OVS網橋與之對應,如果是vlan模式,需要做如下操作:
# outbound,出口的vlan轉換
br.add_flow(priority=4,
in_port=self.phys_ofports[physical_network], #與br-int連接配接的port的标号
dl_vlan=lvid, #自動配置設定的local vlan,從1開始
actions="mod_vlan_vid:%s,normal" % segmentation_id) #segmentation_id是plugin配置設定的vlan号
# inbound,入口的vlan轉換
self.int_br.add_flow(priority=3,
in_port=self.int_ofports[physical_network], #與上面的br連接配接的port的标号
dl_vlan=segmentation_id,
actions="mod_vlan_vid:%s,normal" % lvid)
d) ovs-vsctl set Port qvo1e2b09d7-e9 tag=1 #這裡的1是為network配置設定的local vlan id
e) ovs-ofctl del-flows br-int in_port=6 #這裡的6指port的标号,表示不允許資料流入
dhcp agent
dhcp agent需要為不同的plugin配置不同的interface_driver
OVS:quantum.agent.linux.interface.OVSInterfaceDriver
LinuxBridge:quantum.agent.linux.interface.BridgeInterfaceDriver
dhcp agent執行的指令及輸出:
#檢視裝置是否存在
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip -o link show tap9739ea30-d6
'17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n'
#檢視裝置IP
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip addr show tap9739ea30-d6 permanent scope global
'17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \n link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n inet 10.10.11.2/24 brd 10.10.11.255 scope global tap9739ea30-d6\n'
#檢視DHCP程序資訊,其中的程序号是從檔案中擷取/var/lib/quantum/dhcp/{netid}/pid
cat /proc/13695/cmdline
'dnsmasq\x00--no-hosts\x00--no-resolv\x00--strict-order\x00--bind-interfaces\x00--inter
#停止程序
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 kill -9 13695
#啟動程序
QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay QUANTUM_NETWORK_ID=c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap9739ea30-d6 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/host --dhcp-optsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.10.11.0,static,120s
dhcp agent需要為不同的plugin配置不同的interface_driver
OVS:quantum.agent.linux.interface.OVSInterfaceDriver
LinuxBridge:quantum.agent.linux.interface.BridgeInterfaceDriver
dhcp agent執行的指令及輸出:
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip -o link show tap9739ea30-d6
'17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n'
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip addr show tap9739ea30-d6 permanent scope global
'17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \n link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n inet 10.10.11.2/24 brd 10.10.11.255 scope global tap9739ea30-d6\n'
cat /proc/13695/cmdline
'dnsmasq\x00--no-hosts\x00--no-resolv\x00--strict-order\x00--bind-interfaces\x00--inter
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 kill -9 13695
QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay QUANTUM_NETWORK_ID=c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap9739ea30-d6 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/host --dhcp-optsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.10.11.0,static,120s
l3 agent
l3 agent需要配置與plugin對應的interface_driver:
OVS: quantum.agent.linux.interface.OVSInterfaceDriver,此時external_network_bridge為br-ex
LinuxBridge: quantum.agent.linux.interface.BridgeInterfaceDriver
初始化:
1. 加載interface_driver
2. ip netns list #列出以'qrouter-'開頭的namespace
輸出:
qdhcp-487f81ab-98d3-457a-b712-b29e71e89b52
qdhcp-084ae80a-b108-4f8a-90ca-f44aa1ca738a
qdhcp-7c25296d-bc81-45f6-bcc0-37fa44588b83
qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869
qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23
qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc
循環: ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -o link list
'9: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN \\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n25: qr-012c9d13-85: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:f2:a8:56 brd ff:ff:ff:ff:ff:ff\n26: qg-388798a1-55: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:f6:23:b9 brd ff:ff:ff:ff:ff:ff\n'
對命名空間内的裝置名作循環(加粗字型):
如果是qr開頭:ovs-vsctl --timeout=2 -- --if-exists del-port br-int qr-012c9d13-85
如果是qg開頭:ovs-vsctl --timeout=2 -- --if-exists del-port br-ex qg-388798a1-55
循環結束
循環結束
工作任務:
1. 保證br-ex存在
2. 向Quantum擷取router:external的網絡(一個l3 agent隻處理一個external網絡,預設是br-ex,可以在配置檔案中配置external_network_bridge)
3.
循環擷取Quantum中的router對象(隻處理連接配接外網的router):
增加qrouter-routerid命名空間
ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 sysctl -w net.ipv4.ip_forward=1
如果配置了metadata_ip,執行
rules.append(('INPUT', '-s 0.0.0.0/0 -d %s -p tcp -m tcp --dport %s -j ACCEPT' % (self.conf.metadata_ip, self.conf.metadata_port)))
rules.append(('PREROUTING', '-s 0.0.0.0/0 -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination %s:%s' % (self.conf.metadata_ip, self.conf.metadata_port)))
應用iptable規則
ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-save -t filter
ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-restore
ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-save -t nat
ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-restore
對于router的新增内部port循環:
ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -o link show qr-012c9d13-85,如果裝置不存在:
ovs-vsctl -- --may-exist add-port br-int qr-012c9d13-85
-- set Interface qr-012c9d13-85 type=internal
-- set Interface qr-012c9d13-85 external-ids:iface-id=012c9d13-8554-4b39-96b8-e4bd2e787559
-- set Interface qr-012c9d13-85 external-ids:iface-status=active
-- set Interface qr-012c9d13-85 external-ids:attached-mac=fa:16:3e:f2:a8:56
ip link set qr-012c9d13-85 address fa:16:3e:f2:a8:56
ip link set qr-012c9d13-85 netns qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc
ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip link set qr-012c9d13-85 up
ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip addr show qr-012c9d13-85 permanent scope global
ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -4 addr add 10.10.10.1/24 brd 10.10.10.255 scope global dev qr-012c9d13-85
如果router連接配接到外部網絡,增加snat規則(将内部的IP轉換為外網的IP),應用iptable規則
循環結束
對于router上删除的port循環:
在OVS上删除port
删除nat規則
循環結束
初始化gw_port:ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -o link show qg-388798a1-55,如果裝置不存在,同上(操作br-ex)
ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc route add default <gw_port的網關IP>
增加snat規則
處理router上的floatingIP,對br-ex上的port(qg-388798a1-55)配置外網位址,配置snat/dnat規則
循環結束