一，前言
阿裡雲Kubernetes叢集網絡目前有兩種方案，一種是flannel方案，另外一種是基于Calico和彈性網卡ENI的Terway方案。Terway和Flannel類似，不同的地方在于，Terway支援Pod彈性網卡，以及Network Policy功能。
這個系列文章中我們以實際案例排查，來看一看在容器網絡裡面，抓包分析問題的技巧，在看這一章之前，建議先看聲東同學的《叢集網絡詳解》這一章節
叢集網絡的架構如下圖所示，在問題實際發生時，抓eth0的封包會有什麼問題？如何抓veth網卡呢？影響pod網絡通訊的幾個因素？（安全組，vpc路由表指向，系統内的轉發參數等） 
叢集網絡示意圖1-1
二，示例
之前遇到一個liveness健康檢查失敗的問題，liveness是kubelet發起本機通路pod（如發送get請求）來确認的pod健康與否，對應健康檢查的通路日志在pod内是可以看到的，是以可以嘗試抓包排查，但是pod内部并不能簡簡單單的使用tcpdump抓包即可，docker/kubectl cp 一個tcpdump指令進去也不能直接抓封包，那麼，pod内的網絡封包怎麼抓呢？直接在ecs上抓包，有的時候并不能很明朗的展示問題，這次我們就說一說pod内的網絡封包怎麼抓？直接抓-i any接口的封包， 是否存在不利于問題的分析幹擾因素?
容器的網絡隔離使用的是linux的network namespace ，是以我們需要切換到對應的ns裡面抓包，下面開始示範
1.檢視指定 pod 運作在哪個主控端上
拿到主控端的資訊，然後登陸上去 cn-shenzhen.192.168.0.130
# kubectl get pods coredns-79989b94b6-d8kqn -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-79989b94b6-d8kqn 1/1 Running 0 20d 172.20.1.24 cn-shenzhen.192.168.0.130 <none> <none> 2.獲得容器的 pid
登入到對應的node上，找對應業務pod pid的方法（k8s裡面引入了pause概念的容器忽略即可）
# docker ps |grep coredns-79989b94b6-d8kqn
1285db52efdd registry-vpc.cn-shenzhen.aliyuncs.com/acs/coredns "/coredns -conf /etc…" 2 weeks ago Up 2 weeks k8s_coredns_coredns-79989b94b6-d8kqn_kube-system_78659545-8c40-4e11-8998-4d5eb7e553e5_0
9f31f9fc7887 registry-vpc.cn-shenzhen.aliyuncs.com/acs/pause-amd64:3.0 "/pause" 2 weeks ago Up 2 weeks k8s_POD_coredns-79989b94b6-d8kqn_kube-system_78659545-8c40-4e11-8998-4d5eb7e553e5_0
# docker inspect -f {{.State.Pid}} 1285db52efdd
2606 3.進入該容器的 network namespace
#切換ns的指令 nsenter 依賴 util-linux，預設未安裝的話，使用yum安裝即可（centos）
# yum -y install util-linux.x86_64
進入到對應容器的network ns裡面，并指向ip a檢視ip
# nsenter --target 2606 -n
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.1.24 netmask 255.255.255.255 broadcast 0.0.0.0
ether 42:75:08:d7:d8:49 txqueuelen 0 (Ethernet)
RX packets 43296250 bytes 7672294979 (7.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 42881762 bytes 10464280231 (9.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 17453316 bytes 1392774176 (1.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 17453316 bytes 1392774176 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
對比一下第一步拿到的pod ip 是一樣的
看下pod的監聽
# netstat -antpl |grep -vi wait
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.20.1.24:50416 172.21.0.1:443 ESTABLISHED 2606/coredns
tcp6 0 0 :::9153 :::* LISTEN 2606/coredns
tcp6 0 0 :::8080 :::* LISTEN 2606/coredns
tcp6 0 0 :::8181 :::* LISTEN 2606/coredns
tcp6 0 0 :::53 :::* LISTEN 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.2.69:33138 ESTABLISHED 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.3.52:40684 ESTABLISHED 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.3.53:59034 ESTABLISHED 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.3.52:40598 ESTABLISHED 2606/coredns 4.使用
tcpdump
抓包，指定 eth0 網卡
直接抓包嘗試,示範抓的是coredns的包，是以看到的是一些解析相關的封包
# tcpdump -i eth0 port 53 -nnvv -xxx |more
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:34:21.420121 IP (tos 0x0, ttl 63, id 51339, offset 0, flags [DF], proto UDP (17), length 152)
172.20.2.69.59687 > 172.20.1.24.53: [udp sum ok] 40871+ AAAA? proj-arms-prometheus-1721440269810815-cn-shenzhen.cn-shenzhen.log.aliyunc
s.com.arms-prom.svc.cluster.local. (124)
0x0000: 4275 08d7 d849 2686 12b1 bde5 0800 4500
0x0010: 0098 c88b 4000 3f11 1744 ac14 0245 ac14
0x0020: 0118 e927 0035 0084 0382 9fa7 0100 0001
0x0030: 0000 0000 0000 3170 726f 6a2d 6172 6d73
0x0040: 2d70 726f 6d65 7468 6575 732d 3137 3231
0x0050: 3434 3032 3639 3831 3038 3135 2d63 6e2d
0x0060: 7368 656e 7a68 656e 0b63 6e2d 7368 656e
0x0070: 7a68 656e 036c 6f67 0861 6c69 7975 6e63
0x0080: 7303 636f 6d09 6172 6d73 2d70 726f 6d03
0x0090: 7376 6307 636c 7573 7465 7205 6c6f 6361
0x00a0: 6c00 001c 0001
5.退出network namespace
直接exit即可
# exit
logout
# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.130 netmask 255.255.255.0 broadcast 192.168.0.255
ether 00:16:3e:10:51:22 txqueuelen 1000 (Ethernet)
RX packets 3500312304 bytes 2290117520419 (2.0 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2159089229 bytes 2977416649618 (2.7 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
...... 指令擴充：
nsenter切換namespace
nsenter切換namespace，常用的是網絡的命名空間，對應-t是pid，-n為切換網絡命名空間
# nsenter -h
Usage:
nsenter [options] <program> [<argument>...]
Run a program with namespaces of other processes.
Options:
-t, --target <pid> target process to get namespaces from
-m, --mount[=<file>] enter mount namespace
-u, --uts[=<file>] enter UTS namespace (hostname etc)
-i, --ipc[=<file>] enter System V IPC namespace
-n, --net[=<file>] enter network namespace
-p, --pid[=<file>] enter pid namespace
-U, --user[=<file>] enter user namespace
-S, --setuid <uid> set uid in entered namespace
-G, --setgid <gid> set gid in entered namespace
--preserve-credentials do not touch uids or gids
-r, --root[=<dir>] set the root directory
-w, --wd[=<dir>] set the working directory
-F, --no-fork do not fork before exec'ing <program>
-Z, --follow-context set SELinux context according to --target PID
-h, --help display this help and exit
-V, --version output version information and exit
For more details see nsenter(1).
除了docker inspect找pid，還有别的方式查namespace麼？lsns
lsns顯示的pid列，就是前面docke拿到的pid，nsenter需要切換ns的pid
# lsns -h
Usage:
lsns [options] [<namespace>]
List system namespaces.
Options:
-l, --list use list format output
-n, --noheadings don't print headings
-o, --output <list> define which output columns to use
-p, --task <pid> print process namespaces
-r, --raw use the raw output format
-u, --notruncate don't truncate text in columns
-t, --type <name> namespace type (mnt, net, ipc, user, pid, uts)
-h, --help display this help and exit
-V, --version output version information and exit
Available columns (for --output):
NS namespace identifier (inode number)
TYPE kind of namespace
PATH path to the namespace
NPROCS number of processes in the namespace
PID lowest PID in the namespace
PPID PPID of the PID
COMMAND command line of the PID
UID UID of the PID
USER username of the PID
For more details see lsns(8).
demo：
# lsns |grep -vi "pause"
NS TYPE NPROCS PID USER COMMAND
4026531836 pid 173 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531837 user 247 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531838 uts 172 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531839 ipc 165 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531840 mnt 163 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531856 mnt 1 28 root kdevtmpfs
4026531956 net 177 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026532183 mnt 1 2888 chrony /usr/sbin/chronyd
4026532188 mnt 1 7740 root /usr/bin/terwayd
4026532190 mnt 1 5172 nfsnobody /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --web.listen-address=0.0.0.0:9100 --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
4026532192 mnt 2 28585 root nginx: master process nginx -g daemon off
4026532193 uts 2 28585 root nginx: master process nginx -g daemon off
4026532194 pid 2 28585 root nginx: master process nginx -g daemon off
4026532195 mnt 4 2868 root sleep 3
4026532196 mnt 1 15067 root /opt/services/service-catalog controller-manager --port 8080 --leader-elect=false -v 10 --resync-interval 5m --broker-relist-interval 24h
4026532197 uts 1 15067 root /opt/services/service-catalog controller-manager --port 8080 --leader-elect=false -v 10 --resync-interval 5m --broker-relist-interval 24h
4026532198 pid 1 15067 root /opt/services/service-catalog controller-manager --port 8080 --leader-elect=false -v 10 --resync-interval 5m --broker-relist-interval 24h
4026532199 uts 4 2868 root sleep 3
4026532200 pid 4 2868 root sleep 3
4026532290 mnt 1 7451 nfsnobody /alicloud-monitor-controller agent --regionId=cn-zhangjiakou --clusterId=c50c9b260a208475987c2b502f5cc60b1 --logtostderr --v=4
4026532291 pid 1 7451 nfsnobody /alicloud-monitor-controller agent --regionId=cn-zhangjiakou --clusterId=c50c9b260a208475987c2b502f5cc60b1 --logtostderr --v=4
4026532426 mnt 1 5118 root /usr/local/bin/kube-proxy --proxy-mode=iptables --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf --cluster-cidr=172.20.0.0/16 --hostname-override=cn-zhangjiakou.i-8vbalkelzd05pwihto21
4026532427 pid 1 5118 root /usr/local/bin/kube-proxy --proxy-mode=iptables --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf --cluster-cidr=172.20.0.0/16 --hostname-override=cn-zhangjiakou.i-8vbalkelzd05pwihto21
4026532611 mnt 3 24418 root nginx: master process nginx -g daemon off
4026532612 uts 3 24418 root nginx: master process nginx -g daemon off
4026532613 pid 3 24418 root nginx: master process nginx -g daemon off
4026532614 mnt 2 28721 root /bin/sh -c /usr/local/bin/kubectl apply -f /etc/istio-operator/crds; sleep 4; /manager --watch-created-resources-events=false --zap-time-encoding=iso8601
4026532615 uts 2 28721 root /bin/sh -c /usr/local/bin/kubectl apply -f /etc/istio-operator/crds; sleep 4; /manager --watch-created-resources-events=false --zap-time-encoding=iso8601
4026532616 pid 2 28721 root /bin/sh -c /usr/local/bin/kubectl apply -f /etc/istio-operator/crds; sleep 4; /manager --watch-created-resources-events=false --zap-time-encoding=iso8601
4026532617 mnt 1 29704 1000 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
4026532618 uts 1 29704 1000 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
4026532619 pid 1 29704 1000 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
4026532620 mnt 1 29759 1000 /configmap-reload --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-ack-prometheus-operator-prometheus-rulefiles-0
4026532621 uts 1 29759 1000 /configmap-reload --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-ack-prometheus-operator-prometheus-rulefiles-0
4026532622 pid 1 29759 1000 /configmap-reload --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-ack-prometheus-operator-prometheus-rulefiles-0
4026532623 mnt 1 29838 1000 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://ack-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/
4026532624 uts 1 29838 1000 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://ack-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/
4026532625 pid 1 29838 1000 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://ack-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/
4026532628 mnt 1 18129 root /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
4026532629 uts 1 18129 root /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
4026532630 pid 1 18129 root /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
4026533118 mnt 2 5799 root /bin/sh /acs/entrypoint.sh
4026533968 mnt 1 8792 root calico-felix
4026534201 ipc 5 2868 root sleep 3
4026534204 net 5 2868 root sleep 3
4026534452 mnt 1 10617 root /coredns -conf /etc/coredns/Corefile
4026534453 uts 1 10617 root /coredns -conf /etc/coredns/Corefile
4026534454 pid 1 10617 root /coredns -conf /etc/coredns/Corefile
4026534460 mnt 1 10835 root /alicloud-disk-controller
4026534461 uts 1 10835 root /alicloud-disk-controller
4026534462 pid 1 10835 root /alicloud-disk-controller
4026534463 mnt 1 10841 1000 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --cluster.listen-address=[172.20.12.202]:6783 --storage.path=/alertmanager --data.retention=120h --web.listen-address=:9093 --web.external-url=http://ack-prometheus-operator-alertmanager.monitoring:9093 --web.route-prefix=/ --cluster.peer=alertmanager-ack-prometheus-operator-alertmanager-0.alertmanager-operated.monitoring.svc:6783
4026534464 uts 1 10841 1000 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --cluster.listen-address=[172.20.12.202]:6783 --storage.path=/alertmanager --data.retention=120h --web.listen-address=:9093 --web.external-url=http://ack-prometheus-operator-alertmanager.monitoring:9093 --web.route-prefix=/ --cluster.peer=alertmanager-ack-prometheus-operator-alertmanager-0.alertmanager-operated.monitoring.svc:6783
4026534465 pid 1 10841 1000 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --cluster.listen-address=[172.20.12.202]:6783 --storage.path=/alertmanager --data.retention=120h --web.listen-address=:9093 --web.external-url=http://ack-prometheus-operator-alertmanager.monitoring:9093 --web.route-prefix=/ --cluster.peer=alertmanager-ack-prometheus-operator-alertmanager-0.alertmanager-operated.monitoring.svc:6783
4026534468 mnt 1 12874 472 python -u /app/sidecar.py
4026534469 uts 1 12874 472 python -u /app/sidecar.py
4026534470 pid 1 12874 472 python -u /app/sidecar.py
... ip netns為什麼看不到namespace？
# ip netns list
我們跟蹤一下ip netns的執行過程，看到有目錄找不到
# strace -F -ff -t -tt -s 4096 -o ip.out ip netns list
1599206106.304532 execve("/usr/sbin/ip", ["ip", "netns", "list"], [/* 22 vars */]) = 0
1599206106.305867 brk(NULL) = 0x2592000
1599206106.305962 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f953edeb000
1599206106.306041 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
1599206106.306130 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
1599206106.306209 fstat(3, {st_mode=S_IFREG|0644, st_size=21765, ...}) = 0
1599206106.306277 mmap(NULL, 21765, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f953ede5000
1599206106.306337 close(3) = 0
1599206106.306398 open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
1599206106.306456 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\r\0\0\0\0\0\0@\0\0\0\0\0\0\0\30C\0\0\0\0\0\0\0\0\0\0@\0008\0\7\0@\0!\0 \0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\4\37\0\0\0\0\0\0\4\37\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0X-\0\0\0\0\0\0X- \0\0\0\0\0X- \0\0\0\0\0\20\3\0\0\0\0\0\0\270\3\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\210-\0\0\0\0\0\0\
210- \0\0\0\0\0\210- \0\0\0\0\0\20\2\0\0\0\0\0\0\20\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\310\1\0\0\0\0\0\0\310\1\0\0\0\0\0\0\310\1\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0P\34
5td\4\0\0\0@\31\0\0\0\0\0\0@\31\0\0\0\0\0\0@\31\0\0\0\0\0\0\274\0\0\0\0\0\0\0\274\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0X-\0\0\0\0\0\0X- \0\0\0\0\0X- \0\0\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0005v\223\310\361\364\235\223\1\fN1R\234\7\31
5\322\275=\10\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\33\0\0\0\32\0\0\0\2\0\0\0\7\0\0\0\230\2\21\0\200H\0\4\22\0\0@\203(\10\236\32\0\0\0\0\0\0\0\33\0\0\0\0\0\0\0\0\0\0\0\34\
0\0\0\0\0\0\0\35\0\0\0\0\0\0\0\36\0\0\0\0\0\0\0\37\0\0\0\0\0\0\0 \0\0\0\"\0\0\0#\0\0\0%\0\0\0&\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0'\0\0\0\0\0\0\0\0\0\0\0\353\26\251\30a\257\0\371\301S\
200\30\273\25sB\257\304M\17\221!\374\370\6\2\4\371\3733\373\17\371\31sB\372\31sB\225\263_\31\177\236\320\30a\242\222\6\5\350\7\371\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0=\1\0\0\22\0\0\0
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\375\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\337\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0 \0\0\0", 832) = 832
1599206106.306534 fstat(3, {st_mode=S_IFREG|0755, st_size=19288, ...}) = 0
1599206106.306593 mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f953e9c7000
1599206106.306647 mprotect(0x7f953e9c9000, 2097152, PROT_NONE) = 0
1599206106.306706 mmap(0x7f953ebc9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f953ebc9000
1599206106.306771 close(3) = 0
1599206106.306827 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
1599206106.306883 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340$\2\0\0\0\0\0@\0\0\0\0\0\0\0\370\301 \0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0L\0K\0\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0
\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\220\332\30\0\0\0\0\0\220\332\30\0\0\0\0\0\220\332\30\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\
5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\250\35\34\0\0\0\0\0\250\35\34\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0 '\34\0\0\0\0\0 '<\0\0\0\0\0 '<\0\0\0\0\0\200Q\0\0\0\0\0\0\300\232\0\0\0\0\0
\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\200[\34\0\0\0\0\0\200[<\0\0\0\0\0\200[<\0\0\0\0\0\360\1\0\0\0\0\0\0\360\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0
\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0 '\34\0\0\0\0\0 '<\0\0\0\0\0 '<\0\0\0\0\0\20\0\0\0\0\0\0\0\240\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0P\345td\4\0\0\0\254\332\30\0\0\0\0\0\254\332\30\0\0\
0\0\0\254\332\30\0\0\0\0\0\214j\0\0\0\0\0\0\214j\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\
0 '\34\0\0\0\0\0 '<\0\0\0\0\0 '<\0\0\0\0\0\3408\0\0\0\0\0\0\3408\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\246D\350e\241\366r\264j\0222\367\302u\365;\6\4&\226\4\0\0\0\20\0\0\0\1\0\0\0GNU
\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\363\3\0\0\7\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\300\0\10\0\5\200\0`\300\200\0\r\212\f\0\4\20\0\210D2\10.@\210T<, \0162H&\204\300\2
14\4\10\0\2\2\16\241\254\32\4f\300\0\3002\0\300\0P\1 \201\10\204\v ($\0\4 Z\0\20X\200\312DB(\0\6\200\20\30B\0 @\200\0\tP\0Q\212@\20\0\0\0\0\10\0\0\21\20", 832) = 832
1599206106.306950 fstat(3, {st_mode=S_IFREG|0755, st_size=2151672, ...}) = 0
1599206106.307013 mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f953e5fa000
1599206106.307066 mprotect(0x7f953e7bc000, 2097152, PROT_NONE) = 0
1599206106.307120 mmap(0x7f953e9bc000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7f953e9bc000
1599206106.307205 mmap(0x7f953e9c2000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f953e9c2000
1599206106.307260 close(3) = 0
1599206106.307312 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f953ede4000
1599206106.307381 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f953ede2000
1599206106.307438 arch_prctl(ARCH_SET_FS, 0x7f953ede2740) = 0
1599206106.307591 mprotect(0x7f953e9bc000, 16384, PROT_READ) = 0
1599206106.307648 mprotect(0x7f953ebc9000, 4096, PROT_READ) = 0
1599206106.307714 mprotect(0x669000, 4096, PROT_READ) = 0
1599206106.307789 mprotect(0x7f953edec000, 4096, PROT_READ) = 0
1599206106.307863 munmap(0x7f953ede5000, 21765) = 0
1599206106.307978 socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 3
1599206106.308052 setsockopt(3, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
1599206106.308108 setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
1599206106.308162 setsockopt(3, SOL_NETLINK, 11, [1], 4) = -1 ENOPROTOOPT (Protocol not available)
1599206106.308221 bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
1599206106.308284 getsockname(3, {sa_family=AF_NETLINK, pid=3965, groups=00000000}, [12]) = 0
1599206106.308367 open("/proc/self/ns/net", O_RDONLY) = 4
1599206106.308453 sendto(3, "\34\0\0\0Z\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\3\0\4\0\0\0", 28, 0, NULL, 0) = 28
1599206106.308530 recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\34\0\0\0X\0\0\0\0\0\0\0}\17\0\0\0\0\0\0\10\0\1\0\377\377\377\377", 16384}], msg_controllen=0, ms
g_flags=0}, 0) = 28
1599206106.308604 close(4) = 0
1599206106.308658 socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 4
1599206106.308708 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
1599206106.308754 setsockopt(4, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
1599206106.308800 setsockopt(4, SOL_NETLINK, 11, [1], 4) = -1 ENOPROTOOPT (Protocol not available)
1599206106.308847 bind(4, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
1599206106.308895 getsockname(4, {sa_family=AF_NETLINK, pid=-22296596, groups=00000000}, [12]) = 0
1599206106.308952 openat(AT_FDCWD, "/var/run/netns", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1599206106.309017 openat(AT_FDCWD, "/var/run/netns", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1599206106.309075 exit_group(0) = ?
1599206106.309181 +++ exited with 0 +++
預設 Docker 建立的網絡 namespace 不在ip netns讀取的預設路徑下（/var/run/netns） ，是以ip netns list直接看不到，需要 ln 軟連結一下。連結完畢以後，我們就可以通過 ip netns 指令操作了。
# docker ps |grep f9bc06a148c9
f9bc06a148c9 15bd4d827695 "/alibabacloud-log-c…" 6 weeks ago Up 6 weeks k8s_alibaba-log-controller_alibaba-log-controller-74d85bbf76-tzth9_kube-system_828b171d-9915-11ea-831b-e672548dc4da_1
# docker inspect '--format={{ .State.Pid }}' f9bc06a148c9
12428
# nsenter -t 12428 -n
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.12.223 netmask 255.255.255.255 broadcast 172.20.12.223
ether fa:ca:54:f6:59:69 txqueuelen 0 (Ethernet)
RX packets 426522 bytes 54197077 (51.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 435689 bytes 57149830 (54.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
軟鍊給ip netns讀取的目錄
# mkdir /var/run/netns
# ln -s /proc/12428/ns/net /var/run/netns/12428
# ip netns list
12428 (id: 23)
# ip netns exec 12428 ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.12.223 netmask 255.255.255.255 broadcast 172.20.12.223
ether fa:ca:54:f6:59:69 txqueuelen 0 (Ethernet)
RX packets 426497 bytes 54193478 (51.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 435664 bytes 57146029 (54.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 那麼問題來了：
預設lsns為什麼可以找到程序的namespace，ip netns為什麼不行？
![使用 ctypes 進行 Python 和 C 的混合程式設計[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)