環境介紹
在本例中一共部署了3個容器：
- front-envoy 容器作為 API 網關，所有的入向請求都通過 front-envoy 容器進行路由。front-envoy 容器暴露了 8080，8443 端口分别來接受 HTTP，HTTPS 請求，并根據路徑分别将它們路由到對應的服務上，以及通過 8001 端口來接受 Envoy 自帶的 admin 服務。
- 分别部署 service1 和 service2 兩個Flask 應用程式，在該容器中啟動 Envoy 服務， 通過 loopback 位址将請求路由到 Flask 應用程式。
service1 & service2 服務代碼
service1 和 service2 都使用相同的代碼啟動 Flask 服務，通過 SERVICE_NAME 這個環境變量在通路的時候可以區分服務是 service1 還是 service2 。
# service.py
from flask import Flask
from flask import request
import os
import requests
import socket
import sys
app = Flask(__name__)
@app.route('/service/<service_number>')
def hello(service_number):
return ('Hello from behind Envoy (service {})! hostname: {} resolved'
'hostname: {}\n'.format(os.environ['SERVICE_NAME'], socket.gethostname(),
socket.gethostbyname(socket.gethostname())))
if __name__ == "__main__":
app.run(host='127.0.0.1', port=8080, debug=True) service1 & service2 的 envoy 服務配置
在 service1 和 service2 容器中還啟動了 envoy 服務，外部用戶端（本例中是 front-envoy 容器）通路 service1 和 service2 時是去通路 envoy , 然後由 envoy 通過 loopback 位址 将請求路由到 Flask 應用程式。
# service-envoy.yaml
static_resources: #定義靜态資源
listeners: #監聽器
- address:
socket_address:
address: 0.0.0.0 #envoy監聽位址
port_value: 8000 #envoy監聽端口号
filter_chains:
- filters:
- name: cr7_filters #自定義filters的名字
typed_config:
#https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto #預設配置，connection manager 使用的編解碼器，自動适配HTTP/1.1和HTTP/2
stat_prefix: ingress_http #connection manager 發出統計資訊時使用的字首
route_config: #靜态配置路由管理，可選參數有3個：rds（通過RDS API動态加載）,route_config（靜态）,scoped_routes（根據請求參數比對路）
name: cr7_route #自定義路由配置名稱
virtual_hosts: #定義一組虛拟主機
- name: cr7_service #自定義虛拟主機名稱
domains: #比對所有域名
- "*"
routes:
- match:
prefix: "/service" #比對的URI路徑
route:
cluster: cr7_cluster #上遊叢集名稱
http_filters:
- name: envoy.filters.http.router #實作HTTP轉發
# - name: cr7-router
# typed_config:
# "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: cr7_cluster #自定義上遊叢集名稱
connect_timeout: 0.25s #新連接配接到上遊叢集主機的逾時時間
#https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#envoy-v3-api-enum-config-cluster-v3-cluster-discoverytype
type: strict_dns #服務發現機制：通過域名解析
lb_policy: round_robin #負載均衡政策
load_assignment: #設定負載均衡的成員， 取代了V2 API中的hosts字段
cluster_name: cr7_upstream
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1 #通過loopback轉發給本地flask服務
port_value: 8080
#https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto.html?highlight=access_log_path#envoy-v3-api-msg-config-bootstrap-v3-admin
admin: #envoy管理接口
access_log_path: "/tmp/access.log"
address:
socket_address:
address: 0.0.0.0
port_value: 8001 service1 & service2 Dockerfile 檔案
service1 和 service2 容器啟動腳本：首先啟動Flask 服務，然後啟動 envoy 服務。
# start_service.sh
#!/bin/sh
python3 /code/service.py &
envoy -c /etc/service-envoy.yaml --service-cluster "service${SERVICE_NAME}" #Dockerfile-service
FROM envoyproxy/envoy-alpine-dev:latest
RUN apk update && apk add py3-pip bash curl
RUN pip3 install -q Flask==0.11.1 requests==2.18.4
RUN mkdir /code
ADD ./service.py /code
ADD ./start_service.sh /usr/local/bin/start_service.sh
RUN chmod u+x /usr/local/bin/start_service.sh
ENTRYPOINT ["/bin/sh", "/usr/local/bin/start_service.sh"] front-envoy envoy 配置檔案
front-envoy 容器中隻有 envoy 服務， 負責接收所有入訪的流量，并且根據URI請求路徑分發給service1 或者 service2 。另外還配置了 HTTPS 加密，生成了證書和私鑰。
# front-envoy.yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
route:
cluster: service2
http_filters:
- name: envoy.filters.http.router
typed_config: {}
- address:
socket_address:
address: 0.0.0.0
port_value: 8443
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
route:
cluster: service2
http_filters:
- name: envoy.filters.http.router
typed_config: {}
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
# The following self-signed certificate pair is generated using:
# $ openssl req -x509 -newkey rsa:2048 -keyout a/front-proxy-key.pem -out a/front-proxy-crt.pem -days 3650 -nodes -subj '/CN=front-envoy'
#
# Instead of feeding it as an inline_string, certificate pair can also be fed to Envoy
# via filename. Reference: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#config-core-v3-datasource.
#
# Or in a dynamic configuration scenario, certificate pair can be fetched remotely via
# Secret Discovery Service (SDS). Reference: https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret.
certificate_chain:
inline_string: |
-----BEGIN CERTIFICATE-----
MIICqDCCAZACCQCquzpHNpqBcDANBgkqhkiG9w0BAQsFADAWMRQwEgYDVQQDDAtm
cm9udC1lbnZveTAeFw0yMDA3MDgwMTMxNDZaFw0zMDA3MDYwMTMxNDZaMBYxFDAS
BgNVBAMMC2Zyb250LWVudm95MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAthnYkqVQBX+Wg7aQWyCCb87hBce1hAFhbRM8Y9dQTqxoMXZiA2n8G089hUou
oQpEdJgitXVS6YMFPFUUWfwcqxYAynLK4X5im26Yfa1eO8La8sZUS+4Bjao1gF5/
VJxSEo2yZ7fFBo8M4E44ZehIIocipCRS+YZehFs6dmHoq/MGvh2eAHIa+O9xssPt
ofFcQMR8rwBHVbKy484O10tNCouX4yUkyQXqCRy6HRu7kSjOjNKSGtjfG+h5M8bh
10W7ZrsJ1hWhzBulSaMZaUY3vh5ngpws1JATQVSK1Jm/dmMRciwlTK7KfzgxHlSX
58ENpS7yPTISkEICcLbXkkKGEQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCmj6Hg
vwOxWz0xu+6fSfRL6PGJUGq6wghCfUvjfwZ7zppDUqU47fk+yqPIOzuGZMdAqi7N
v1DXkeO4A3hnMD22Rlqt25vfogAaZVToBeQxCPd/ALBLFrvLUFYuSlS3zXSBpQqQ
Ny2IKFYsMllz5RSROONHBjaJOn5OwqenJ91MPmTAG7ujXKN6INSBM0PjX9Jy4Xb9
zT+I85jRDQHnTFce1WICBDCYidTIvJtdSSokGSuy4/xyxAAc/BpZAfOjBQ4G1QRe
9XwOi790LyNUYFJVyeOvNJwveloWuPLHb9idmY5YABwikUY6QNcXwyHTbRCkPB2I
m+/R4XnmL4cKQ+5Z
-----END CERTIFICATE-----
private_key:
inline_string: |
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC2GdiSpVAFf5aD
tpBbIIJvzuEFx7WEAWFtEzxj11BOrGgxdmIDafwbTz2FSi6hCkR0mCK1dVLpgwU8
VRRZ/ByrFgDKcsrhfmKbbph9rV47wtryxlRL7gGNqjWAXn9UnFISjbJnt8UGjwzg
Tjhl6EgihyKkJFL5hl6EWzp2Yeir8wa+HZ4Achr473Gyw+2h8VxAxHyvAEdVsrLj
zg7XS00Ki5fjJSTJBeoJHLodG7uRKM6M0pIa2N8b6HkzxuHXRbtmuwnWFaHMG6VJ
oxlpRje+HmeCnCzUkBNBVIrUmb92YxFyLCVMrsp/ODEeVJfnwQ2lLvI9MhKQQgJw
tteSQoYRAgMBAAECggEAeDGdEkYNCGQLe8pvg8Z0ccoSGpeTxpqGrNEKhjfi6NrB
NwyVav10iq4FxEmPd3nobzDPkAftfvWc6hKaCT7vyTkPspCMOsQJ39/ixOk+jqFx
lNa1YxyoZ9IV2DIHR1iaj2Z5gB367PZUoGTgstrbafbaNY9IOSyojCIO935ubbcx
DWwL24XAf51ez6sXnI8V5tXmrFlNXhbhJdH8iIxNyM45HrnlUlOk0lCK4gmLJjy9
10IS2H2Wh3M5zsTpihH1JvM56oAH1ahrhMXs/rVFXXkg50yD1KV+HQiEbglYKUxO
eMYtfaY9i2CuLwhDnWp3oxP3HfgQQhD09OEN3e0IlQKBgQDZ/3poG9TiMZSjfKqL
xnCABMXGVQsfFWNC8THoW6RRx5Rqi8q08yJrmhCu32YKvccsOljDQJQQJdQO1g09
e/adJmCnTrqxNtjPkX9txV23Lp6Ak7emjiQ5ICu7iWxrcO3zf7hmKtj7z+av8sjO
mDI7NkX5vnlE74nztBEjp3eC0wKBgQDV2GeJV028RW3b/QyP3Gwmax2+cKLR9PKR
nJnmO5bxAT0nQ3xuJEAqMIss/Rfb/macWc2N/6CWJCRT6a2vgy6xBW+bqG6RdQMB
xEZXFZl+sSKhXPkc5Wjb4lQ14YWyRPrTjMlwez3k4UolIJhJmwl+D7OkMRrOUERO
EtUvc7odCwKBgBi+nhdZKWXveM7B5N3uzXBKmmRz3MpPdC/yDtcwJ8u8msUpTv4R
JxQNrd0bsIqBli0YBmFLYEMg+BwjAee7vXeDFq+HCTv6XMva2RsNryCO4yD3I359
XfE6DJzB8ZOUgv4Dvluie3TB2Y6ZQV/p+LGt7G13yG4hvofyJYvlg3RPAoGAcjDg
+OH5zLN2eqah8qBN0CYa9/rFt0AJ19+7/smLTJ7QvQq4g0gwS1couplcCEnNGWiK
72y1n/ckvvplmPeAE19HveMvR9UoCeV5ej86fACy8V/oVpnaaLBvL2aCMjPLjPP9
DWeCIZp8MV86cvOrGfngf6kJG2qZTueXl4NAuwkCgYEArKkhlZVXjwBoVvtHYmN2
o+F6cGMlRJTLhNc391WApsgDZfTZSdeJsBsvvzS/Nc0burrufJg0wYioTlpReSy4
ohhtprnQQAddfjHP7rh2LGt+irFzhdXXQ1ybGaGM9D764KUNCXLuwdly0vzXU4HU
q5sGxGrC1RECGB5Zwx2S2ZY=
-----END PRIVATE KEY-----
clusters:
- name: service1
connect_timeout: 0.25s
type: strict_dns #服務發現機制：通過域名解析
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: service1
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service1 # 通過 DNS 解析 service1 可以得到 service1 容器的 IP 位址
port_value: 8000
- name: service2
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: service2
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service2
port_value: 8000
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
layered_runtime:
layers:
- name: static_layer_0
static_layer:
envoy:
resource_limits:
listener:
example_listener_name:
connection_limit: 10000 front-envoy Dockerfile 檔案
# Dockerfile-frontenvoy
FROM envoyproxy/envoy-dev:latest
RUN apt-get update && apt-get -q install -y \
curl
COPY ./front-envoy.yaml /etc/front-envoy.yaml
RUN chmod go+r /etc/front-envoy.yaml
CMD ["/usr/local/bin/envoy", "-c", "/etc/front-envoy.yaml", "--service-cluster", "front-proxy"] docker-compose.yml 檔案
# docker-compose.yaml
version: "3.7"
services:
front-envoy:
build:
context: .
dockerfile: Dockerfile-frontenvoy
networks:
- envoymesh
expose:
- "8080"
- "8443"
- "8001"
ports:
- "8080:8080"
- "8443:8443"
- "8001:8001"
service1:
build:
context: .
dockerfile: Dockerfile-service
volumes:
- ./service-envoy.yaml:/etc/service-envoy.yaml
networks:
envoymesh:
aliases:
- service1
environment:
- SERVICE_NAME=1 #通過環境變量來區分服務
expose:
- "8000"
service2:
build:
context: .
dockerfile: Dockerfile-service
volumes:
- ./service-envoy.yaml:/etc/service-envoy.yaml
networks:
envoymesh:
aliases:
- service2
environment:
- SERVICE_NAME=2
expose:
- "8000"
networks:
envoymesh: {} 運作驗證
步驟一：安裝 Docker
確定你已安裝較新版本的 docker 和 docker-compose 。
步驟二：克隆倉庫
git clone https://github.com/cr7258/envoy-lab.git 步驟三：啟動所有容器
- up：啟動容器
- -d：在背景運作
- --build：重新建構鏡像
cd envoy-lab/front-proxy
docker-compose up -d --build 檢視容器：
[root@envoy ~]# docker-compose ps
Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------------
root_front-envoy_1 /docker-entrypoint.sh /usr ... Up 10000/tcp, 0.0.0.0:8001->8001/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8443->8443/tcp
root_service1_1 /bin/sh /usr/local/bin/sta ... Up 10000/tcp, 8000/tcp
root_service2_1 /bin/sh /usr/local/bin/sta ... Up 10000/tcp, 8000/tcp 步驟四：測試 Envoy 的路由能力
你現在可以通過 front-envoy 向兩個服務發送請求。
向 service1 發請求：
[root@envoy ~]# curl -v localhost:8080/service/1
* About to connect() to localhost port 8080 (#0)
* Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /service/1 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 89
< server: envoy
< date: Mon, 15 Mar 2021 15:29:28 GMT
< x-envoy-upstream-service-time: 2
<
Hello from behind Envoy (service 1)! hostname: e60ba6d0671c resolvedhostname: 172.18.0.2 向 service2 發請求：
[root@envoy ~]# curl -v localhost:8080/service/2
* About to connect() to localhost port 8080 (#0)
* Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /service/2 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 89
< server: envoy
< date: Mon, 15 Mar 2021 15:29:32 GMT
< x-envoy-upstream-service-time: 2
<
Hello from behind Envoy (service 2)! hostname: 9727cf7b9303 resolvedhostname: 172.18.0.4
* Connection #0 to host localhost left intact 能看到，每個請求在發送給前端 Envoy 後被正确路由到相應的應用程式。
我們也可以通過 HTTPS 請求前端 Envoy 後的服務。例如，向 service1：
[root@envoy ~]# curl https://localhost:8443/service/1 -k -v
* About to connect() to localhost port 8443 (#0)
* Trying ::1...
* Connected to localhost (::1) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=front-envoy
* start date: 7月 08 01:31:46 2020 GMT
* expire date: 7月 06 01:31:46 2030 GMT
* common name: front-envoy
* issuer: CN=front-envoy
> GET /service/1 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8443
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 89
< server: envoy
< date: Mon, 15 Mar 2021 15:30:10 GMT
< x-envoy-upstream-service-time: 2
<
Hello from behind Envoy (service 1)! hostname: e60ba6d0671c resolvedhostname: 172.18.0.2
* Connection #0 to host localhost left intact 步驟五：測試 Envoy 的負載均衡能力
現在增加 service1 的節點數量來示範 Envoy 的負載均衡能力：
[root@envoy ~]# docker-compose scale service1=3
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting root_service1_1 ... done
Creating root_service1_2 ... done
Creating root_service1_3 ... done 現在，如果我們多次向 service1 發送請求，前端 Envoy 将通過 round-robin 輪詢三台 service1 機器來實作負載均衡：
docker-compose exec -T front-envoy bash -c "\
curl -s http://localhost:8080/service/1 \
&& curl -s http://localhost:8080/service/1 \
&& curl -s http://localhost:8080/service/1" \
| grep Hello | grep "service 1"
# 傳回結果
Hello from behind Envoy (service 1)! hostname: 707d6f830af2 resolvedhostname: 172.18.0.5
Hello from behind Envoy (service 1)! hostname: 64eebf06b9db resolvedhostname: 172.18.0.6
Hello from behind Envoy (service 1)! hostname: e60ba6d0671c resolvedhostname: 172.18.0.2 步驟六：進入容器并 curl admin
當 Envoy 啟動時，也會同時啟動一個 admin 服務并綁定指定的端口。在示例配置中 admin 綁定到了 8001 端口。
我們可以通過 curl 它獲得有用的資訊：
[root@envoy ~]# curl localhost:8001/stats
cluster.service1.external.upstream_rq_200: 7
...
cluster.service1.membership_change: 2
cluster.service1.membership_total: 3
...
cluster.service1.upstream_cx_http2_total: 3
...
cluster.service1.upstream_rq_total: 7
...
cluster.service2.external.upstream_rq_200: 2
...
cluster.service2.membership_change: 1
cluster.service2.membership_total: 1
...
cluster.service2.upstream_cx_http2_total: 1
...
cluster.service2.upstream_rq_total: 2
... 能看到，我們可以擷取上遊叢集的成員數量，它們完成的請求數量，有關 http 入口的資訊以及大量其他有用的統計資料。