概述
資料分發主要包含兩種方法：跨賬号多目标Logstore的分發和動态目标Logstore的分發, 其特點如下表. 需要時也可以合并兩種方法.
注意 目前資料加工僅支援同Region下分發
| 方法 | 優點 | 缺點 |
|---|---|---|
| 配置多個目标Logstore | 支援不同賬号(AK)的目标Logstore | 多個目标時, 配置與使用較為繁瑣. 目标一般在代碼中靜态指定. 目前最多20個目标. |
| 配置少量目标, 再代碼中重置project, logstore參數 | 目标project, logstore可以在LOG DSL規則中動态擷取與設定. 可以發送超過20個以上目标. | 秘鑰使用目标中的配置, 不能動态改. 是以最多跨20個賬号. |
場景1：跨賬号多目标Logstore的分發
原始日志
"""
以下日志都存儲在同一個logstore中，該logstore的預設邏輯名為target0
"""
"日志1"
http_host: m1.abcd.com
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"日志2"
http_host: m2.abcd.com
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"日志3"
http_host: m3.abcd.com
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"日志4"
http_host: m4.abcd.com
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https 分發目标
-
http_status
2XX
target0
success_event
-
http_status
3XX
target1
redirection_event
-
http_status
4XX
target2
unauthorized_event
-
http_status
5XX
target3
internal_server_error_event
LOG DSL編排
e_switch(e_match("status", r"2\d+"), e_set("__topic__", "success_event"),
e_match("status", r"3\d+"), e_compose(e_set("__topic__", "redirection_event"), e_output("target1")),
e_match("status", r"4\d+"), e_compose(e_set("__topic__", "unauthorized_event"), e_output("target2")),
e_match("status", r"5\d+"), e_compose(e_set("__topic__", "internal_server_error_event`"), e_output("target3"))
) 加工後的日志
"""
源logstore: target0
"""
__topic__: success_event
http_host: m1.abcd.com
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"""
目标logstore: target1
"""
__topic__: redirection_event
http_host: m2.abcd.com
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"""
目标logstore: target2
"""
__topic__: unauthorized_event
http_host: m3.abcd.com
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"""
目标logstore: target3
"""
__topic__: internal_server_error_event
http_host: m4.abcd.com
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https - 不同Logstore的邏輯名(target0, target1等）可在LOG DSL規則配置項中的
存儲目标
AccessKey
日志服務資料加工最佳實踐: 跨賬号多目标logstore資料分發進一步參考
- 調用
e_output
e_coutput
場景2：動态目标Logstore的分發
'''
project1和project2屬于同一賬号
project1下有兩個Logstore：logstore1, logstore2
project2中也有兩個Logstore：logstore1, lostore2
'''
"日志1"
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"日志2"
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"日志3"
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"日志4"
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https - 根據日志事件
project
logstore
- 為日志事件添加标簽
__tag:__type
dynamic_dispatch
e_output(project=v("project"), logstore=v("logstore"), tags={"type": "dynamic_dispatch"}) - 動态目标分發預設使用的AccessKey資訊是加工配置項中配置的第一個存儲目标對應的AccessKey賬号資訊。
- 是以第一個存儲目标的project和logstore資訊對以上加工規則不會有影響，因為該加工規則中的
e_output
加工後日志
"project1 logstore1"
__tag__:type: dynamic_dispatch
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"project1 logstore2"
__tag__:type: dynamic_dispatch
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"project2 logstore1"
__tag__:type: dynamic_dispatch
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"project2 logstore2"
__tag__:type: dynamic_dispatch
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https 場景3：跨賬号動态目标Logstore的分發
本場景是場景1和場景2的結合使用場景。
"""
project1屬于賬号1，該Project中有兩個Logstore：logstore1, lostore2
project2屬于賬号2，該Project中有兩個Logstore：logstore1, lostore2
"""
"日志1"
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"日志2"
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"日志3"
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"日志4"
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https -
project
logstore
- 分發的目标屬于不同的賬号, project1屬于賬号1，project2屬于賬号2。
e_switch(e_match(v("project"), "project1"), e_output(name="target0", project=v("project"), logstore=v("logstore")),
e_match(v("project"), "project2"), e_output(name="target1", project=v("project"), logstore=v("logstore")))
- 在任務配置項中為存儲目标
target0
target1
- 存儲目标
target0
target1
e_output
"""
賬号1
project1 logstore1
"""
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"""
賬号1
project1 logstore2
"""
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"""
賬号2
project2 logstore1
"""
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"""
賬号2
project2 logstore2
"""
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https 進一步參考
歡迎掃碼加入官方釘釘群獲得實時更新與阿裡雲工程師的及時直接的支援: