概述
数据分发主要包含两种方法：跨账号多目标Logstore的分发和动态目标Logstore的分发, 其特点如下表. 需要时也可以合并两种方法.
注意 目前数据加工仅支持同Region下分发
| 方法 | 优点 | 缺点 |
|---|---|---|
| 配置多个目标Logstore | 支持不同账号(AK)的目标Logstore | 多个目标时, 配置与使用较为繁琐. 目标一般在代码中静态指定. 目前最多20个目标. |
| 配置少量目标, 再代码中重置project, logstore参数 | 目标project, logstore可以在LOG DSL规则中动态获取与设置. 可以发送超过20个以上目标. | 秘钥使用目标中的配置, 不能动态改. 因此最多跨20个账号. |
场景1：跨账号多目标Logstore的分发
原始日志
"""
以下日志都存储在同一个logstore中，该logstore的默认逻辑名为target0
"""
"日志1"
http_host: m1.abcd.com
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"日志2"
http_host: m2.abcd.com
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"日志3"
http_host: m3.abcd.com
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"日志4"
http_host: m4.abcd.com
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https 分发目标
-
http_status
2XX
target0
success_event
-
http_status
3XX
target1
redirection_event
-
http_status
4XX
target2
unauthorized_event
-
http_status
5XX
target3
internal_server_error_event
LOG DSL编排
e_switch(e_match("status", r"2\d+"), e_set("__topic__", "success_event"),
e_match("status", r"3\d+"), e_compose(e_set("__topic__", "redirection_event"), e_output("target1")),
e_match("status", r"4\d+"), e_compose(e_set("__topic__", "unauthorized_event"), e_output("target2")),
e_match("status", r"5\d+"), e_compose(e_set("__topic__", "internal_server_error_event`"), e_output("target3"))
) 加工后的日志
"""
源logstore: target0
"""
__topic__: success_event
http_host: m1.abcd.com
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"""
目标logstore: target1
"""
__topic__: redirection_event
http_host: m2.abcd.com
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"""
目标logstore: target2
"""
__topic__: unauthorized_event
http_host: m3.abcd.com
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"""
目标logstore: target3
"""
__topic__: internal_server_error_event
http_host: m4.abcd.com
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https - 不同Logstore的逻辑名(target0, target1等）可在LOG DSL规则配置项中的
存储目标
AccessKey
日志服务数据加工最佳实践: 跨账号多目标logstore数据分发进一步参考
- 调用
e_output
e_coutput
场景2：动态目标Logstore的分发
'''
project1和project2属于同一账号
project1下有两个Logstore：logstore1, logstore2
project2中也有两个Logstore：logstore1, lostore2
'''
"日志1"
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"日志2"
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"日志3"
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"日志4"
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https - 根据日志事件
project
logstore
- 为日志事件添加标签
__tag:__type
dynamic_dispatch
e_output(project=v("project"), logstore=v("logstore"), tags={"type": "dynamic_dispatch"}) - 动态目标分发默认使用的AccessKey信息是加工配置项中配置的第一个存储目标对应的AccessKey账号信息。
- 因此第一个存储目标的project和logstore信息对以上加工规则不会有影响，因为该加工规则中的
e_output
加工后日志
"project1 logstore1"
__tag__:type: dynamic_dispatch
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"project1 logstore2"
__tag__:type: dynamic_dispatch
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"project2 logstore1"
__tag__:type: dynamic_dispatch
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"project2 logstore2"
__tag__:type: dynamic_dispatch
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https 场景3：跨账号动态目标Logstore的分发
本场景是场景1和场景2的结合使用场景。
"""
project1属于账号1，该Project中有两个Logstore：logstore1, lostore2
project2属于账号2，该Project中有两个Logstore：logstore1, lostore2
"""
"日志1"
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"日志2"
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"日志3"
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"日志4"
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https -
project
logstore
- 分发的目标属于不同的账号, project1属于账号1，project2属于账号2。
e_switch(e_match(v("project"), "project1"), e_output(name="target0", project=v("project"), logstore=v("logstore")),
e_match(v("project"), "project2"), e_output(name="target1", project=v("project"), logstore=v("logstore")))
- 在任务配置项中为存储目标
target0
target1
- 存储目标
target0
target1
e_output
"""
账号1
project1 logstore1
"""
host: a.b.c.com
project: project1
logstore: logstore1
http_status: 200
request_method: GET
request_uri: /pic/icon.jpg
scheme: https
"""
账号1
project1 logstore2
"""
host: m.n.q.com
project: project1
logstore: logstore2
http_status: 301
request_method: POST
request_uri: /data/data.php
scheme: http
"""
账号2
project2 logstore1
"""
host: e.f.d.com
project: project2
logstore: logstore1
http_status: 404
request_method: GET
request_uri: /category/abc/product_id
scheme: https
"""
账号2
project2 logstore2
"""
host: p.q.t.com
project: project2
logstore: logstore2
http_status: 504
request_method: GET
request_uri: /data/index.html
scheme: https 进一步参考
欢迎扫码加入官方钉钉群获得实时更新与阿里云工程师的及时直接的支持: