背景
CTE 遞歸文法是PG 8.4引入的功能, 至今已經10多年, 非常文檔.
CTE 遞歸可以解決很多問題: 時序場景取所有傳感器最新的value, 圖式資料的搜尋(一度人脈,N度人脈,最近的路徑關系), 樹狀資料的累加分析, 知識圖譜, 去稀疏資料的唯一值等.
使用CTE遞歸比通用的方法通常有數百倍的性能提升.
https://github.com/digoal/blog/blob/master/202105/20210529_01.md#%E7%94%A8%E4%BE%8B 用例
假設傳感器有1萬個, 每個傳感器每秒上傳一條記錄.
取出今天處于活躍狀态(有資料)的傳感器的最後一個值.
1、建立測試表
create unlogged table tbl_sensor_log (
id serial8 ,
sid int, -- 傳感器ID (例如 網約車、警車、巡邏車、共享單車、物聯網傳感器等裝置)
val jsonb, -- 傳感器上傳的資料
crt_time timestamp -- 上傳時間
)
partition by range (crt_time)
;
2、建立分區
do language plpgsql $$
declare
begin
for i in 0..365 loop
execute format($_$
create unlogged table tbl_sensor_log_%s PARTITION of tbl_sensor_log
for values from (%L) to (%L)
$_$, to_char(current_date+i, 'yyyymmdd'), current_date+i, current_date+i+1);
end loop;
end;
$$;
3、建立索引
insert into tbl_sensor_log (sid, val, crt_time)
select random()*10000 , row_to_json(row(random(),random(),clock_timestamp())), current_date+(round(random()::numeric*72::numeric,2)||' hour')::interval
from generate_series(1,50000000);
4、寫入5000萬條記錄, 均勻分布在最近3天的分區内
insert into tbl_sensor_log (sid, val, crt_time)
select random()*10000 , row_to_json(row(random(),random(),clock_timestamp())), current_date+(round(random()::numeric*72::numeric,2)||' hour')::interval
from generate_series(1,50000000);
方法1: 使用傳統的視窗查詢
select id,sid,val,crt_time from
(
select *, row_number() over w as RN
from tbl_sensor_log
where crt_time >= current_date and crt_time < current_date+1
window w as (partition by sid order by crt_time desc)
) t
where rn=1;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Subquery Scan on t (cost=5057407.30..5598899.49 rows=83306 width=119) (actual time=40763.507..52716.622 rows=10001 loops=1)
Filter: (t.rn = 1)
Rows Removed by Filter: 16653784
-> WindowAgg (cost=5057407.30..5390633.26 rows=16661298 width=127) (actual time=40763.503..51533.043 rows=16663785 loops=1)
-> Sort (cost=5057407.30..5099060.55 rows=16661298 width=119) (actual time=40763.483..44945.556 rows=16663785 loops=1)
Sort Key: tbl_sensor_log.sid, tbl_sensor_log.crt_time DESC
Sort Method: external merge Disk: 2177080kB
-> Append (cost=0.00..1257703.57 rows=16661298 width=119) (actual time=0.065..5597.541 rows=16663785 loops=1)
Subplans Removed: 365
-> Seq Scan on tbl_sensor_log_20210529 tbl_sensor_log_1 (cost=0.00..683635.38 rows=16660933 width=119) (actual time=0.064..4066.655 rows=16663785 loops=1)
Filter: ((crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
Planning Time: 219.559 ms
Execution Time: 53133.463 ms
(13 rows)
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Subquery Scan on t (cost=65.16..10962058.77 rows=83306 width=119) (actual time=0.041..104082.386 rows=10001 loops=1)
Filter: (t.rn = 1)
Rows Removed by Filter: 16653784
-> WindowAgg (cost=65.16..10753792.55 rows=16661298 width=127) (actual time=0.040..102532.935 rows=16663785 loops=1)
-> Merge Append (cost=65.16..10462219.83 rows=16661298 width=119) (actual time=0.029..92266.387 rows=16663785 loops=1)
Sort Key: tbl_sensor_log.sid, tbl_sensor_log.crt_time DESC
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 tbl_sensor_log_1 (cost=0.44..9177871.39 rows=16660933 width=119) (actual time=0.029..89908.207 rows=16663785 loops=1)
Index Cond: ((crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
Planning Time: 39.511 ms
Execution Time: 104088.128 ms
(11 rows)
方法2: 使用索引連結清單跳跳糖
遞歸, 每次掃描定位到1個目标SID, 然後跳到第二個SID, 而不是通過索引連結清單順序掃描.
連結清單順序掃描的缺點: 整張索引的時間範圍内的所有葉子結點的每個page都要掃描到, 性能爛到家.
with recursive tmp as (
(select t from tbl_sensor_log as t where crt_time >= current_date and crt_time < current_date+1 order by sid, crt_time desc limit 1)
union all
select (select tbl_sensor_log from tbl_sensor_log where sid>(tmp.t).sid
and crt_time >= current_date and crt_time < current_date+1 order by sid, crt_time desc limit 1) as t
from tmp where tmp.* is not null
)
select (tmp.t).* from tmp
where tmp.* is not null;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CTE Scan on tmp (cost=6650.50..6652.52 rows=100 width=52)
Filter: (tmp.* IS NOT NULL)
CTE tmp
-> Recursive Union (cost=65.16..6650.50 rows=101 width=32)
-> Subquery Scan on "*SELECT* 1" (cost=65.16..65.80 rows=1 width=32)
-> Limit (cost=65.16..65.79 rows=1 width=44)
-> Merge Append (cost=65.16..10462219.83 rows=16661298 width=44)
Sort Key: t.sid, t.crt_time DESC
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 t_1 (cost=0.44..9177871.39 rows=16660933 width=44)
Index Cond: ((crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
-> WorkTable Scan on tmp tmp_1 (cost=0.00..658.27 rows=10 width=32)
Filter: (tmp_1.* IS NOT NULL)
(13 rows)
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CTE Scan on tmp (cost=6650.50..6652.52 rows=100 width=52) (actual time=0.036..124.680 rows=10001 loops=1)
Filter: (tmp.* IS NOT NULL)
Rows Removed by Filter: 1
CTE tmp
-> Recursive Union (cost=65.16..6650.50 rows=101 width=32) (actual time=0.032..119.486 rows=10002 loops=1)
-> Subquery Scan on "*SELECT* 1" (cost=65.16..65.80 rows=1 width=32) (actual time=0.031..0.033 rows=1 loops=1)
-> Limit (cost=65.16..65.79 rows=1 width=44) (actual time=0.031..0.032 rows=1 loops=1)
-> Merge Append (cost=65.16..10462219.83 rows=16661298 width=44) (actual time=0.030..0.030 rows=1 loops=1)
Sort Key: t.sid, t.crt_time DESC
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 t_1 (cost=0.44..9177871.39 rows=16660933 width=44) (actual time=0.029..0.030 rows=1 loops=1)
Index Cond: ((crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
-> WorkTable Scan on tmp tmp_1 (cost=0.00..658.27 rows=10 width=32) (actual time=0.011..0.011 rows=1 loops=10002)
Filter: (tmp_1.* IS NOT NULL)
Rows Removed by Filter: 0
SubPlan 1
-> Limit (cost=65.16..65.81 rows=1 width=44) (actual time=0.011..0.011 rows=1 loops=10001)
-> Merge Append (cost=65.16..3572009.02 rows=5554009 width=44) (actual time=0.011..0.011 rows=1 loops=10001)
Sort Key: tbl_sensor_log.sid, tbl_sensor_log.crt_time DESC
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 tbl_sensor_log_1 (cost=0.44..3115515.85 rows=5553644 width=44) (actual time=0.010..0.010 rows=1 loops=10001)
Index Cond: ((sid > (tmp_1.t).sid) AND (crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
Planning Time: 69.016 ms
Execution Time: 125.866 ms
(24 rows)
方法3: 使用subquery
但是, 需要維護一張SID表, 實際業務邏輯可能比這複雜, SID表可能沒有這麼好維護.
而且還有1個問題: 今天沒有活躍的SID也會被查出來. 如果選擇過濾今天不活躍的記錄, 需要多次評估, 性能就會下降.
create table tbl_sensor (sid int primary key);
insert into tbl_sensor select generate_series(0,10010);
select (select tbl_sensor_log from tbl_sensor_log where sid=t.sid and crt_time >= current_date and crt_time < current_date+1 order by crt_time desc limit 1) as val
from tbl_sensor as t;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using tbl_sensor_pkey on tbl_sensor t (cost=0.29..509812.03 rows=10011 width=32)
SubPlan 1
-> Limit (cost=49.58..50.91 rows=1 width=40)
-> Append (cost=49.58..2751.07 rows=2036 width=40)
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 tbl_sensor_log_1 (cost=0.44..1880.54 rows=1671 width=40)
Index Cond: ((sid = t.sid) AND (crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
(7 rows)
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using tbl_sensor_pkey on tbl_sensor t (cost=0.29..509812.03 rows=10011 width=32) (actual time=0.039..116.315 rows=10011 loops=1)
Heap Fetches: 0
SubPlan 1
-> Limit (cost=49.58..50.91 rows=1 width=40) (actual time=0.011..0.011 rows=1 loops=10011)
-> Append (cost=49.58..2751.07 rows=2036 width=40) (actual time=0.011..0.011 rows=1 loops=10011)
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 tbl_sensor_log_1 (cost=0.44..1880.54 rows=1671 width=40) (actual time=0.010..0.010 rows=1 loops=10011)
Index Cond: ((sid = t.sid) AND (crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
Planning Time: 40.798 ms
Execution Time: 117.059 ms
(10 rows)
過濾不活躍的SID将增加計算量, 性能有下降.
select * from
(
select (select tbl_sensor_log from tbl_sensor_log where sid=t.sid and crt_time >= current_date and crt_time < current_date+1 order by crt_time desc limit 1) as val
from tbl_sensor as t
) t1
where t1.val is not null;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using tbl_sensor_pkey on tbl_sensor t (cost=0.29..1016895.27 rows=9961 width=32) (actual time=0.044..173.412 rows=10001 loops=1)
Filter: ((SubPlan 2) IS NOT NULL)
Rows Removed by Filter: 10
Heap Fetches: 0
SubPlan 1
-> Limit (cost=49.58..50.91 rows=1 width=40) (actual time=0.007..0.007 rows=1 loops=10001)
-> Append (cost=49.58..2751.07 rows=2036 width=40) (actual time=0.007..0.007 rows=1 loops=10001)
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 tbl_sensor_log_1 (cost=0.44..1880.54 rows=1671 width=40) (actual time=0.006..0.006 rows=1 loops=10001)
Index Cond: ((sid = t.sid) AND (crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
SubPlan 2
-> Limit (cost=49.58..50.91 rows=1 width=40) (actual time=0.010..0.010 rows=1 loops=10011)
-> Append (cost=49.58..2751.07 rows=2036 width=40) (actual time=0.010..0.010 rows=1 loops=10011)
Subplans Removed: 365
-> Index Scan using tbl_sensor_log_20210529_sid_crt_time_idx on tbl_sensor_log_20210529 tbl_sensor_log_3 (cost=0.44..1880.54 rows=1671 width=40) (actual time=0.009..0.009 rows=1 loops=10011)
Index Cond: ((sid = t.sid) AND (crt_time >= CURRENT_DATE) AND (crt_time < (CURRENT_DATE + 1)))
Planning Time: 68.114 ms
Execution Time: 174.239 ms
(18 rows)