PostgreSQL 邏輯複制、增量複制、邏輯訂閱、增量訂閱中間件 amazonriver - HelloBike開源

标簽

PostgreSQL , 邏輯複制 , 流複制 , testdecoding , amazonriver , hellobike

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E8%83%8C%E6%99%AF 背景

amazonriver 是一個将postgresql的實時資料同步到es或kafka的服務。由hellobike開源。

版本支援

Postgresql 9.4 or later    
Kafka 0.8 or later    
ElasticSearch 5.x    
           

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E6%9E%B6%E6%9E%84%E5%9B%BE 架構圖

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%8E%9F%E7%90%86 原理

amazonriver 利用pg内部的邏輯複制功能,通過在pg建立邏輯複制槽,接收資料庫的邏輯變更,通過解析test_decoding特定格式的消息,得到邏輯資料

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%AE%89%E8%A3%85%E4%BD%BF%E7%94%A8 安裝使用

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%AE%89%E8%A3%85 安裝

$git clone https://github.com/hellobike/amazonriver    
$cd amazonriver    
$glide install    
$go install          

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E4%BD%BF%E7%94%A8 使用

amazonriver -config config.json    
           

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#pg-%E9%85%8D%E7%BD%AE PG 配置

PG資料庫需要預先開啟邏輯複制

pg配置

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#amazonriver-%E9%85%8D%E7%BD%AE amazonriver 配置

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E7%9B%91%E6%8E%A7 監控

amazonriver支援使用prometheus來監控同步資料狀态,

配置Grafana監控

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%90%8C%E6%AD%A5%E5%88%B0-elasticsearch 同步到 elasticsearch

同步到elasticsearch

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%90%8C%E6%AD%A5%E5%88%B0-kafka 同步到 kafka

同步到kafka

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95 性能測試

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E8%AE%B8%E5%8F%AF 許可

amazonriver 使用 Apache License 2 許可

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E6%84%9F%E8%B0%A2hellobike 感謝hellobike

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%8F%82%E8%80%83 參考

go語言版本

https://github.com/hellobike/amazonriver

java語言版本

https://github.com/hellobike/tunnel

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#logical-decoding%E5%9C%A8%E4%B8%BB%E4%BB%8E%E6%9E%B6%E6%9E%84%E4%B8%8B%E7%9A%84%E5%88%87%E6%8D%A2%E9%80%BB%E8%BE%91%E5%A4%84%E7%90%86%E9%97%AE%E9%A2%98 logical decoding在主從架構下的切換邏輯處理問題

1、判斷slot是否存在

2、如果slot不存在，建立slot（這一步說明發生了切換，從庫沒有slot資訊，建立slot，從新的slot繼續消費。會有一些資料丢失。）

3、如果slot已經存在，從slot繼續消費。

《PostgreSQL 使用邏輯decode實作異步主從切換後，時間線分歧變化量補齊、修複》 《PostgreSQL 暫時不支援replication slot資訊的主從同步》 《PostgreSQL Failover slot - 支援将slot資訊發送給實體備庫》

0丢失還是需要核心層面支援slot failover，同時結合同步複制來實作。

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%85%8D%E8%B4%B9%E9%A2%86%E5%8F%96%E9%98%BF%E9%87%8C%E4%BA%91rds-postgresql%E5%AE%9E%E4%BE%8Becs%E8%99%9A%E6%8B%9F%E6%9C%BA 免費領取阿裡雲RDS PostgreSQL執行個體、ECS虛拟機