PostgreSQL 逻辑复制、增量复制、逻辑订阅、增量订阅中间件 amazonriver - HelloBike开源

标签

PostgreSQL , 逻辑复制 , 流复制 , testdecoding , amazonriver , hellobike

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E8%83%8C%E6%99%AF 背景

amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务。由hellobike开源。

版本支持

Postgresql 9.4 or later    
Kafka 0.8 or later    
ElasticSearch 5.x    
           

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E6%9E%B6%E6%9E%84%E5%9B%BE 架构图

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%8E%9F%E7%90%86 原理

amazonriver 利用pg内部的逻辑复制功能,通过在pg创建逻辑复制槽,接收数据库的逻辑变更,通过解析test_decoding特定格式的消息,得到逻辑数据

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%AE%89%E8%A3%85%E4%BD%BF%E7%94%A8 安装使用

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%AE%89%E8%A3%85 安装

$git clone https://github.com/hellobike/amazonriver    
$cd amazonriver    
$glide install    
$go install          

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E4%BD%BF%E7%94%A8 使用

amazonriver -config config.json    
           

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#pg-%E9%85%8D%E7%BD%AE PG 配置

PG数据库需要预先开启逻辑复制

pg配置

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#amazonriver-%E9%85%8D%E7%BD%AE amazonriver 配置

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E7%9B%91%E6%8E%A7 监控

amazonriver支持使用prometheus来监控同步数据状态,

配置Grafana监控

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%90%8C%E6%AD%A5%E5%88%B0-elasticsearch 同步到 elasticsearch

同步到elasticsearch

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%90%8C%E6%AD%A5%E5%88%B0-kafka 同步到 kafka

同步到kafka

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95 性能测试

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E8%AE%B8%E5%8F%AF 许可

amazonriver 使用 Apache License 2 许可

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E6%84%9F%E8%B0%A2hellobike 感谢hellobike

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%8F%82%E8%80%83 参考

go语言版本

https://github.com/hellobike/amazonriver

java语言版本

https://github.com/hellobike/tunnel

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#logical-decoding%E5%9C%A8%E4%B8%BB%E4%BB%8E%E6%9E%B6%E6%9E%84%E4%B8%8B%E7%9A%84%E5%88%87%E6%8D%A2%E9%80%BB%E8%BE%91%E5%A4%84%E7%90%86%E9%97%AE%E9%A2%98 logical decoding在主从架构下的切换逻辑处理问题

1、判断slot是否存在

2、如果slot不存在，新建slot（这一步说明发生了切换，从库没有slot信息，新建slot，从新的slot继续消费。会有一些数据丢失。）

3、如果slot已经存在，从slot继续消费。

《PostgreSQL 使用逻辑decode实现异步主从切换后，时间线分歧变化量补齐、修复》 《PostgreSQL 暂时不支持replication slot信息的主从同步》 《PostgreSQL Failover slot - 支持将slot信息发送给物理备库》

0丢失还是需要内核层面支持slot failover，同时结合同步复制来实现。

https://github.com/digoal/blog/blob/master/201901/20190122_01.md#%E5%85%8D%E8%B4%B9%E9%A2%86%E5%8F%96%E9%98%BF%E9%87%8C%E4%BA%91rds-postgresql%E5%AE%9E%E4%BE%8Becs%E8%99%9A%E6%8B%9F%E6%9C%BA 免费领取阿里云RDS PostgreSQL实例、ECS虚拟机