天天看点
返回

Docker Canal-adapter 1.15

参考文档: canal同步mysql数据到es中

注:

docker搞不定就下载安装包直接安装配置再去研究即可

去github下载自己想安装版本就行

​​canal-1.1.6​​

docker安装canal deployer admin adapter 过程中把conf配置目录同步到宿主机会出现容器崩溃的情况,建议把日志同步出来即可logs

拉取镜像 

docker pull slpcat/canal-adapter:v1.1.5

启动 

docker run -d -p 8081:8081 --name canal-adapter slpcat/canal-adapter:v1.1.5

复制配置文件到容器外

​

​docker cp canal-adapter:/opt/canal-adapter/conf /home/docker/canal-adapter/conf​

​

停止删除canal-adapter容器 

docker stop canal-adapter docker rm canal-adapter

启动容器 

docker run -d -p 8081:8081 \ -v /home/docker/canal-adapter/conf:/opt/canal-adapter/conf \ -v /home/docker/canal-adapter/logs:/opt/canal-adapter/logs \ --name canal-adapter \ slpcat/canal-adapter:v1.1.5

编辑application.yml

模板 

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp # 客户端的模式，可选tcp kafka rocketMQ
  flatMessage: true # 扁平message开关, 是否以json字符串形式投递数据, 仅在kafka/rocketMQ模式下有效
  zookeeperHosts: # 对应集群模式下的zk地址
  syncBatchSize: 1000 # 每次同步的批数量
  retries: 0 # 重试次数, -1为无限重试
  timeout: # 同步超时时间, 单位毫秒
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 114.115.132.128:11111 #设置canal-server的地址
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources: # 源数据库配置
    defaultDS:
      url: jdbc:mysql://114.115.132.128:3306/canal?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
      username: canal
      password: Eei82MDMHNZSRenf
  canalAdapters: # 适配器列表
    - instance: example # canal实例名或者MQ topic名
      groups: # 分组列表
        - groupId: g1 # 分组id, 如果是MQ模式将用到该值
          outerAdapters:
            - name: logger # 日志打印适配器
            - name: es7 # ES同步适配器
              hosts: http://114.115.132.128:9200 # ES连接地址
              properties:
                mode: rest # 模式可选transport(9300) 或者 rest(9200)
                # security.auth: test:123456 #  only used for rest mode
                cluster.name: elasticsearch # ES集群名称

移除注释后 

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null
canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    canal.tcp.server.host: 114.115.132.128:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://114.115.132.128:3306/canal?serverTimezone=GMT%2B8&useUnicode=true&characterEncoding=utf8&useSSL=false&allowMultiQueries=true
      username: canal
      password: Eei82MDMHNZSRenf     
  canalAdapters:
  - instance: demo
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger
      - name: es7
        hosts: 114.115.132.128:9300
        properties:
          mode: transport # or rest （transport模式，hosts中的端口要用9300；用rest模式，hosts中的端口要用9200，阿里云的es有些只支持rest）
          cluster.name: elasticsearch

cd es7

vim product.yml 

dataSourceKey: defaultDS
destination: product
groupId: g1
esMapping:
  _index: canal_product
  _id: id
  sql: "SELECT
  p.id,
  p.title,
  p.sub_title,
  p.price,
  p.pic
FROM`product` p"
  etlCondition: "where p.id > {0}"
  commitBatch: 30

vim uuid_tracknumber.yml 

dataSourceKey: defaultDS
destination: uuid_tracknumber
groupId: g1
esMapping:
  _index: uuid_tracknumber
  _type: _doc
  _id: _id
  sql: "
SELECT
  uuid,
  user_id,
  pkid_number,
 pkid_info
  FROM
  uuid_tracknumber
"
  commitBatch: 3000

异常记录

1.product.log 日志文件中，出现下面的内容，表示启动成功 

2022-09-29 09:12:52.813 [Thread-6] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - stop CannalInstance for null-product 2022-09-29 09:12:53.077 [Thread-6] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - stop successful....

java提示info级别的跳过即可,我还以为失败了呢.....

2.java内存不足 

# There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2147483648 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/admin/canal-server/bin/hs_err_pid1033.log 0.004: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 2147483648 bytes, attempted expansion amount: 2147483648 bytes] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000700000000, 2147483648, 0) failed; error='Cannot allocate memory' (errno=12)

3.Error When doing Register slave:ErrorPacket [errorNumber=1045, fieldCount=-1 mysql binlog权限未开放 

#MySQL Binlog权限需要三个权限 SELECT, REPLICATION SLAVE, REPLICATION CLIENT GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' IDENTIFIED BY 'canal'; FLUSH PRIVILEGES;#保存 #缺乏SELECT权限时，报错为 #com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: #Access denied for user 'canal'@'%' to database 'binlog'

4.Failed to bind properties under 'canal.conf' to com.alibaba.otter.canal.adapter.launcher.config.AdapterCanalConfig: 

Reason: Unable to set value for property src-data-sources

Action:

Update your application's configuration

​

​下载mysql-connector-java-5.1.49.jar [root@localhost lib]# chmod -R 777 mysql-connector-java-5.1.49.jar [root@localhost lib]# chmod +st mysql-connector-java-5.1.49.jar​

​

5.修改配置后未生效

6.canal adapter http: Name or service not known 还有ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed 

- name: es7
        key: es-test
        hosts: http://127.0.0.1:9200 # 127.0.0.1:9200 for rest mode
        properties:
          mode: rest # transport # or rest
#          # security.auth: test:123456 #  only used for rest mode
          cluster.name: es      
- name: es7
        key: es-test
        hosts: 127.0.0.1:9200 # 127.0.0.1:9200 for rest mode
        properties:
          mode: rest # transport # or rest
#          # security.auth: test:123456 #  only used for rest mode
          cluster.name: es
Docker 容器 canal mysql java
上一篇: 《解读window核心编程》 之 内核对象
下一篇: 通过实例理解JavaScript中的arguments,call,apply,callee及caller

继续阅读