天天看點
傳回

Docker Canal-adapter 1.15

參考文檔: canal同步mysql資料到es中

注:

docker搞不定就下載下傳安裝包直接安裝配置再去研究即可

去github下載下傳自己想安裝版本就行

​​canal-1.1.6​​

docker安裝canal deployer admin adapter 過程中把conf配置目錄同步到主控端會出現容器崩潰的情況,建議把日志同步出來即可logs

拉取鏡像 

docker pull slpcat/canal-adapter:v1.1.5

啟動 

docker run -d -p 8081:8081 --name canal-adapter slpcat/canal-adapter:v1.1.5

複制配置檔案到容器外

​

​docker cp canal-adapter:/opt/canal-adapter/conf /home/docker/canal-adapter/conf​

​

停止删除canal-adapter容器 

docker stop canal-adapter docker rm canal-adapter

啟動容器 

docker run -d -p 8081:8081 \ -v /home/docker/canal-adapter/conf:/opt/canal-adapter/conf \ -v /home/docker/canal-adapter/logs:/opt/canal-adapter/logs \ --name canal-adapter \ slpcat/canal-adapter:v1.1.5

編輯application.yml

模闆 

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp # 用戶端的模式，可選tcp kafka rocketMQ
  flatMessage: true # 扁平message開關, 是否以json字元串形式投遞資料, 僅在kafka/rocketMQ模式下有效
  zookeeperHosts: # 對應叢集模式下的zk位址
  syncBatchSize: 1000 # 每次同步的批數量
  retries: 0 # 重試次數, -1為無限重試
  timeout: # 同步逾時時間, 機關毫秒
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 114.115.132.128:11111 #設定canal-server的位址
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources: # 源資料庫配置
    defaultDS:
      url: jdbc:mysql://114.115.132.128:3306/canal?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
      username: canal
      password: Eei82MDMHNZSRenf
  canalAdapters: # 擴充卡清單
    - instance: example # canal執行個體名或者MQ topic名
      groups: # 分組清單
        - groupId: g1 # 分組id, 如果是MQ模式将用到該值
          outerAdapters:
            - name: logger # 日志列印擴充卡
            - name: es7 # ES同步擴充卡
              hosts: http://114.115.132.128:9200 # ES連接配接位址
              properties:
                mode: rest # 模式可選transport(9300) 或者 rest(9200)
                # security.auth: test:123456 #  only used for rest mode
                cluster.name: elasticsearch # ES叢集名稱

移除注釋後 

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null
canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    canal.tcp.server.host: 114.115.132.128:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://114.115.132.128:3306/canal?serverTimezone=GMT%2B8&useUnicode=true&characterEncoding=utf8&useSSL=false&allowMultiQueries=true
      username: canal
      password: Eei82MDMHNZSRenf     
  canalAdapters:
  - instance: demo
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger
      - name: es7
        hosts: 114.115.132.128:9300
        properties:
          mode: transport # or rest （transport模式，hosts中的端口要用9300；用rest模式，hosts中的端口要用9200，阿裡雲的es有些隻支援rest）
          cluster.name: elasticsearch

cd es7

vim product.yml 

dataSourceKey: defaultDS
destination: product
groupId: g1
esMapping:
  _index: canal_product
  _id: id
  sql: "SELECT
  p.id,
  p.title,
  p.sub_title,
  p.price,
  p.pic
FROM`product` p"
  etlCondition: "where p.id > {0}"
  commitBatch: 30

vim uuid_tracknumber.yml 

dataSourceKey: defaultDS
destination: uuid_tracknumber
groupId: g1
esMapping:
  _index: uuid_tracknumber
  _type: _doc
  _id: _id
  sql: "
SELECT
  uuid,
  user_id,
  pkid_number,
 pkid_info
  FROM
  uuid_tracknumber
"
  commitBatch: 3000

異常記錄

1.product.log 日志檔案中，出現下面的内容，表示啟動成功 

2022-09-29 09:12:52.813 [Thread-6] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - stop CannalInstance for null-product 2022-09-29 09:12:53.077 [Thread-6] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - stop successful....

java提示info級别的跳過即可,我還以為失敗了呢.....

2.java記憶體不足 

# There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2147483648 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/admin/canal-server/bin/hs_err_pid1033.log 0.004: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 2147483648 bytes, attempted expansion amount: 2147483648 bytes] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000700000000, 2147483648, 0) failed; error='Cannot allocate memory' (errno=12)

3.Error When doing Register slave:ErrorPacket [errorNumber=1045, fieldCount=-1 mysql binlog權限未開放 

#MySQL Binlog權限需要三個權限 SELECT, REPLICATION SLAVE, REPLICATION CLIENT GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' IDENTIFIED BY 'canal'; FLUSH PRIVILEGES;#儲存 #缺乏SELECT權限時，報錯為 #com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: #Access denied for user 'canal'@'%' to database 'binlog'

4.Failed to bind properties under 'canal.conf' to com.alibaba.otter.canal.adapter.launcher.config.AdapterCanalConfig: 

Reason: Unable to set value for property src-data-sources

Action:

Update your application's configuration

​

​下載下傳mysql-connector-java-5.1.49.jar [root@localhost lib]# chmod -R 777 mysql-connector-java-5.1.49.jar [root@localhost lib]# chmod +st mysql-connector-java-5.1.49.jar​

​

5.修改配置後未生效

6.canal adapter http: Name or service not known 還有ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed 

- name: es7
        key: es-test
        hosts: http://127.0.0.1:9200 # 127.0.0.1:9200 for rest mode
        properties:
          mode: rest # transport # or rest
#          # security.auth: test:123456 #  only used for rest mode
          cluster.name: es      
- name: es7
        key: es-test
        hosts: 127.0.0.1:9200 # 127.0.0.1:9200 for rest mode
        properties:
          mode: rest # transport # or rest
#          # security.auth: test:123456 #  only used for rest mode
          cluster.name: es
Docker 容器 canal mysql java
上一篇: 《解讀window核心程式設計》 之 核心對象
下一篇: 通過執行個體了解JavaScript中的arguments,call,apply,callee及caller

繼續閱讀