åè¨
æä»¬å ¬å¸æ个项ç®çæ°æ®éé«è¾¾äºåä¸ï¼ä½æ¯å 为æ¥è¡¨é£åæ°æ®ä¸å¤ªåç¡®ï¼ä¸å¡åºåæ¥è¡¨åºåæ¯è·¨åºæä½ï¼æ以并ä¸è½ä½¿ç¨ SQL æ¥è¿è¡åæ¥ãå½æ¶çæç®æ¯éè¿ mysqldump æè åå¨çæ¹å¼æ¥è¿è¡åæ¥ï¼ä½æ¯å°è¯ååç°è¿äºæ¹æ¡é½ä¸åå®é ï¼
mysqldumpæ¹æ¡
ä¸ä» å¤ä»½éè¦æ¶é´ï¼åæ¥ä¹éè¦æ¶é´ï¼èä¸å¨å¤ä»½çè¿ç¨ï¼å¯è½è¿ä¼ææ°æ®äº§åºï¼ä¹å°±æ¯è¯´åæ¥çäºæ²¡åæ¥ï¼åå¨æ¹å¼ï¼è¿ä¸ªæçå¤ªæ ¢äºï¼è¦æ¯æ°æ®éå°è¿å¥½ï¼æ们使ç¨è¿ä¸ªæ¹å¼çæ¶åï¼ä¸ä¸ªå°æ¶æåæ¥ä¸¤åæ¡æ°æ®â¦
å®é çæç¯å¢éæ±
1.å°æéäºçMySQL5.7æ°æ®åºA(RDSæ°æ®åº)å ¨éåæ¥å°æ讯äºMySQL5.7æ°æ®åºA
2.åå°å®æ¶å¢éå¤ä»½
æéäºç°æ产åï¼DTSï¼æ¶è´¹äº§åï¼
å¼æºå°å
https://github.com/alibaba/DataX
DataXæ¯ä»ä¹
DataXæ¯é¿éäºDataWorksæ°æ®éæçå¼æºçæ¬ã
DataX æ¯é¿éäº DataWorks æ°æ®éæ çå¼æºçæ¬ï¼ä¸»è¦å°±æ¯ç¨äºå®ç°æ°æ®é´ç离线åæ¥ã DataX è´åäºå®ç°å æ¬å ³ç³»åæ°æ®åºï¼MySQLãOracle çï¼ãHDFSãHiveãODPSãHBaseãFTP ç åç§å¼ææ°æ®æºï¼å³ä¸åçæ°æ®åºï¼ é´ç¨³å®é«æçæ°æ®åæ¥åè½ã
ä¸ºäº è§£å³å¼ææ°æ®æºåæ¥é®é¢ï¼DataX å°å¤æçç½ç¶åæ¥é¾è·¯åæäºæåæ°æ®é¾è·¯ ï¼DataX ä½ä¸ºä¸é´ä¼ è¾è½½ä½è´è´£è¿æ¥åç§æ°æ®æºï¼
å½éè¦æ¥å ¥ä¸ä¸ªæ°çæ°æ®æºæ¶ï¼åªéè¦å°æ¤æ°æ®æºå¯¹æ¥å° DataXï¼ä¾¿è½è·å·²æçæ°æ®æºä½ä¸ºæ ç¼æ°æ®åæ¥ã
设计ç念
为äºè§£å³å¼ææ°æ®æºåæ¥é®é¢ï¼DataXå°å¤æçç½ç¶çåæ¥é¾è·¯åæäºæåæ°æ®é¾è·¯ï¼DataXä½ä¸ºä¸é´ä¼ è¾è½½ä½è´è´£è¿æ¥åç§æ°æ®æºãå½éè¦æ¥å ¥ä¸ä¸ªæ°çæ°æ®æºçæ¶åï¼åªéè¦å°æ¤æ°æ®æºå¯¹æ¥å°DataXï¼ä¾¿è½è·å·²æçæ°æ®æºåå°æ ç¼æ°æ®åæ¥ã
å½å使ç¨ç°ç¶
DataXå¨é¿éå·´å·´éå¢å 被广æ³ä½¿ç¨ï¼æ¿æ äºææ大æ°æ®ç离线åæ¥ä¸å¡ï¼å¹¶å·²æç»ç¨³å®è¿è¡äº6å¹´ä¹ä¹ ãç®åæ¯å¤©å®æåæ¥8wå¤éä½ä¸ï¼æ¯æ¥ä¼ è¾æ°æ®éè¶ è¿300TBã
æ¤åå·²ç»å¼æºDataX1.0çæ¬ï¼æ¤æ¬¡ä»ç»ä¸ºé¿éå·´å·´å¼æºå ¨æ°çæ¬DataX3.0ï¼æäºæ´å¤æ´å¼ºå¤§çåè½åæ´å¥½ç使ç¨ä½éªã
dataxåºæ¬ç»ä»¶ä»ç»
DataXæ¬èº«ä½ä¸ºç¦»çº¿æ°æ®åæ¥æ¡æ¶ï¼éç¨Framework + pluginæ¶ææ建ãå°æ°æ®æºè¯»åååå ¥æ½è±¡æ为Reader/Writeræ件ï¼çº³å ¥å°æ´ä¸ªåæ¥æ¡æ¶ä¸ã
Readerï¼Reader为æ°æ®éé模åï¼è´è´£ééæ°æ®æºçæ°æ®ï¼å°æ°æ®åéç»Frameworkã
Writerï¼ Writer为æ°æ®åå ¥æ¨¡åï¼è´è´£ä¸æåFrameworkåæ°æ®ï¼å¹¶å°æ°æ®åå ¥å°ç®ç端ã
Frameworkï¼Frameworkç¨äºè¿æ¥readeråwriterï¼ä½ä¸ºä¸¤è çæ°æ®ä¼ è¾ééï¼å¹¶å¤çç¼å²ï¼æµæ§ï¼å¹¶åï¼æ°æ®è½¬æ¢çæ ¸å¿ææ¯é®é¢ã
datax主æµæ°æ®åºæ¯ææ åµ
DataXç®åå·²ç»æäºæ¯è¾å ¨é¢çæ件ä½ç³»ï¼ä¸»æµçRDBMSæ°æ®åºãNOSQLã大æ°æ®è®¡ç®ç³»ç»é½å·²ç»æ¥å ¥ï¼ç®åæ¯ææ°æ®å¦ä¸å¾,
详æ 请ç¹å»ï¼DataXæ°æ®æºåèæå
ç±»å | æ°æ®æº | Reader(读) | Writer(å) | ææ¡£ |
RDBMS å ³ç³»åæ°æ®åº | MySQL | â | â | 读 ãå |
Oracle | â | â | 读 ãå | |
OceanBase | â | â | 读 ãå | |
SQLServer | â | â | 读 ãå | |
PostgreSQL | â | â | 读 ãå | |
DRDS | â | â | 读 ãå | |
Kingbase | â | â | 读 ãå | |
éç¨RDBMS(æ¯æææå ³ç³»åæ°æ®åº) | â | â | 读 ãå | |
é¿éäºæ°ä»æ°æ®åå¨ | ODPS | â | â | 读 ãå |
ADB | â | å | ||
ADS | â | å | ||
OSS | â | â | 读 ãå | |
OCS | â | å | ||
Hologres | â | å | ||
AnalyticDB For PostgreSQL | â | å | ||
é¿éäºä¸é´ä»¶ | datahub | â | â | 读 ãå |
SLS | â | â | 读 ãå | |
é¿éäºå¾æ°æ®åº | GDB | â | â | 读 ãå |
NoSQLæ°æ®åå¨ | OTS | â | â | 读 ãå |
Hbase0.94 | â | â | 读 ãå | |
Hbase1.1 | â | â | 读 ãå | |
Phoenix4.x | â | â | 读 ãå | |
Phoenix5.x | â | â | 读 ãå | |
MongoDB | â | â | 读 ãå | |
Cassandra | â | â | 读 ãå | |
æ°ä»æ°æ®åå¨ | StarRocks | â | â | 读 ãå |
ApacheDoris | â | å | ||
ClickHouse | â | å | ||
Databend | â | å | ||
Hive | â | â | 读 ãå | |
kudu | â | å | ||
æ ç»æåæ°æ®åå¨ | TxtFile | â | â | 读 ãå |
FTP | â | â | 读 ãå | |
HDFS | â | â | 读 ãå | |
Elasticsearch | â | å | ||
æ¶é´åºåæ°æ®åº | OpenTSDB | â | 读 | |
TSDB | â | â | 读 ãå | |
TDengine | â | â | 读 ãå |
ç³»ç»è¦æ±
æµè¯ç³»ç»ï¼ Centos7
å ç½IPï¼ 192.168.1.3 ãnode03ï¼å®è£ Dataxã
å ç½IPï¼ 192.168.1.4 ãnode04ï¼å®è£ MySQL5.7ã
éæ±ï¼ éè¿Dataxå·¥å ·å°é¿éäºçRDS(mysql5.7)çæ°æ®åºåæ¥å°æ¬å°node04æå¡å¨ä¸ã
jdk1.8+
pythonè¿è¡ç¯å¢ï¼æ¨èpython2.6.xï¼
Datax å®è£
1.ä¸è½½å®è£ 并解å
node03æå¡å¨æ§è¡ä»¥ä¸å½ä»¤è¿è¡è§£å
# wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
å¤ç¨ä¸è½½ï¼ http://js.funet8.com/centos_software/datax.tar.gz
# mkdir /data/
# tar -zxf datax.tar.gz -C /data/
æ件信æ¯ï¼
æ件å称ï¼datax.tar.gz
æ件大å°ï¼853734462 åè
MD5 ï¼8E93697ADDBD26BEBC157613089A1173
SHA1 ï¼B0735462809F664D721D992DF5FD4813C0DB360C
CRC32 ï¼FA2708CC
2.éªè¯dataxæ¯å¦å®è£ æå
è¿å ¥dataxçå®è£ ç®å½çbinè·¯å¾ä¸ï¼ç¶åæ§è¡ä»¥ä¸å½ä»¤éªè¯dataxæ¯å¦å®è£ æå
node03æ§è¡ä»¥ä¸å½ä»¤è¿å ¥dataxçbinç®å½
# cd /data/datax/bin
# python datax.py -r streamreader -w streamwriter
åºç°ä»¥ä¸ç»æï¼è¯æå®è£ æå
[root@node3 bin]# python datax.py -r streamreader -w streamwriter
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the streamreader document:
https://github.com/alibaba/DataX/blob/master/streamreader/doc/streamreader.md
Please refer to the streamwriter document:
https://github.com/alibaba/DataX/blob/master/streamwriter/doc/streamwriter.md
Please save the following configuration as a json file and use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.
{
"job": {
"content": [
{
"reader": {
"name": "streamreader",
"parameter": {
"column": [],
"sliceRecordCount": ""
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"encoding": "",
"print": true
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}
éè¦å é¤æ件
# rm -rf /data/datax/plugin/*/._* # éè¦å é¤éèæ件 (éè¦)
å½æªå é¤æ¶ï¼å¯è½ä¼è¾åºï¼[datax/plugin/reader/._drdsreader/plugin.json] ä¸åå¨. 请æ£æ¥æ¨çé ç½®æ件.
Dataxå®æ
ä¸ã使ç¨dataxå®ç°stream2streamæ°æ®è¯»å
使ç¨dataxå®ç°è¯»åå符串ï¼ç¶åæå°å°æ§å¶å°å½ä¸æ¥
1ï¼ç¬¬ä¸æ¥ï¼æ¥ç帮å©ææ¡£
# cd /data/datax
# python bin/datax.py -w streamwriter -r streamreader
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the streamreader document:
https://github.com/alibaba/DataX/blob/master/streamreader/doc/streamreader.md
Please refer to the streamwriter document:
https://github.com/alibaba/DataX/blob/master/streamwriter/doc/streamwriter.md
Please save the following configuration as a json file and use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.
{
"job": {
"content": [
{
"reader": {
"name": "streamreader",
"parameter": {
"column": [],
"sliceRecordCount": ""
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"encoding": "",
"print": true
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}
2ï¼ç¬¬äºæ¥ï¼å¼ådataxé ç½®æ件
node03æå¡å¨å¼åstream2streamçé ç½®æ件
# cd /data/datax/job
# vim stream2stream.json
{
"job": {
"setting": {
"speed": {
"byte":10485760
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "streamreader",
"parameter": {
"column" : [
{
"value": "DataX",
"type": "string"
},
{
"value": 19890604,
"type": "long"
},
{
"value": "1989-06-04 00:00:00",
"type": "date"
},
{
"value": true,
"type": "bool"
},
{
"value": "test",
"type": "bytes"
}
],
"sliceRecordCount": 10
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"print": true,
"encoding": "UTF-8"
}
}
}
]
}
}
3ï¼ç¬¬ä¸æ¥ï¼å¯å¨dataxå®ç°æ°æ®çæå°
æ§è¡ä»¥ä¸å½ä»¤å¯å¨datax
# cd /data/datax
# python /data/datax/bin/datax.py /data/datax/job/stream2stream.json
2023-02-27 10:20:04.678 [main] WARN ConfigParser - æ件[streamreader,streamwriter]å 载失败ï¼1såéè¯... Exception:Code:[Common-00], Describe:[æ¨æä¾çé
ç½®æ件åå¨é误信æ¯ï¼è¯·æ£æ¥æ¨çä½ä¸é
ç½® .] - é
置信æ¯é误ï¼æ¨æä¾çé
ç½®æ件[/home/data/datax/plugin/reader/._drdsreader/plugin.json]ä¸åå¨. 请æ£æ¥æ¨çé
ç½®æ件.
2023-02-27 10:20:05.686 [main] ERROR Engine -
ç»DataXæºè½åæ,该任å¡æå¯è½çé误åå æ¯:
com.alibaba.datax.common.exception.DataXException: Code:[Common-00], Describe:[æ¨æä¾çé
ç½®æ件åå¨é误信æ¯ï¼è¯·æ£æ¥æ¨çä½ä¸é
ç½® .] - é
置信æ¯é误ï¼æ¨æä¾çé
ç½®æ件[/home/data/datax/plugin/reader/._drdsreader/plugin.json]ä¸åå¨. 请æ£æ¥æ¨çé
ç½®æ件.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.common.util.Configuration.from(Configuration.java:95)
at com.alibaba.datax.core.util.ConfigParser.parseOnePluginConfig(ConfigParser.java:153)
at com.alibaba.datax.core.util.ConfigParser.parsePluginConfig(ConfigParser.java:125)
at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:63)
at com.alibaba.datax.core.Engine.entry(Engine.java:137)
at com.alibaba.datax.core.Engine.main(Engine.java:204)
解å³æ¥é
é¨ç½²dataxå°æ¬å°åé¦æ¬¡æ§è¡ä»»å¡æ¥é
åå«è¿å ¥å° readerå writer ç®å½,å é¤æè¿ç±»åæ件
cd /data/datax/plugin/reader
rm -rf ./._*
cd /data/datax/plugin/writer
rm -rf ./._*
å次æ§è¡
# python /data/datax/bin/datax.py /data/datax/job/stream2stream.json
æ¡ä¾äºï¼ä½¿ç¨dataxå®ç°mysql2stream
使ç¨dataxå®ç°å°mysqlä¸å¼ 表çæå®å段çæ°æ®æ½ååºæ¥ï¼å¹¶æå°åºæ¥
1ï¼ç¬¬ä¸æ¥ï¼å建mysqlæ°æ®åºä»¥ååmysqlå½ä¸æå ¥æ°æ®
æ§è¡ä»¥ä¸å½ä»¤å建mysql表æ°æ®
å¨ 192.168.1.6 æ°æ®åºä¸æä½
# mysql -u root -h 192.168.1.6 -P 3306 -p'123456'
mysql> CREATE DATABASE `userdb`;
mysql> USE `userdb`;
mysql> DROP TABLE IF EXISTS `emp`;
mysql> CREATE TABLE `emp` (
`id` int(11) DEFAULT NULL,
`name` varchar(100) DEFAULT NULL,
`deg` varchar(100) DEFAULT NULL,
`salary` int(11) DEFAULT NULL,
`dept` varchar(10) DEFAULT NULL,
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`is_delete` bigint(20) DEFAULT '1'
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
mysql> insert into `emp`(`id`,`name`,`deg`,`salary`,`dept`,`create_time`,`update_time`,`is_delete`) values (1201,'gopal','manager',50000,'TP','2018-06-17 18:54:32','2019-01-17 11:19:32',1),(1202,'manishahello','Proof reader',50000,'TPP','2018-06-15 18:54:32','2018-06-17 18:54:32',0),(1203,'khalillskjds','php dev',30000,'AC','2018-06-17 18:54:32','2019-03-14 09:18:27',1),(1204,'prasanth_xxx','php dev',30000,'AC','2018-06-17 18:54:32','2019-04-07 09:09:24',1),(1205,'kranthixxx','admin',20000,'TP','2018-06-17 18:54:32','2018-12-08 11:50:33',0),(1206,'garry','manager',50000,'TPC','2018-12-10 21:41:09','2018-12-10 21:41:09',1),(1207,'oliver','php dev',2000,'AC','2018-12-15 13:49:13','2018-12-15 13:49:13',1),(1208,'hello','phpDev',200,'TP','2018-12-16 09:41:48','2018-12-16 09:41:48',1),(1209,'ABC','HELLO',300,NULL,'2018-12-16 09:42:04','2018-12-16 09:42:24',1),(1210,'HELLO','HELLO',5800,'TP','2019-01-24 09:02:43','2019-01-24 09:02:43',1),(1211,'WORLD','TEST',8800,'AC','2019-01-24 09:03:15','2019-01-24 09:03:15',1),(1212,'sdfs','sdfsdf',8500,'AC','2019-03-13 22:01:38','2019-03-13 22:01:38',1),(1213,NULL,'sdfsdf',9800,'sdfsdf','2019-03-14 09:08:31','2019-03-14 09:08:54',1),(1214,'xxx','sdfsdf',9500,NULL,'2019-03-14 09:13:32','2019-03-14 09:13:44',0),(1215,'sdfsf','sdfsdfsdf',9870,'TP','2019-04-07 09:10:39','2019-04-07 09:11:18',0),(1216,'hello','HELLO',5600,'AC','2019-04-07 09:37:05','2019-04-07 09:37:05',1),(1217,'HELLO2','hello2',7800,'TP','2019-04-07 09:37:40','2019-04-07 09:38:17',1);
æ¥çæ°æ®
2ï¼ç¬¬äºæ¥ï¼å¼ådataxçé ç½®æ件
node03æ§è¡ä»¥ä¸å½ä»¤æ¥ç帮å©ææ¡£
# cd /data/datax
# python bin/datax.py -r mysqlreader -w streamwriter
node03æ§è¡ä»¥ä¸å½ä»¤å¼ådataxé ç½®æ件ï¼æ ¹æ®å®é æ åµå¡«åç¨æ·ååå¯ç çã
# cd /data/datax/job
# vim mysql2stream.json
{
"job": {
"setting": {
"speed": {
"channel": 3
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"id",
"name"
],
"connection": [
{
"table": [
"emp"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.6:3306/userdb"
]
}
]
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"encoding":"GBK",
"print":true
}
}
}
]
}
}
3ï¼ç¬¬ä¸æ¥ï¼å¯å¨dataxå®ç°æ°æ®åæ¥
node03æ§è¡ä»¥ä¸å½ä»¤å®ç°dataxæ°æ®åæ¥
# cd /data/datax
# python /data/datax/bin/datax.py /data/datax/job/mysql2stream.json
æ¥éï¼
2023-02-27 10:49:31.382 [main] WARN ConfigParser - æ件[mysqlreader,streamwriter]å 载失败ï¼1såéè¯... Exception:Code:[Common-00], Describe:[æ¨æä¾çé
ç½®æ件åå¨é误信æ¯ï¼è¯·æ£æ¥æ¨çä½ä¸é
ç½® .] - é
置信æ¯é误ï¼æ¨æä¾çé
ç½®æ件[/home/data/datax/plugin/reader/._drdsreader/plugin.json]ä¸åå¨. 请æ£æ¥æ¨çé
ç½®æ件.
2023-02-27 10:49:32.389 [main] ERROR Engine -
ç»DataXæºè½åæ,该任å¡æå¯è½çé误åå æ¯:
com.alibaba.datax.common.exception.DataXException: Code:[Common-00], Describe:[æ¨æä¾çé
ç½®æ件åå¨é误信æ¯ï¼è¯·æ£æ¥æ¨çä½ä¸é
ç½® .] - é
置信æ¯é误ï¼æ¨æä¾çé
ç½®æ件[/home/data/datax/plugin/reader/._drdsreader/plugin.json]ä¸åå¨. 请æ£æ¥æ¨çé
ç½®æ件.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.common.util.Configuration.from(Configuration.java:95)
at com.alibaba.datax.core.util.ConfigParser.parseOnePluginConfig(ConfigParser.java:153)
at com.alibaba.datax.core.util.ConfigParser.parsePluginConfig(ConfigParser.java:125)
at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:63)
at com.alibaba.datax.core.Engine.entry(Engine.java:137)
at com.alibaba.datax.core.Engine.main(Engine.java:204)
é¨ç½²dataxå°æ¬å°åé¦æ¬¡æ§è¡ä»»å¡æ¥é
åå«è¿å ¥å° readerå writer ç®å½,å é¤æè¿ç±»åæ件
cd /data/datax/plugin/reader
rm -rf ./._*
cd /data/datax/plugin/writer
rm -rf ./._*
å次æ§è¡
# python /data/datax/bin/datax.py /data/datax/job/mysql2stream.json
æ¡ä¾ä¸ï¼ä½¿ç¨dataxå®ç°å¢éæ°æ®åæ¥
使ç¨dataxå®ç°å¢éæ°æ®åæ¥æå°å°æ§å¶å°
1ï¼ç¬¬ä¸æ¥ï¼å¼ådataxçé ç½®æ件
node03æ§è¡ä»¥ä¸å½ä»¤å¼ådataxé ç½®æ件
# cd /data/datax/job
# vim mysql2streamadd.json
{
"job": {
"setting": {
"speed": {
"channel": 3
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"id",
"name"
],
"where": "create_time > '${start_time}' and create_time < '${end_time}'",
"connection": [
{
"table": [
"emp"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.6:3306/userdb"
]
}
]
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"encoding":"GBK",
"print":true
}
}
}
]
}
}
2ï¼ç¬¬äºæ¥ï¼å¯å¨dataxå®ç°æ°æ®åæ¥
# cd /data/datax
# /data/datax/bin/datax.py /data/datax/job/mysql2streamadd.json -p "-Dstart_time='2018-06-15 00:00:00' -Dend_time='2023-06-15 23:59:59'"
SQLï¼
select id,name from emp where (create_time > '2018-06-15 00:00:00' and create_time < '2023-06-15 23:59:59'
æ¡ä¾åï¼ä½¿ç¨dataxå®ç°mysql2mysql
使ç¨dataxå®ç°å°æ°æ®ä»mysqlå½ä¸è¯»åï¼å¹¶ä¸éè¿sqlè¯å¥å®ç°æ°æ®çè¿æ»¤ï¼å¹¶ä¸å°æ°æ®åå ¥å°mysqlå¦å¤ä¸å¼ 表å½ä¸å»
1ï¼ç¬¬ä¸æ¥ï¼å建mysqlå¦å¤ä¸å¼ 表
# mysql -u root -h 192.168.1.6 -P 3306 -p
mysql> USE userdb;
mysql> CREATE TABLE `emp2` (
`id` INT(11) DEFAULT NULL,
`name` VARCHAR(100) DEFAULT NULL,
`deg` VARCHAR(100) DEFAULT NULL,
`salary` INT(11) DEFAULT NULL
) ENGINE=INNODB DEFAULT CHARSET=latin1;
2ï¼ç¬¬äºæ¥ï¼å¼ådataxçé ç½®æ件
æ¥ç帮å©ææ¡£
# cd /data/datax
# python /data/datax/bin/datax.py -r mysqlreader -w mysqlwriter
node03æ§è¡ä»¥ä¸å½ä»¤å¼ådataxé ç½®æ件
# cd /data/datax/job/
# vim mysql2mysql.json
{
"job": {
"setting": {
"speed": {
"channel":1
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"connection": [
{
"querySql": [
"select id,name,deg,salary from emp where id < 1208;"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.6:3306/userdb"
]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "root",
"password": "123456",
"column": [
"id",
"name",
"deg",
"salary"
],
"session": [
"set session sql_mode='ANSI'"
],
"preSql": [
"delete from emp2"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://192.168.1.6:3306/userdb?useUnicode=true&characterEncoding=utf-8",
"table": [
"emp2"
]
}
]
}
}
}
]
}
}
3ï¼ç¬¬ä¸æ¥ï¼å¯å¨dataxå®ç°æ°æ®åæ¥
node03æ§è¡ä»¥ä¸å½ä»¤å®ç°dataxæ°æ®åæ¥
cd /data/datax
python /data/datax/bin/datax.py /data/datax/job/mysql2mysql.json
å®æ
2023-02-27 15:57:43.684 [job-0] INFO JobContainer - PerfTrace not enable!
2023-02-27 15:57:43.685 [job-0] INFO StandAloneJobContainerCommunicator - Total 7 records, 177 bytes | Speed 17B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-02-27 15:57:43.687 [job-0] INFO JobContainer -
ä»»å¡å¯å¨æ¶å» : 2023-02-27 15:57:33
ä»»å¡ç»ææ¶å» : 2023-02-27 15:57:43
ä»»å¡æ»è®¡èæ¶ : 10s
ä»»å¡å¹³åæµé : 17B/s
è®°å½åå
¥é度 : 0rec/s
读åºè®°å½æ»æ° : 7
读å失败æ»æ° : 0
4ï¼æ¥çæ°æ®
帮å©ææ¡£
MysqlReader æ件ææ¡£ï¼ https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md
DataX MysqlWriter https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md
# python /data/datax/bin/datax.py -r mysqlreader -w mysqlwriter
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader", # 读å端
"parameter": {
"column": [], # éè¦åæ¥çå (* 表示ææçå)
"connection": [
{
"jdbcUrl": [], # è¿æ¥ä¿¡æ¯
"table": [] # è¿æ¥è¡¨
}
],
"password": "", # è¿æ¥ç¨æ·
"username": "", # è¿æ¥å¯ç
"where": "" # æè¿°çéæ¡ä»¶
}
},
"writer": {
"name": "mysqlwriter", # åå
¥ç«¯
"parameter": {
"column": [], # éè¦åæ¥çå
"connection": [
{
"jdbcUrl": "", # è¿æ¥ä¿¡æ¯
"table": [] # è¿æ¥è¡¨
}
],
"password": "", # è¿æ¥å¯ç
"preSql": [], # åæ¥å. è¦åçäº
"session": [],
"username": "", # è¿æ¥ç¨æ·
"writeMode": "" # æä½ç±»å
}
}
}
],
"setting": {
"speed": {
"channel": "" # æå®å¹¶åæ°
}
}
}
}