作者简介：木子相心，具有多年数据库运维经验，擅长MySQL、Redis 的日常运维、高可用和备份方案，目前在持续学习OceanBase运维，现在一互联网公司从事DBA运维的工作。
如果你已经接触一段时间 OceanBase 的安装测试等，有没有想把现有的数据库迁移到 OceanBase 环境玩玩？如果有这种想法，那么我们需要一款支持异构环境(MySQL,Oracle等)数据库迁移到 OceanBase 的高效稳定同步工具。生产环境有更好的 OMS 可以使用，鉴于大多数同学可能跟本人一样目前没有商业版的环境，这里使用阿里开源的 DataX 同步工具来完成将 MySQL 数据同步到 OceanBase 集群。
说明：由于目前的 DataX 二进制包里不含 oceanbasev10reader 和 oceanbasev10writer 插件，需要源码编译后方可使用。
1、源码安装
参考：https://github.com/alibaba/DataX/blob/master/userGuid.md
1.1. 若没有 git 环境，先安装一下。
[root@test datax_code2]# yum install git -y [root@test datax_code2]#
1.2 git clone git@github.com:alibaba/DataX.git
[root@test datax_code2]# git clone git@github.com:alibaba/DataX.git 正克隆到 'DataX'... The authenticity of host 'github.com (52.74.223.119)' can't be established. RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8. RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'github.com,52.74.223.119' (RSA) to the list of known hosts. Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
登录个人的 github 账号，点击头像，选择 Settings，选择 SSH and GPS keys，添加一个 ssh key，将执行 git 命令的机器上对应用户的公钥(/root/.ssh/id_rsa.pub)内容拷贝到 github 网页。
再次执行成功。
[root@test datax_code2]# git clone git@github.com:alibaba/DataX.git 正克隆到 'DataX'... Warning: Permanently added the RSA host key for IP address '20.205.243.166' to the list of known hosts. remote: Enumerating objects: 2977, done. remote: Counting objects: 100% (190/190), done. remote: Compressing objects: 100% (115/115), done. remote: Total 2977 (delta 41), reused 135 (delta 23), pack-reused 2787 接收对象中: 100% (2977/2977), 18.93 MiB | 5.23 MiB/s, done. 处理 delta 中: 100% (770/770), done. [root@test datax_code2]#
1.3 通过maven打包
wget https://dlcdn.apache.org/maven/maven-3/3.8.2/binaries/apache-maven-3.8.2-bin.tar.gz
如果版本不可用，可以登录 http://maven.apache.org/download.cgi 选择可用的版本。
[root@test opt]# tar zxf apache-maven-3.8.2-bin.tar.gz [root@test opt]# cd apache-maven-3.8.2 [root@test apache-maven-3.8.2]# ll 总用量 36 drwxr-xr-x. 2 root root 97 8月 18 17:58 bin drwxr-xr-x. 2 root root 76 8月 18 17:58 boot drwxr-xr-x. 3 root root 63 8月 5 02:57 conf drwxr-xr-x. 3 root root 4096 8月 18 17:58 lib -rw-r--r--. 1 root root 17511 8月 5 02:57 LICENSE -rw-r--r--. 1 root root 5141 8月 5 02:57 NOTICE -rw-r--r--. 1 root root 2612 8月 5 02:57 README.txt [root@test bin]# ln -s /opt/apache-maven-3.8.2/bin/mvn /usr/bin/mvn [root@test DataX]# mvn -U clean package assembly:assembly -Dmaven.test.skip=true which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) The JAVA_HOME environment variable is not defined correctly, this environment variable is needed to run this program. [root@test DataX]#
下载 JDK https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html
[root@test ~]# java -version java version "1.8.0_77" Java(TM) SE Runtime Environment (build 1.8.0_77-b03) Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode) [root@test ~]# [root@test DataX]# pwd /opt/datax_code2/DataX [root@test DataX]# mvn -U clean package assembly:assembly -Dmaven.test.skip=true
说明：如果想一次性打包成功，经过测试可以直接把不需要的插件删除掉
[root@test DataX]# pwd /opt/datax_code2/DataX/pom.xml
删除<module>hdfsreader</module>
删除<module>hdfswriter</module>
删除<module>oscarwriter</module>
第一次执行报错
#第一次执行报错 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:06 min [INFO] Finished at: 2021-08-18T18:14:03+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project hdfsreader: Could not resolve dependencies for project com.alibaba.datax:hdfsreader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-exec:jar:1.1.1 -> org.apache.calcite:calcite-core:jar:1.0.0-incubating -> org.apache.calcite:calcite-avatica:jar:1.0.0-incubating -> eigenbase:eigenbase-properties:jar:1.1.4: Failed to read artifact descriptor for eigenbase:eigenbase-properties:jar:1.1.4: Could not transfer artifact eigenbase:eigenbase-properties:pom:1.1.4 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [datanucleus (http://www.datanucleus.org/downloads/maven2, default, releases), glassfish-repository (http://maven.glassfish.org/content/groups/glassfish, default, disabled), glassfish-repo-archive (http://maven.glassfish.org/content/groups/glassfish, default, disabled), apache.snapshots (http://repository.apache.org/snapshots, default, snapshots), conjars (http://conjars.org/repo, default, releases+snapshots)] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :hdfsreader [root@test DataX]#
第二次执行报错
[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:24 min [INFO] Finished at: 2021-08-18T18:17:31+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project hdfswriter: Could not resolve dependencies for project com.alibaba.datax:hdfswriter:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-exec:jar:1.1.1 -> org.apache.calcite:calcite-core:jar:1.0.0-incubating -> org.apache.calcite:calcite-avatica:jar:1.0.0-incubating -> eigenbase:eigenbase-properties:jar:1.1.4: Failed to read artifact descriptor for eigenbase:eigenbase-properties:jar:1.1.4: Could not transfer artifact eigenbase:eigenbase-properties:pom:1.1.4 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [datanucleus (http://www.datanucleus.org/downloads/maven2, default, releases), glassfish-repository (http://maven.glassfish.org/content/groups/glassfish, default, disabled), glassfish-repo-archive (http://maven.glassfish.org/content/groups/glassfish, default, disabled), apache.snapshots (http://repository.apache.org/snapshots, default, snapshots), conjars (http://conjars.org/repo, default, releases+snapshots)] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :hdfswriter [root@test DataX]#
第三次执行报错
[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:46 min [INFO] Finished at: 2021-08-18T18:21:23+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project oscarwriter: Could not resolve dependencies for project com.alibaba.datax:oscarwriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.oscar:oscar:jar:7.0.8 at specified path /opt/datax_code2/DataX/oscarwriter/src/main/lib/oscarJDBC.jar -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :oscarwriter [root@test DataX]#
第四次执行成功
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] datax-all 0.0.1-SNAPSHOT ........................... SUCCESS [01:57 min] [INFO] datax-common 0.0.1-SNAPSHOT ........................ SUCCESS [ 1.981 s] [INFO] datax-transformer 0.0.1-SNAPSHOT ................... SUCCESS [ 1.095 s] [INFO] datax-core 0.0.1-SNAPSHOT .......................... SUCCESS [ 2.626 s] [INFO] plugin-rdbms-util 0.0.1-SNAPSHOT ................... SUCCESS [ 1.055 s] [INFO] mysqlreader 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.872 s] [INFO] drdsreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 0.996 s] [INFO] sqlserverreader 0.0.1-SNAPSHOT ..................... SUCCESS [ 1.237 s] [INFO] postgresqlreader 0.0.1-SNAPSHOT .................... SUCCESS [ 0.876 s] [INFO] kingbaseesreader 0.0.1-SNAPSHOT .................... SUCCESS [ 0.878 s] [INFO] oraclereader 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.867 s] [INFO] odpsreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.542 s] [INFO] otsreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 1.992 s] [INFO] otsstreamreader 0.0.1 .............................. SUCCESS [ 2.048 s] [INFO] plugin-unstructured-storage-util 0.0.1-SNAPSHOT .... SUCCESS [ 0.904 s] [INFO] txtfilereader 0.0.1-SNAPSHOT ....................... SUCCESS [ 3.129 s] [INFO] streamreader 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.872 s] [INFO] ossreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 3.184 s] [INFO] ftpreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.921 s] [INFO] mongodbreader 0.0.1-SNAPSHOT ....................... SUCCESS [ 2.812 s] [INFO] rdbmsreader 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.958 s] [INFO] hbase11xreader 0.0.1-SNAPSHOT ...................... SUCCESS [ 4.730 s] [INFO] hbase094xreader 0.0.1-SNAPSHOT ..................... SUCCESS [ 2.840 s] [INFO] tsdbreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.235 s] [INFO] opentsdbreader 0.0.1-SNAPSHOT ...................... SUCCESS [ 1.967 s] [INFO] cassandrareader 0.0.1-SNAPSHOT ..................... SUCCESS [ 1.607 s] [INFO] gdbreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 3.300 s] [INFO] oceanbasev10reader 0.0.1-SNAPSHOT .................. SUCCESS [ 1.205 s] [INFO] mysqlwriter 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.788 s] [INFO] drdswriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 0.831 s] [INFO] odpswriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.582 s] [INFO] txtfilewriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 2.693 s] [INFO] ftpwriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.816 s] [INFO] streamwriter 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.829 s] [INFO] otswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 1.704 s] [INFO] oraclewriter 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.820 s] [INFO] sqlserverwriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 0.814 s] [INFO] postgresqlwriter 0.0.1-SNAPSHOT .................... SUCCESS [ 0.838 s] [INFO] kingbaseeswriter 0.0.1-SNAPSHOT .................... SUCCESS [ 0.873 s] [INFO] osswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.925 s] [INFO] mongodbwriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 2.936 s] [INFO] adswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.580 s] [INFO] ocswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 1.692 s] [INFO] rdbmswriter 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.995 s] [INFO] hbase11xwriter 0.0.1-SNAPSHOT ...................... SUCCESS [ 4.564 s] [INFO] hbase094xwriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 2.812 s] [INFO] hbase11xsqlwriter 0.0.1-SNAPSHOT ................... SUCCESS [ 7.919 s] [INFO] hbase11xsqlreader 0.0.1-SNAPSHOT ................... SUCCESS [ 8.665 s] [INFO] elasticsearchwriter 0.0.1-SNAPSHOT ................. SUCCESS [ 1.460 s] [INFO] tsdbwriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.038 s] [INFO] adbpgwriter 0.0.1-SNAPSHOT ......................... SUCCESS [ 1.734 s] [INFO] gdbwriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 3.027 s] [INFO] cassandrawriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 1.564 s] [INFO] clickhousewriter 0.0.1-SNAPSHOT .................... SUCCESS [ 1.753 s] [INFO] oceanbasev10writer 0.0.1-SNAPSHOT .................. SUCCESS [ 4.283 s] [INFO] hbase20xsqlreader 0.0.1-SNAPSHOT ................... SUCCESS [ 2.773 s] [INFO] hbase20xsqlwriter 0.0.1-SNAPSHOT ................... SUCCESS [ 0.984 s] [INFO] kuduwriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 2.710 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:59 min [INFO] Finished at: 2021-08-18T18:26:06+08:00 [INFO] ------------------------------------------------------------------------ [root@test DataX]#
注意：目前源码编译完成后以下2个模板文件不存在，需要手动创建
/opt/datax_code2/DataX/target/datax/datax/plugin/reader/oceanbasev10reader/plugin_job_template.json /opt/datax_code2/DataX/target/datax/datax/plugin/writer/oceanbasev10writer/plugin_job_template.json
2、添加OB对应的插件模板文件
2.1 oceanbasev10reader 模板
虽然本次测试未使用到，便于后续的测试也一起添加。
[root@test oceanbasev10reader]# pwd /opt/datax_code2/DataX/target/datax/datax/plugin/reader/oceanbasev10reader [root@test oceanbasev10reader]# cat plugin_job_template.json { "name": "oceanbasev10reader", "parameter": { "username": "", "password": "", "readBatchSize": 100000, "weakRead": false, "column": [ "*" ], "connection": [ { "jdbcUrl": [ "||_dsc_ob10_dsc_||clusterName:tenantName||_dsc_ob10_dsc_||jdbc:oceanbase://obproxyIp:port/dbname" ], "table": [ "tabName" ] } ] } }
2.2 oceanbasev10writer 模板
说明：obWriteMode 群里老师提供的 ob2ob.json 里对应的是 update，不是太理解，这里测试先调整成 insert。
[root@test oceanbasev10writer]# pwd /opt/datax_code/DataX/target/datax/datax/plugin/writer/oceanbasev10writer [root@test oceanbasev10writer]# cat plugin_job_template.json { "name": "oceanbasev10writer", "parameter": { "obWriteMode": "insert", "username": "", "password": "", "preSql": [ "" ], "column": [ "*" ], "connection": [ { "jdbcUrl": "||_dsc_ob10_dsc_||clusterName:tenantName||_dsc_ob10_dsc_||jdbc:oceanbase://obproxyIp:port/dbname?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true", "table": [ "tabName" ] } ] } }
3、同步数据
3.1 编辑 JOB 配置文件
[root@test bin]# cat mysql2ob.json
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "root654321",
"column": ["name"],
"splitPk": "id",
"connection": [
{
"table": [
"t1","t2"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db1"
]
},
{
"table": [
"t3","t4"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db2"
]
},
{
"table": [
"t5","t6"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db3"
]
},
{
"table": [
"t7","t8"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db4"
]
}
]
}
},
"writer": {
"name": "oceanbasev10writer",
"parameter": {
"column": [
"name"
],
"connection": [
{
"jdbcUrl": "||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true",
"table": [
"t_sum"
]
}
],
"obWriteMode": "insert",
"password": "root",
"preSql": [
"truncate table t_sum;"
],
"username": "root"
}
}
}
],
"setting": {
"speed": {
"channel": "1"
}
}
}
}
[root@test bin]#
3.2 启动 Datax 同步数据
[root@test bin]# python datax.py mysql2ob.json DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2021-08-18 18:31:47.498 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2021-08-18 18:31:47.506 [main] INFO Engine - the machine info => osInfo: Oracle Corporation 1.8 25.77-b03jvmInfo: Linux amd64 3.10.0-862.el7.x86_64cpu num: 32 totalPhysicalMemory: -0.00GfreePhysicalMemory: -0.00GmaxFileDescriptorCount: -1currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size PS Eden Space | 256.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 683.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB 2021-08-18 18:31:47.527 [main] INFO Engine - {"content":[{"reader":{"name":"mysqlreader","parameter":{"column":["name"],"connection":[{"jdbcUrl":["jdbc:mysql://192.168.1.100:6008/db1"],"table":["t1","t2"]},{"jdbcUrl":["jdbc:mysql://192.168.1.100:6008/db2"],"table":["t3","t4"]},{"jdbcUrl":["jdbc:mysql://192.168.1.100:6008/db3"],"table":["t5","t6"]},{"jdbcUrl":["jdbc:mysql://192.168.1.100:6008/db4"],"table":["t7","t8"]}],"password":"**********","splitPk":"id","username":"root"}},"writer":{"name":"oceanbasev10writer","parameter":{"column":["name"],"connection":[{"jdbcUrl":"||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true","table":["t_sum"]}],"obWriteMode":"insert","password":"****","preSql":["truncate table t_sum;"],"username":"root"}}}],"setting":{"speed":{"channel":"1"}} } 2021-08-18 18:31:47.546 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2021-08-18 18:31:47.548 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2021-08-18 18:31:47.549 [main] INFO JobContainer - DataX jobContainer starts job. 2021-08-18 18:31:47.550 [main] INFO JobContainer - Set jobId = 0 Wed Aug 18 18:31:47 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:48.205 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:48.233 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:48.255 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:48.281 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:48.306 [job-0] INFO OriginalConfPretreatmentUtil - table:[t1] has columns:[id,name]. 2021-08-18 18:31:48.320 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.320 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.579 [job-0] INFO DbUtils - value for query [SHOW VARIABLES LIKE 'ob_compatibility_mode'] is [MYSQL] 2021-08-18 18:31:48.587 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.587 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.601 [job-0] INFO OriginalConfPretreatmentUtil - table:[t_sum] all columns:[ id,name ]. 2021-08-18 18:31:48.602 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.602 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.615 [job-0] INFO OriginalConfPretreatmentUtil - Write data [ INSERT INTO %s (name) VALUES(?) ], which jdbcUrl like:[||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true] 2021-08-18 18:31:48.616 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2021-08-18 18:31:48.616 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work . 2021-08-18 18:31:48.616 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do prepare work . 2021-08-18 18:31:48.617 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.617 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.626 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table t_sum;]. context info:||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. 2021-08-18 18:31:48.730 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.730 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.743 [job-0] INFO DbUtils - value for query [show variables like 'version'] is [3.1.0] 2021-08-18 18:31:48.743 [job-0] INFO JobContainer - jobContainer starts to do split ... 2021-08-18 18:31:48.743 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels. 2021-08-18 18:31:48.752 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [8] tasks. 2021-08-18 18:31:48.753 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] splits to [8] tasks. 2021-08-18 18:31:48.779 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2021-08-18 18:31:48.789 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2021-08-18 18:31:48.792 [job-0] INFO JobContainer - Running by standalone Mode. 2021-08-18 18:31:48.801 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [8] tasks. 2021-08-18 18:31:48.807 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated. 2021-08-18 18:31:48.807 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2021-08-18 18:31:48.867 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2021-08-18 18:31:48.869 [0-0-0-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:48.871 [0-0-0-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2021-08-18 18:31:48.871 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t1 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:48.880 [0-0-0-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:48.897 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t1 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:48.900 [0-0-0-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:48.901 [0-0-0-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:48.902 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.902 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.912 [0-0-0-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:48.912 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:48.915 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:48.915 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:48.931 [0-0-0-writer] INFO ColumnMetaCache - fetch columnMeta of table t_sum success 2021-08-18 18:31:48.949 [0-0-0-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:48.950 [0-0-0-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.068 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[203]ms 2021-08-18 18:31:49.070 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started 2021-08-18 18:31:49.070 [0-0-1-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.070 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t2 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.070 [0-0-1-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.079 [0-0-1-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.088 [0-0-1-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.089 [0-0-1-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.089 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.089 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.091 [0-0-1-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t2 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.099 [0-0-1-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.099 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.099 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.100 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.124 [0-0-1-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.124 [0-0-1-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.170 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[100]ms 2021-08-18 18:31:49.172 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started 2021-08-18 18:31:49.172 [0-0-2-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.172 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t3 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.173 [0-0-2-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.181 [0-0-2-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.191 [0-0-2-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.192 [0-0-2-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.192 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.192 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.197 [0-0-2-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t3 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.199 [0-0-2-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.200 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.200 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.200 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.223 [0-0-2-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.224 [0-0-2-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.272 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[100]ms 2021-08-18 18:31:49.274 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started 2021-08-18 18:31:49.274 [0-0-3-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.275 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t4 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.275 [0-0-3-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.283 [0-0-3-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.290 [0-0-3-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.291 [0-0-3-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.291 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.291 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.295 [0-0-3-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t4 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.299 [0-0-3-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.300 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.300 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.300 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.319 [0-0-3-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.319 [0-0-3-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.375 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[101]ms 2021-08-18 18:31:49.377 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started 2021-08-18 18:31:49.377 [0-0-4-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.377 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t5 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.378 [0-0-4-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.388 [0-0-4-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.397 [0-0-4-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.397 [0-0-4-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.397 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.397 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.398 [0-0-4-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t5 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.406 [0-0-4-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.407 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.407 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.407 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.428 [0-0-4-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.428 [0-0-4-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.478 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[101]ms 2021-08-18 18:31:49.480 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started 2021-08-18 18:31:49.480 [0-0-5-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.480 [0-0-5-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t6 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.480 [0-0-5-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.489 [0-0-5-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.498 [0-0-5-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.498 [0-0-5-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.498 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.498 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.503 [0-0-5-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t6 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.507 [0-0-5-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.508 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.508 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.508 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.532 [0-0-5-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.532 [0-0-5-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.580 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[100]ms 2021-08-18 18:31:49.582 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started 2021-08-18 18:31:49.582 [0-0-6-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.583 [0-0-6-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t7 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.583 [0-0-6-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.592 [0-0-6-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.600 [0-0-6-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.600 [0-0-6-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.600 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.600 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.602 [0-0-6-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t7 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.608 [0-0-6-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.608 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.608 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.608 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.630 [0-0-6-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.630 [0-0-6-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.683 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[101]ms 2021-08-18 18:31:49.685 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started 2021-08-18 18:31:49.685 [0-0-7-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2021-08-18 18:31:49.685 [0-0-7-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t8 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.686 [0-0-7-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2021-08-18 18:31:49.693 [0-0-7-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2021-08-18 18:31:49.703 [0-0-7-writer] INFO CommonRdbmsWriter$Task - write mode: insert 2021-08-18 18:31:49.703 [0-0-7-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?) 2021-08-18 18:31:49.703 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.703 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.705 [0-0-7-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t8 ] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2021-08-18 18:31:49.710 [0-0-7-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2021-08-18 18:31:49.710 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2021-08-18 18:31:49.711 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. 2021-08-18 18:31:49.711 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2021-08-18 18:31:49.732 [0-0-7-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2021-08-18 18:31:49.732 [0-0-7-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0 2021-08-18 18:31:49.785 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[100]ms 2021-08-18 18:31:49.786 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks. 2021-08-18 18:31:58.868 [job-0] INFO StandAloneJobContainerCommunicator - Total 16 records, 16 bytes | Speed 1B/s, 1 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2021-08-18 18:31:58.868 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks. 2021-08-18 18:31:58.868 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do post work. 2021-08-18 18:31:58.868 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work. 2021-08-18 18:31:58.868 [job-0] INFO JobContainer - DataX jobId [0] completed successfully. 2021-08-18 18:31:58.869 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /opt/datax_code2/DataX/target/datax/datax/hook 2021-08-18 18:31:58.870 [job-0] INFO JobContainer -[total cpu info] =>averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00% [total gc info] =>NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime PS MarkSweep | 1 | 1 | 1 | 0.027s | 0.027s | 0.027s PS Scavenge | 1 | 1 | 1 | 0.020s | 0.020s | 0.020s 2021-08-18 18:31:58.870 [job-0] INFO JobContainer - PerfTrace not enable! 2021-08-18 18:31:58.871 [job-0] INFO StandAloneJobContainerCommunicator - Total 16 records, 16 bytes | Speed 1B/s, 1 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2021-08-18 18:31:58.872 [job-0] INFO JobContainer - 任务启动时刻 : 2021-08-18 18:31:47 任务结束时刻 : 2021-08-18 18:31:58 任务总计耗时 : 11s 任务平均流量 : 1B/s 记录写入速度 : 1rec/s 读出记录总数 : 16 读写失败总数 : 0 [root@test bin]#
4、目标端 OB 租户下验证数据
MySQL [db1]> show tenant; +---------------------+ | Current_tenant_name | +---------------------+ | test_tenant_1 | +---------------------+ 1 row in set (0.003 sec) MySQL [db1]> select * from t_sum; +----+------+ | id | name | +----+------+ | 1 | a | | 2 | b | | 3 | c | | 4 | d | | 5 | e | | 6 | f | | 7 | g | | 8 | h | | 9 | i | | 10 | j | | 11 | k | | 12 | l | | 13 | m | | 14 | n | | 15 | o | | 16 | p | +----+------+ 16 rows in set (0.003 sec)
5、测试描述
本测试源端有 4 个分片实例（这里使用一个实例 4 个库模拟），每个分片 2 个库，每个库 2 张表。
同步到目表端 oceanbase MySQL 租户(test_tenant_1)下的 t_sum 表中，源表主键 id ，不具有业务实际意义。
6、测试数据
6.1 目标端 OB 环境 test_tenant_1 租户 db1 库下创建汇总表
create table t_sum(id int not null auto_increment primary key,name varchar(20)); MySQL [db1]> show tenant; +---------------------+ | Current_tenant_name | +---------------------+ | test_tenant_1 | +---------------------+ 1 row in set (0.003 sec) MySQL [db1]> show create table t_sum \G *************************** 1. row *************************** Table: t_sum Create Table: CREATE TABLE `t_sum` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(20) DEFAULT NULL, PRIMARY KEY (`id`) ) AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8mb4 ROW_FORMAT = COMPACT COMPRESSION = 'zstd_1.3.8' \n<br>REPLICA_NUM = 3 BLOCK_SIZE = 16384 USE_BLOOM_FILTER = FALSE TABLET_SIZE = 134217728 PCTFREE = 0 1 row in set (0.004 sec)
6.2 源端 MySQL 分片实例库表
#MySQL模拟分片1 create table db1.t1(id int not null auto_increment primary key,name varchar(20)); create table db1.t2(id int not null auto_increment primary key,name varchar(20)); insert into db1.t1 values(1,'a'); insert into db1.t1 values(2,'b'); insert into db1.t2 values(1,'c'); insert into db1.t2 values(2,'d'); #MySQL模拟分片2 create table db2.t3(id int not null auto_increment primary key,name varchar(20)); create table db2.t4(id int not null auto_increment primary key,name varchar(20)); insert into db2.t3 values(1,'e'); insert into db2.t3 values(2,'f'); insert into db2.t4 values(1,'g'); insert into db2.t4 values(2,'h'); #MySQL模拟分片3 create table db3.t5(id int not null auto_increment primary key,name varchar(20)); create table db3.t6(id int not null auto_increment primary key,name varchar(20)); insert into db3.t5 values(1,'i'); insert into db3.t5 values(2,'j'); insert into db3.t6 values(1,'k'); insert into db3.t6 values(2,'l'); #MySQL模拟分片4 create table db4.t7(id int not null auto_increment primary key,name varchar(20)); create table db4.t8(id int not null auto_increment primary key,name varchar(20)); insert into db4.t7 values(1,'m'); insert into db4.t7 values(2,'n'); insert into db4.t8 values(1,'o'); insert into db4.t8 values(2,'p');
本次测试只是做了一个简单的通过 Datax 同步工具将 MySQL 数据同步到 OceanBase 环境，里面的参数还需要进一步学习，这里也期待后续开源的数据库同步工具能有以下功能：
1、同步配置支持库、表支持通配符写法。
2、同步工具自身支持高可用部署。
3、同步期间支持上游环境 DDL 操作。
4、同步期间支持暂停功能。
如果您有任何疑问，可以通过以下方式与我们进行交流：
微信群：扫码添加小助手，将拉你进群哟～
钉钉群：33254054
