DataX3.0安装部署及同步测试-每日运维

DataX3.0安装部署及同步测试
文档概述DataX是一个异构数据源离线同步工具，致力于实现包括关系型数据库(MySQL、Oracle、SqlServer、PostgreSQL等)、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间稳定高效的数据同步功能。DataX 是阿里云 DataWorks数据集成的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。

DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统,每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。

第一部分：使用docker安装mysql5.7
1.安装docker
2.安装mysql5.7

1.1安装最新版本docker，如果存在旧版本的Docker，可以卸载

sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine

1.2安装 Docker依赖包：

sudo yum install -y yum-utils device-mapper-persistent-data lvm2

1.3添加 Docker YUM 源

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

1.4.安装docker

sudo yum install docker-ce docker-ce-cli containerd.io

Dependency Installed:
docker-buildx-plugin.x86_64 0:0.12.1-1.el7 docker-ce-rootless-extras.x86_64 0:25.0.2-1.el7 docker-compose-plugin.x86_64 0:2.24.5-1.el7

Complete!

安装如果报错：
Error: Package: docker-ce-rootless-extras-25.0.2-1.el7.x86_64 (docker-ce-stable)
Requires: slirp4netns >= 0.4
Error: Package: 3:docker-ce-25.0.2-1.el7.x86_64 (docker-ce-stable)
Requires: container-selinux >= 2:2.74
Error: Package: containerd.io-1.6.28-3.1.el7.x86_64 (docker-ce-stable)
Requires: container-selinux >= 2:2.74
Error: Package: docker-ce-rootless-extras-25.0.2-1.el7.x86_64 (docker-ce-stable)
Requires: fuse-overlayfs >= 0.7

解决：
cd /etc/yum.repos.d
vim centos-extras.repo
[centos-extras]
name=Centos extras - $basearch
baseurl=http://mirror.centos.org/centos/7/extras/x86_64
enabled=1
gpgcheck=0

yum -y install slirp4netns fuse-overlayfs container-selinux

1.5.配置镜像加速器

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json 3306/tcp, :::3306->3306/tcp, 33060/tcp mysql

[root@node2 ]# docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
mysql 5.7 c20987f18b13 2 years ago 448MB

2.4.进入容器
docker exec -it mysql bash

mysql -u root -p123456

切换到mysql数据库，并查看user表。
use mysql;

select user,host from user;

[root@node2 ]# docker exec -it mysql bash
root@84c6627f5440:/# mysql -u root -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.36 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

到此，使用docker安装mysql5.7完成

第二部分：Docker环境下Oracle12c安装步骤
镜像仓库地址：https://github.com/MaksymBilenko/docker-oracle-12c

1.拉取镜像文件
拉取Oracle12c的Docker文件

docker pull quay.io/maksymbilenko/oracle-12c

[root@node1 container]# docker pull quay.io/maksymbilenko/oracle-12c
Using default tag: latest
latest: Pulling from maksymbilenko/oracle-12c
8ba884070f61: Pull complete
ef9513b81046: Pull complete
6f1de349e202: Pull complete
5376ebfa0fa3: Pull complete
5f632c3633d2: Pull complete
3e74293031d2: Pull complete
Digest: sha256:de3f7d4a608107d6271d7b2c40368caae7b4fc14bb893e164346fb951e15a70b
Status: Downloaded newer image for quay.io/maksymbilenko/oracle-12c:latest
quay.io/maksymbilenko/oracle-12c:latest

2.建立Oracle数据库本地目录
在系统根目录下，新建/oracle/data目录

mkdir -p /oracle/data

chown -R 1000:1000 /oracle/data

4.运行容器
docker运行12C容器

docker run --name o12c -d -p 8080:8080 -p 1521:1521 -v /oracle/data:/u01/app/oracle quay.io/maksymbilenko/oracle-12c

[root@node1 container]# mkdir -p /oracle/data
[root@node1 container]# chown -R 1000:1000 /oracle/data
[root@node1 container]# docker run --name o12c -d -p 8080:8080 -p 1521:1521 -v /oracle/data:/u01/app/oracle quay.io/maksymbilenko/oracle-12c
4044f830d3637b1f119a1a01598c5e9d45ada1637227fc4a062855bcf1c590f8

[root@node1 container]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4044f830d363 quay.io/maksymbilenko/oracle-12c "/entrypoint.sh " 16 seconds ago Up 14 seconds 0.0.0.0:1521->1521/tcp, :::1521->1521/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp o12c

[root@node1 container]# docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/maksymbilenko/oracle-12c latest a063ba7a3cab 2 years ago 8GB
[root@node1 container]#

4.1数据库连接信息

hostname: localhost
port: 1521
sid: xe
service name: xe
username: system
password: oracle

SYS & SYSTEM 的密码

oracle

4.2进入容器

docker exec -it 4044f830d363 bash

4.3修改环境变量，设置ORACLE_SID，ORACLE_HOME

[oracle@4044f830d363 ~]$ vim .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH
export ORACLE_HOME=/u01/app/oracle/product/12.2.0/SE

export ORACLE_SID=xe

export PATH=$ORACLE_HOME/bin:$PATH

[oracle@4044f830d363 ~]$ source .bash_profile

4.4检查实例状态
[oracle@4044f830d363 ~]$ sqlplus /nolog
SQL> conn / as sysdba;
Connected.
SQL> select instance_name,status from v$instance;

INSTANCE_NAME STATUS
---------------- ------------
xe OPEN

4.5检查监听状态，lsnrctl status

[oracle@4044f830d363 ~]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 01-FEB-2024 08:27:44

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date 01-FEB-2024 08:01:25
Uptime 0 days 0 hr. 26 min. 23 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Log File /u01/app/oracle/diag/tnslsnr/4044f830d363/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=4044f830d363)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=4044f830d363)(PORT=8080))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "xe" has 1 instance(s).
Instance "xe", status READY, has 1 handler(s) for this service...
Service "xeXDB" has 1 instance(s).
Instance "xe", status READY, has 1 handler(s) for this service...
The command completed successfully

5.新建表空间和用户

#创建test表空间

create tablespace test datafile '/u01/app/oracle/oradata/xe/test.dbf' size 1M autoextend on next 50M maxsize unlimited;

#创建test用户

create user test identified by "admin@123" default tablespace test profile DEFAULT ACCOUNT UNLOCK;

#表空间和用户赋值权限

CREATE USER test identified by test;
alter user test identified by test;
GRANT DBA to test ;
grant SELECT ANY DICTIONARY to test;
GRANT EXECUTE ON SYS.DBMS_LOCK TO test;

[oracle@4044f830d363 ~]$ sqlplus /nolog

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 1 08:10:59 2024

SQL> conn /as sysdba
Connected.

SQL> CREATE USER test identified by test;

User created.

SQL> alter user test identified by test;
GRANT DBA to test ;
grant SELECT ANY DICTIONARY to test;
GRANT EXECUTE ON SYS.DBMS_LOCK TO test;
User altered.

SQL>
Grant succeeded.

SQL>

Grant succeeded.

SQL> create tablespace test datafile '/u01/app/oracle/oradata/xe/test.dbf' size 1M autoextend on next 50M maxsize unlimited;

Tablespace created.

SQL> create user test identified by "admin@123" default tablespace test profile DEFAULT ACCOUNT UNLOCK;

User created.

SQL> grant dba to test;

Grant succeeded.

SQL> grant unlimited tablespace to test;

Grant succeeded.

至此，Oracle12c安装完成.

第三部分：使用docker部署datax和datax-web
一.准备基础环境

二.安装mysql5.7数据库见【使用docker安装mysql5.7】

三.datax环境配置与执行datax_web容器

一.准备基础环境

1.0 服务器配置

172.17.0.29 mysql CENTOS7.6 8C16G

172.17.0.34 datax CENTOS7.6 8C16G

172.17.0.32 oracle CENTOS7.6 8C16G

1.1 开通所需端口3306,9527

firewall-cmd --zone=public --add-port=3306/tcp --permanent

firewall-cmd --zone=public --add-port=9527/tcp --permanent

firewall-cmd --reload

1.2 重启docker服务

systemctl stop docker

systemctl restart docker

1.3 拉取datax_web镜像

docker pull linshellfeng/datax_web:3.0.1

注意：datax_web:3.0.1镜像已经整合了datax3.0和datax-web，整合后的镜像大小约990M

[root@datax ~]# docker pull linshellfeng/datax_web:3.0.1

3.0.1: Pulling from linshellfeng/datax_web

ac9208207ada: Pull complete

96c0123b30b0: Pull complete

a8db204c040d: Downloading [=================================> ] 47.28MB/70.51MB

b21c9e28dc6d: Downloading [===================> ] 84.28MB/217.6MB

8099a5760bef: Downloading [==============> ] 56.92MB/190.8MB

55b3abe48486: Waiting

2882620f39a9: Waiting

872999062d9e: Waiting

b21c9e28dc6d: Downloading

55b3abe48486: Downloading

unauthorized: authentication required

报错：unauthorized: authentication required

解决：注册docker账号

https://hub.docker.com/signup

[root@datax ~]# docker login

Log in with your Docker ID or email address to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com/ to create one.

You can log in with your password or a Personal Access Token (PAT). Using a limited-scope PAT grants better security and is required for organizations using SSO. Learn more at https://docs.docker.com/go/access-tokens/

Username: ********* ##输入注册的docker账号

Password: ********* ##输入docker密码

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.

Configure a credential helper to remove this warning. See

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

[root@datax ~]# cat /root/.docker/config.json

{

"auths": {

"https://index.docker.io/v1/": {

"auth": "Mjc3NTY4ODIxQHFxLmNvbToxYSMyQiQzYyU0RA=="

}

docker login

[root@datax ~]# docker pull linshellfeng/datax_web:3.0.1

3.0.1: Pulling from linshellfeng/datax_web

Digest: sha256:c64395010bf9552ed56bd0f5ff23e728db6c28ab4a7f5a1e2eee7d0796078fc0

Status: Image is up to date for linshellfeng/datax_web:3.0.1

docker.io/linshellfeng/datax_web:3.0.1

[root@datax ~]# docker image ls

REPOSITORY TAG IMAGE ID CREATED SIZE

linshellfeng/datax_web 3.0.1 7d1bcef9d5ee 21 months ago 974MB

二. 安装mysql数据库

2.1 安装步骤见docker安装mysql5.7

2.2 初始化datax_web_db数据库和配置datax_web账号密码：在mysql数据库节点上执行

使用root账号登录mysql数据库以后，使用以下脚本初始化datax_web_db数据库和配置datax_web账号密码，注意：xxxxxx替换为自定义的密码

create database datax_web_db default character set utf8mb4 collate utf8mb4_general_ci;

#create user 'datax_web'@'%' identified with mysql_native_password by 'xxxxxx';

CREATE USER 'datax_web'@'%' IDENTIFIED BY '123456';

grant all privileges on datax_web_db.* to 'datax_web'@'%';

flush privileges;

[root@mysql ~]# docker exec -it mysql bash

root@84c6627f5440:/# mysql -u root -p123456

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 3

Server version: 5.7.36 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| mysql |

| performance_schema |

| sys |

+--------------------+

4 rows in set (0.00 sec)

mysql> create database datax_web_db default character set utf8mb4 collate utf8mb4_general_ci;

Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'datax_web'@'%' IDENTIFIED BY '123456';

Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on datax_web_db.* to 'datax_web'@'%';

Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

mysql> exit

2.3 初始化datax_web_db数据库

#执行以下脚本，完成datax_web_db数据库初始化，满足datax_web服务应用需要

使用datax_web登录，然后切换到datax_web_db，复制datax_web.sql的内执行即可，这里不再详细介绍数据库方面的操作。

root@84c6627f5440:/# mysql -u datax_web -p123456

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 4

Server version: 5.7.36 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| datax_web_db |

+--------------------+

2 rows in set (0.00 sec)

mysql> use datax_web_db

#执行datax_web.sql 复制sql语句直接执行即可

#执行完成后，生成的表如下

mysql> show tables;

+------------------------+

| Tables_in_datax_web_db |

+------------------------+

| job_group |

| job_info |

| job_jdbc_datasource |

| job_lock |

| job_log |

| job_log_report |

| job_logglue |

| job_permission |

| job_project |

| job_registry |

| job_template |

| job_user |

+------------------------+

12 rows in set (0.00 sec)

以下为datax_web.sql的内容

Navicat Premium Data Transfer

Source Server : localhost

Source Server Type : MySQL

Source Server Version : 50725

Source Host : localhost:3306

Source Schema : datax_web

Target Server Type : MySQL

Target Server Version : 50725

File Encoding : 65001

Date: 15/12/2019 22:27:10

SET NAMES utf8mb4;

SET FOREIGN_KEY_CHECKS = 0;

-- ----------------------------

-- Table structure for job_group

-- ----------------------------

DROP TABLE IF EXISTS `job_group`;

CREATE TABLE `job_group` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`app_name` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '执行器AppName',

`title` varchar(12) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '执行器名称',

`order` int(11) NOT NULL DEFAULT 0 COMMENT '排序',

`address_type` tinyint(4) NOT NULL DEFAULT 0 COMMENT '执行器地址类型：0=自动注册、1=手动录入',

`address_list` varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器地址列表，多地址逗号分隔',

PRIMARY KEY (`id`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Records of job_group

-- ----------------------------

INSERT INTO `job_group` VALUES (1, 'datax-executor', 'datax执行器', 1, 0, NULL);

-- ----------------------------

-- Table structure for job_info

-- ----------------------------

DROP TABLE IF EXISTS `job_info`;

CREATE TABLE `job_info` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`job_group` int(11) NOT NULL COMMENT '执行器主键ID',

`job_cron` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '任务执行CRON',

`job_desc` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,

`add_time` datetime(0) NULL DEFAULT NULL,

`update_time` datetime(0) NULL DEFAULT NULL,

`author` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '作者',

`alarm_email` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '报警邮件',

`executor_route_strategy` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器路由策略',

`executor_handler` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器任务handler',

`executor_param` varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器任务参数',

`executor_block_strategy` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '阻塞处理策略',

`executor_timeout` int(11) NOT NULL DEFAULT 0 COMMENT '任务执行超时时间，单位秒',

`executor_fail_retry_count` int(11) NOT NULL DEFAULT 0 COMMENT '失败重试次数',

`glue_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT 'GLUE类型',

`glue_source` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL COMMENT 'GLUE源代码',

`glue_remark` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT 'GLUE备注',

`glue_updatetime` datetime(0) NULL DEFAULT NULL COMMENT 'GLUE更新时间',

`child_jobid` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '子任务ID，多个逗号分隔',

`trigger_status` tinyint(4) NOT NULL DEFAULT 0 COMMENT '调度状态：0-停止，1-运行',

`trigger_last_time` bigint(13) NOT NULL DEFAULT 0 COMMENT '上次调度时间',

`trigger_next_time` bigint(13) NOT NULL DEFAULT 0 COMMENT '下次调度时间',

`job_json` text CHARACTER SET utf8 COLLATE utf8_general_ci NULL COMMENT 'datax运行脚本',

PRIMARY KEY (`id`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 7 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Table structure for job_jdbc_datasource

-- ----------------------------

DROP TABLE IF EXISTS `job_jdbc_datasource`;

CREATE TABLE `job_jdbc_datasource` (

`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '自增主键',

`datasource_name` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '数据源名称',

`datasource_group` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT 'Default' COMMENT '数据源分组',

`jdbc_username` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '用户名',

`jdbc_password` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '密码',

`jdbc_url` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT 'jdbc url',

`jdbc_driver_class` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT 'jdbc驱动类',

`status` tinyint(1) NOT NULL DEFAULT 1 COMMENT '状态：0删除 1启用 2禁用',

`create_by` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '创建人',

`create_date` datetime(0) NULL DEFAULT CURRENT_TIMESTAMP(0) COMMENT '创建时间',

`update_by` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '更新人',

`update_date` datetime(0) NULL DEFAULT NULL COMMENT '更新时间',

`comments` varchar(1000) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '备注',

PRIMARY KEY (`id`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 6 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci COMMENT = 'jdbc数据源配置' ROW_FORMAT = Dynamic;

-- ----------------------------

-- Table structure for job_lock

-- ----------------------------

DROP TABLE IF EXISTS `job_lock`;

CREATE TABLE `job_lock` (

`lock_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '锁名称',

PRIMARY KEY (`lock_name`) USING BTREE

) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Records of job_lock

-- ----------------------------

INSERT INTO `job_lock` VALUES ('schedule_lock');

-- ----------------------------

-- Table structure for job_log

-- ----------------------------

DROP TABLE IF EXISTS `job_log`;

CREATE TABLE `job_log` (

`id` bigint(20) NOT NULL AUTO_INCREMENT,

`job_group` int(11) NOT NULL COMMENT '执行器主键ID',

`job_id` int(11) NOT NULL COMMENT '任务，主键ID',

`job_desc` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,

`executor_address` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器地址，本次执行的地址',

`executor_handler` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器任务handler',

`executor_param` varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器任务参数',

`executor_sharding_param` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器任务分片参数，格式如 1/2',

`executor_fail_retry_count` int(11) NULL DEFAULT 0 COMMENT '失败重试次数',

`trigger_time` datetime(0) NULL DEFAULT NULL COMMENT '调度-时间',

`trigger_code` int(11) NOT NULL COMMENT '调度-结果',

`trigger_msg` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL COMMENT '调度-日志',

`handle_time` datetime(0) NULL DEFAULT NULL COMMENT '执行-时间',

`handle_code` int(11) NOT NULL COMMENT '执行-状态',

`handle_msg` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL COMMENT '执行-日志',

`alarm_status` tinyint(4) NOT NULL DEFAULT 0 COMMENT '告警状态：0-默认、1-无需告警、2-告警成功、3-告警失败',

`process_id` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT 'datax进程Id',

`max_id` bigint(20) NULL DEFAULT NULL COMMENT '增量表max id',

PRIMARY KEY (`id`) USING BTREE,

INDEX `I_trigger_time`(`trigger_time`) USING BTREE,

INDEX `I_handle_code`(`handle_code`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 0 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Table structure for job_log_report

-- ----------------------------

DROP TABLE IF EXISTS `job_log_report`;

CREATE TABLE `job_log_report` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`trigger_day` datetime(0) NULL DEFAULT NULL COMMENT '调度-时间',

`running_count` int(11) NOT NULL DEFAULT 0 COMMENT '运行中-日志数量',

`suc_count` int(11) NOT NULL DEFAULT 0 COMMENT '执行成功-日志数量',

`fail_count` int(11) NOT NULL DEFAULT 0 COMMENT '执行失败-日志数量',

PRIMARY KEY (`id`) USING BTREE,

UNIQUE INDEX `i_trigger_day`(`trigger_day`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 28 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Records of job_log_report

-- ----------------------------

INSERT INTO `job_log_report` VALUES (20, '2019-12-07 00:00:00', 0, 0, 0);

INSERT INTO `job_log_report` VALUES (21, '2019-12-10 00:00:00', 77, 52, 23);

INSERT INTO `job_log_report` VALUES (22, '2019-12-11 00:00:00', 9, 2, 11);

INSERT INTO `job_log_report` VALUES (23, '2019-12-13 00:00:00', 9, 48, 74);

INSERT INTO `job_log_report` VALUES (24, '2019-12-12 00:00:00', 10, 8, 30);

INSERT INTO `job_log_report` VALUES (25, '2019-12-14 00:00:00', 78, 45, 66);

INSERT INTO `job_log_report` VALUES (26, '2019-12-15 00:00:00', 24, 76, 9);

INSERT INTO `job_log_report` VALUES (27, '2019-12-16 00:00:00', 23, 85, 10);

-- ----------------------------

-- Table structure for job_logglue

-- ----------------------------

DROP TABLE IF EXISTS `job_logglue`;

CREATE TABLE `job_logglue` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`job_id` int(11) NOT NULL COMMENT '任务，主键ID',

`glue_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT 'GLUE类型',

`glue_source` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL COMMENT 'GLUE源代码',

`glue_remark` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT 'GLUE备注',

`add_time` datetime(0) NULL DEFAULT NULL,

`update_time` datetime(0) NULL DEFAULT NULL,

PRIMARY KEY (`id`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Table structure for job_registry

-- ----------------------------

DROP TABLE IF EXISTS `job_registry`;

CREATE TABLE `job_registry` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`registry_group` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,

`registry_key` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,

`registry_value` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,

`update_time` datetime(0) NULL DEFAULT NULL,

PRIMARY KEY (`id`) USING BTREE,

INDEX `i_g_k_v`(`registry_group`, `registry_key`, `registry_value`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 26 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Table structure for job_user

-- ----------------------------

DROP TABLE IF EXISTS `job_user`;

CREATE TABLE `job_user` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`username` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '账号',

`password` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '密码',

`role` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '角色：0-普通用户、1-管理员',

`permission` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '权限：执行器ID列表，多个逗号分割',

PRIMARY KEY (`id`) USING BTREE,

UNIQUE INDEX `i_username`(`username`) USING BTREE

) ENGINE = InnoDB AUTO_INCREMENT = 10 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------

-- Records of job_user

-- ----------------------------

INSERT INTO `job_user` VALUES (1, 'admin', '$2a$10$2KCqRbra0Yn2TwvkZxtfLuWuUP5KyCWsljO/ci5pLD27pqR3TV1vy', 'ROLE_ADMIN', NULL);

/**

v2.1.1脚本更新

ALTER TABLE `job_info`

ADD COLUMN `replace_param` VARCHAR(100) NULL DEFAULT NULL COMMENT '动态参数' AFTER `job_json`,

ADD COLUMN `jvm_param` VARCHAR(200) NULL DEFAULT NULL COMMENT 'jvm参数' AFTER `replace_param`,

ADD COLUMN `time_offset` INT(11) NULL DEFAULT '0'COMMENT '时间偏移量' AFTER `jvm_param`;

/**

增量改版脚本更新

ALTER TABLE `job_info` DROP COLUMN `time_offset`;

ALTER TABLE `job_info`

ADD COLUMN `inc_start_time` DATETIME NULL DEFAULT NULL COMMENT '增量初始时间' AFTER `jvm_param`;

-- ----------------------------

-- Table structure for job_template

-- ----------------------------

DROP TABLE IF EXISTS `job_template`;

CREATE TABLE `job_template` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`job_group` int(11) NOT NULL COMMENT '执行器主键ID',

`job_cron` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '任务执行CRON',

`job_desc` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,

`add_time` datetime(0) NULL DEFAULT NULL,

`update_time` datetime(0) NULL DEFAULT NULL,

`user_id` int(11) NOT NULL COMMENT '修改用户',

`alarm_email` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '报警邮件',

`executor_route_strategy` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器路由策略',

`executor_handler` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器任务handler',

`executor_param` varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL COMMENT '执行器参数',