Doris和Mysql的批同步测试

Doris和Mysql测试结果

同步类型	更新表（列级）	更新表（行级）	追加表	覆盖表
Doris->Mysql	支持	支持	支持	支持
Mysql->Doris	支持	不支持	不支持	不支持

Mysql->Doris不支持更新表（行级）、追加表、覆盖表的原因:Mysql的load data的语法和doris的语法有区别，需要适配！

Doris的Load Data的语法

mysql-load: 使用MySql客户端导入本地数据

该语句用于向指定的 table 导入数据，与普通Load区别是，这种导入方式是同步导入。

这种导入方式仍然能够保证一批导入任务的原子性，要么全部数据导入成功，要么全部失败。

1. MySQL Load以语法LOAD DATA开头, 无须指定LABEL

2. 指定LOCAL表示读取客户端文件.不指定表示读取FE服务端本地文件. 导入FE本地文件的功能默认是关闭的, 需要在FE节点上设置mysql_load_server_secure_path来指定安全路径, 才能打开该功能.

3. INFILE内填写本地文件路径, 可以是相对路径, 也可以是绝对路径.目前只支持单个文件, 不支持多个文件

4. INTO TABLE的表名可以指定数据库名, 如案例所示. 也可以省略, 则会使用当前用户所在的数据库.

5. PARTITION语法支持指定分区导入

6. COLUMNS TERMINATED BY指定列分隔符

7. LINES TERMINATED BY指定行分隔符

8. IGNORE num LINES用户跳过CSV的表头, 可以跳过任意行数. 该语法也可以用IGNORE num ROWS代替

9. 列映射语法, 具体参数详见导入的数据转换 的列映射章节

10. PROPERTIES参数配置, 详见下文

Doris的官网：https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD

大数据平台的Load data的语法

LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' 
INTO TABLE `temp_e4d256bf87398d45` FIELDS TERMINATED BY '' 
OPTIONALLY ENCLOSED BY '' 
ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);

Mysql->Doris不支持更新表（行级）的报错信息：

2023/03/24 10:30:29 - unique_test_output.0 - Starting the null bulk Load in a separate thread - LOAD DATA LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' INTO TABLE `temp_e4d256bf87398d45` FIELDS TERMINATED BY '' OPTIONALLY ENCLOSED BY '' ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);
2023/03/24 10:30:29 - unique_test_output.0 - Opening fifo /tmp/5e5c2e727e2e409a8d5ec25d156ca0cf for writing.
2023/03/24 10:30:29 - unique_test_output.0 - Make sure user has been granted the FILE privilege.
2023/03/24 10:30:29 - unique_test_output.0 - 
2023/03/24 10:30:29 - unique_test_output.0 - Error in step, asking everyone to stop because of:
2023/03/24 10:30:29 - unique_test_output.0 - org.pentaho.di.core.exception.KettleException: 
2023/03/24 10:30:29 - unique_test_output.0 - org.pentaho.di.core.exception.KettleDatabaseException: 
2023/03/24 10:30:29 - unique_test_output.0 - Couldn't execute SQL: LOAD DATA LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' INTO TABLE `temp_e4d256bf87398d45` FIELDS TERMINATED BY '' OPTIONALLY ENCLOSED BY '' ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);
2023/03/24 10:30:29 - unique_test_output.0 - 
2023/03/24 10:30:29 - unique_test_output.0 - errCode = 2, detailMessage = Syntax error in line 1:
2023/03/24 10:30:29 - unique_test_output.0 - LOAD DATA LOCAL INFILE '/tmp/5e5c...
2023/03/24 10:30:29 - unique_test_output.0 -      ^
2023/03/24 10:30:29 - unique_test_output.0 - Encountered: DATA
2023/03/24 10:30:29 - unique_test_output.0 - Expected
2023/03/24 10:30:29 - unique_test_output.0 - 
2023/03/24 10:30:29 - unique_test_output.0 - 
2023/03/24 10:30:29 - unique_test_output.0 - 

Mysql->Doris不支持追加表的报错信息：

2023/03/24 10:32:00 - unique_test_output.0 - Starting the null bulk Load in a separate thread - LOAD DATA LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' INTO TABLE `temp_e4d256bf87398d45` FIELDS TERMINATED BY '' OPTIONALLY ENCLOSED BY '' ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);
2023/03/24 10:32:00 - unique_test_output.0 - Opening fifo /tmp/5e5c2e727e2e409a8d5ec25d156ca0cf for writing.
2023/03/24 10:32:00 - unique_test_output.0 - Make sure user has been granted the FILE privilege.
2023/03/24 10:32:00 - unique_test_output.0 - 
2023/03/24 10:32:00 - unique_test_output.0 - Error in step, asking everyone to stop because of:
2023/03/24 10:32:00 - unique_test_output.0 - org.pentaho.di.core.exception.KettleException: 
2023/03/24 10:32:00 - unique_test_output.0 - org.pentaho.di.core.exception.KettleDatabaseException: 
2023/03/24 10:32:00 - unique_test_output.0 - Couldn't execute SQL: LOAD DATA LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' INTO TABLE `temp_e4d256bf87398d45` FIELDS TERMINATED BY '' OPTIONALLY ENCLOSED BY '' ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);
2023/03/24 10:32:00 - unique_test_output.0 - 
2023/03/24 10:32:00 - unique_test_output.0 - errCode = 2, detailMessage = Syntax error in line 1:
2023/03/24 10:32:00 - unique_test_output.0 - LOAD DATA LOCAL INFILE '/tmp/5e5c...
2023/03/24 10:32:00 - unique_test_output.0 -      ^
2023/03/24 10:32:00 - unique_test_output.0 - Encountered: DATA
2023/03/24 10:32:00 - unique_test_output.0 - Expected
2023/03/24 10:32:00 - unique_test_output.0 - 
2023/03/24 10:32:00 - unique_test_output.0 - 
2023/03/24 10:32:00 - unique_test_output.0 - 

Mysql->Doris不支持覆盖表的报错信息：

2023/03/24 10:32:47 - unique_test_output.0 - Starting the null bulk Load in a separate thread - LOAD DATA LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' INTO TABLE `unique_test` FIELDS TERMINATED BY '' OPTIONALLY ENCLOSED BY '' ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);
2023/03/24 10:32:47 - unique_test_output.0 - Opening fifo /tmp/5e5c2e727e2e409a8d5ec25d156ca0cf for writing.
2023/03/24 10:32:48 - unique_test_output.0 - Make sure user has been granted the FILE privilege.
2023/03/24 10:32:48 - unique_test_output.0 - 
2023/03/24 10:32:48 - unique_test_output.0 - Error in step, asking everyone to stop because of:
2023/03/24 10:32:48 - unique_test_output.0 - org.pentaho.di.core.exception.KettleException: 
2023/03/24 10:32:48 - unique_test_output.0 - org.pentaho.di.core.exception.KettleDatabaseException: 
2023/03/24 10:32:48 - unique_test_output.0 - Couldn't execute SQL: LOAD DATA LOCAL INFILE '/tmp/5e5c2e727e2e409a8d5ec25d156ca0cf' INTO TABLE `unique_test` FIELDS TERMINATED BY '' OPTIONALLY ENCLOSED BY '' ESCAPED BY '' (`user_id`,`date`,`city`,`age`,`sex`);
2023/03/24 10:32:48 - unique_test_output.0 - 
2023/03/24 10:32:48 - unique_test_output.0 - errCode = 2, detailMessage = Syntax error in line 1:
2023/03/24 10:32:48 - unique_test_output.0 - LOAD DATA LOCAL INFILE '/tmp/5e5c...
2023/03/24 10:32:48 - unique_test_output.0 -      ^
2023/03/24 10:32:48 - unique_test_output.0 - Encountered: DATA
2023/03/24 10:32:48 - unique_test_output.0 - Expected
2023/03/24 10:32:48 - unique_test_output.0 - 
2023/03/24 10:32:48 - unique_test_output.0 - 
2023/03/24 10:32:48 - unique_test_output.0 -