CDH 集群 SQOOP 集群增量同步任务 使用MySQL作为metastore 储存增量同步任务 2021-03-09

1. 进入cdh集群sqoop配置管理界面

2. 找到sqoop-conf/sqoop-site.xml 的 Sqoop 1 Client 客户端高级配置代码段（安全阀）配置

3. 选择xml配置

4. 新增

    sqoop.metastore.client.enable.autoconnect
    true
    If true, Sqoop will connect to a local metastore for job management when no other metastore arguments are provided.

sqoop.metastore.client.autoconnect.url
jdbc:mysql://data1:3306/sqoop?createDatabaseIfNotExist=true

sqoop.metastore.client.autoconnect.username
root

sqoop.metastore.client.autoconnect.password
password

5.创建增量同步任务

sqoop job --create task_name \
-- import --connect 'jdbc:mysql://data2:3306/xxx?rewriteBatchedStatements=true&autoReconnect=true&useSSL=false' \
--username root --password-file /user/hdfs/password \
--query "select * from tb where \$CONDITIONS" \
--split-by id \
--fields-terminated-by '\01' \
--lines-terminated-by '\n' \
--target-dir "/sqoop/tb/" -m 1 \
--hive-delims-replacement ' ' \
--incremental append \
--check-column id \
--last-value 0

6. 在集群任意机器执行任务
   sqoop job --exec task_name