DATAX从hdfs导入数据到mysql

json配置如下:

{
	"job": {
		"setting": {
			"speed": {
				"channel": 1
			}
		},
		"content": [{
			"reader": {
				"name": "hdfsreader",
				"parameter": {
					"path": "/user/hive/warehouse/ads.db/ads_member_active_count/dt=$do_date/*",
					"defaultFS": "hdfs://centos7-1:9000",
					"column": [{
						"type": "string",
						"value": "$do_date"
					}, {
						"index": 0,
						"type": "string"
					}, {
						"index": 1,
						"type": "string"
					}, {
						"index": 2,
						"type": "string"
					}],
					"fileType": "text",
					"encoding": "UTF-8",
					"fieldDelimiter": ","
				}
			},
			"writer": {
				"name": "mysqlwriter",
				"parameter": {
					"writeMode": "replace",
					"username": "root",
					"password": "12345678",
					"column": ["dt", "day_count", "week_count", "month_count"],
					"connection": [{
						"jdbcUrl": "jdbc:mysql://centos7-3:3306/dwads?useUnicode=true&characterEncoding=utf-8",
						"table": ["ads_member_active_count"]
					}]
				}
			}
		}]
	}
}

执行上面的配置:

python $DATAX_HOME/bin/datax.py -p "-Ddo_date=2020-07-21" /opt/lagou/servers/datax/job/export_member_active_count.json

但是报错,报错情况如下:

java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at com.alibaba.datax.common.util.RetryUtil$Retry.doRetry(RetryUtil.java:148)
	at com.alibaba.datax.common.util.RetryUtil.executeWithRetry(RetryUtil.java:30)
	at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:319)
	at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:303)
	at com.alibaba.datax.plugin.rdbms.util.JdbcConnectionFactory.getConnecttion(JdbcConnectionFactory.java:27)
	at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:105)
	at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:140)
	at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:35)
	at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41)
	at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Job.init(MysqlWriter.java:31)
	at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
	at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
	at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
	at com.alibaba.datax.core.Engine.start(Engine.java:92)
	at com.alibaba.datax.core.Engine.entry(Engine.java:171)
	at com.alibaba.datax.core.Engine.main(Engine.java:204)

DATAX从hdfs导入数据到mysql_第1张图片
提示数据库连接不上,密码或者用户错误,可是新开一个窗口,使用该用户名和密码,是可以登录的
DATAX从hdfs导入数据到mysql_第2张图片
然后尝试使用SQLyog去连接这个mysql,也是显示连接失败,本机可以登录,但是外部连接都失败
检查防火墙,这些全都关闭了的,应该不是是防火墙的问题,所以推测是mysql权限没有放开,查看mysql的权限如下:
DATAX从hdfs导入数据到mysql_第3张图片
果然是权限问题,在不同机器上都装了mysql,这台机器忘了给root用户加上允许外部连接的权限了,执行如下语句:
1、删除已经存在的root账号

delect from user where host = 'localhost' and user = 'root';

2、创建一个root新的账号

CREATE USER 'root'@'%' IDENTIFIED BY '12345678';

3、给root用户分配权限

GRANT ALL ON *.* TO 'root'@'%';
flush privileges;

完成上面三步,就可以啦
DATAX从hdfs导入数据到mysql_第4张图片
再执行datax导出数据,执行成功
DATAX从hdfs导入数据到mysql_第5张图片
解决了一个很小的问题,但是有点开心,哈哈哈哈

你可能感兴趣的:(datax,mysql,大数据)