StreamSet mysql数据导入到hive异常问题记录

1. 异常问题Permission denied: user [anonymous] does not have [USE] privilege on [default]

com.streamsets.pipeline.api.base.OnRecordErrorException: HIVE_23 - TBL Properties 'com.streamsets.pipeline.stage.lib.hive.exceptions.HiveStageCheckedException: HIVE_20 - Error executing SQL: DESCRIBE DATABASE `default`, Reason:Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [default]' Mismatch: Actual: {} , Expected: {}
  at   com.streamsets.pipeline.stage.processor.hive.HiveMetadataProcessor.process(HiveMetadataProcessor.java:585)
	at com.streamsets.pipeline.api.base.RecordProcessor.process(RecordProcessor.java:52)
	at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
	at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:286)
	at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:235)
	at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:298)
	at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:244)


解决办法:

在集群的ranger中为hive服务添加{user} 群组的权限。

StreamSet mysql数据导入到hive异常问题记录_第1张图片

2.异常问题二

using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead

原因:因为Steamsets中配置的hive metadata存在格式是AVRO格式,而实际上创建的表格式和AVRO格式不一致。

解决办法:删掉之前的表,重新建一个存储格式是AVRO的表

CREATE TABLE `test3`(
  `id` int,
  `name` string)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  LINES TERMINATED BY '\n'
STORED AS AVRO
LOCATION
  'hdfs://ns1/apps/hive/warehouse/test3';
 

你可能感兴趣的:(Steamsets)