hive写入Elasticsearch参数设置


1. 在hive中增加number of Hadoop input splits

Reduce the input split size from the default value. The mappers will get increased.

SET mapreduce.input.fileinputformat.split.maxsize;

参考:http://stackoverflow.com/questions/30222032/increase-number-of-hive-mappers-in-hadoop-2

参考:https://hadoopjournal.wordpress.com/2015/06/13/set-mappers-in-pig-hive-and-mapreduce/


2. Elasticsearch-Hadoop参数

es.batch.size.bytes (default 1mb)

es.batch.size.entries (default 1000)

es.batch.write.refresh (default true)

参考:https://www.elastic.co/guide/en/elasticsearch/hadoop/current/performance.html#_decrease_bulk_size

参考:https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html


你可能感兴趣的:(hive写入Elasticsearch参数设置)