1、hadoop集群(伪分布式也可以)
2、mysql数据库
这里假设hadoop集群和mysql数据库都安装好了。
1、创建hive元数据存放库
mysql> create database hivemeta character set = latin1;
Query OK, 1 row affected (0.00 sec)
2、创建hive数据库用户
mysql> grant all on hivemeta.* to hiveuser@'%' identified by 'hiveuser';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on hivemeta.* to hiveuser@'localhost' identified by 'hiveuser';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on hivemeta.* to hiveuser@'hbaselx' identified by 'hiveuser';
Query OK, 0 rows affected (0.05 sec)
mysql> use mysql;
Database changed
mysql> select user,host,password from user where user='hiveuser';
+----------+-----------+-------------------------------------------+
| user | host | password |
+----------+-----------+-------------------------------------------+
| hiveuser | % | *9B0A19544652EC0C7748682AB93554C85080C326 |
| hiveuser | localhost | *9B0A19544652EC0C7748682AB93554C85080C326 |
+----------+-----------+-------------------------------------------+
2 rows in set (0.00 sec)
mysql> exit
Bye
[hadoop@hbaselx ~]$ mysql -uhiveuser -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 5.1.71 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
1、解压
[hadoop@hbaselx ~]$ ls
apache-hive-1.2.2-bin.tar.gz app hadoop-2.6.5.tar.gz jdk-8u172-linux-x64.tar.gz
[hadoop@hbaselx ~]$ tar -zxvf apache-hive-1.2.2-bin.tar.gz -C app
2、复制配置文件
[hadoop@hbaselx ~]$ cd app
[hadoop@hbaselx app]$ ls
apache-hive-1.2.2-bin hadoop-2.6.5 jdk1.8.0_172
[hadoop@hbaselx app]$ cd apache-hive-1.2.2-bin/conf
[hadoop@hbaselx conf]$ ls
beeline-log4j.properties.template hive-env.sh.template hive-log4j.properties.template
hive-default.xml.template hive-exec-log4j.properties.template ivysettings.xml
[hadoop@hbaselx conf]$ cp hive-default.xml.template hive-site.xml
[hadoop@hbaselx conf]$ cp hive-env.sh.template hive-env.sh
[hadoop@hbaselx conf]$ cp hive-log4j.properties.template hive-log4j.properties
[hadoop@hbaselx conf]$ cp hive-exec-log4j.properties.template hive-exec-log4j.properties
[hadoop@hbaselx conf]$
3、在hdfs上创建hive需要的目录
[hadoop@hbaselx conf]$ hdfs dfs -mkdir -p /hive/warehouse
18/06/14 10:54:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hbaselx conf]$ hdfs dfs -mkdir -p /hive/log
18/06/14 10:54:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hbaselx conf]$ hdfs dfs -mkdir -p /hive/tmp
18/06/14 10:54:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hbaselx conf]$
[hadoop@hbaselx apache-hive-1.2.2-bin]$ hdfs dfs -chmod -R 777 /hive
4、在本地创建如下目录
[hadoop@hbaselx apache-hive-1.2.2-bin]$ mkdir log
[hadoop@hbaselx apache-hive-1.2.2-bin]$ mkdir exec
[hadoop@hbaselx apache-hive-1.2.2-bin]$ mkdir downloadedsource
5、配置hive-site.xml
hive-site.xml文件里配置项很多,我们只需要修改下面这些即可。
<property>
<name>javax.jdo.option.ConnectionDriverNamename>
<value>com.mysql.jdbc.Drivervalue>
<description>Driver class name for a JDBC metastoredescription>
property>
<property>
<name>javax.jdo.option.ConnectionUserNamename>
<value>hiveuservalue>
<description>Username to use against metastore databasedescription>
property>
<property>
<name>javax.jdo.option.ConnectionPasswordname>
<value>hiveuservalue>
<description>password to use against metastore databasedescription>
property>
<property>
<name>javax.jdo.option.ConnectionURLname>
<value>jdbc:mysql://192.168.16.66:3306/hivemetavalue>
<description>JDBC connect string for a JDBC metastoredescription>
property>
<property>
<name>hive.metastore.urisname>
<value/>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.description>
property>
<property>
<name>hive.metastore.warehouse.dirname>
<value>/hive/warehousevalue> //hive数据文件在hdfs上存放目录
<description>location of default database for the warehousedescription>
property>
<property>
<name>hive.exec.local.scratchdirname>
<value>/home/hadoop/app/apache-hive-1.2.2-bin/execvalue>
<description>Local scratch space for Hive jobsdescription>
property>
<property>
<name>hive.downloaded.resources.dirname>
<value>/home/hadoop/app/apache-hive-1.2.2-bin/downloadedsourcevalue>
<description>Temporary local directory for added resources in the remote file system.description>
property>
6、配置hive-exec-log4j.properties和hive-log4j.properties
修改hive.log.dir为:
hive.log.dir=/home/hadoop/app/apache-hive-1.2.2-bin/log
7、拷贝mysql连接驱动包到${HIVE_HOME}/lib
[hadoop@hbaselx ~]$ cp mysql-connector-java-5.1.38.jar app/apache-hive-1.2.2-bin/lib
[hadoop@hbaselx bin]$ ./hive --service metastore &
[1] 25981
[hadoop@hbaselx bin]$ Starting Hive Metastore Server
[hadoop@hbaselx bin]$
[hadoop@hbaselx bin]$ ./hive
Logging initialized using configuration in file:/home/hadoop/app/apache-hive-1.2.2-bin/conf/hive-log4j.properties
hive> create table t1
> (id int,
> name string);
OK
Time taken: 1.665 seconds
hive> show tables;
OK
t1
Time taken: 0.236 seconds, Fetched: 1 row(s)
hive> select * from t1;
OK
Time taken: 0.672 seconds
问题1:
[hadoop@hbaselx bin]$ ./hive
Logging initialized using configuration in file:/home/hadoop/app/apache-hive-1.2.2-bin/conf/hive-log4j.properties
[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
at jline.TerminalFactory.create(TerminalFactory.java:101)
at jline.TerminalFactory.get(TerminalFactory.java:158)
at jline.console.ConsoleReader.(ConsoleReader.java:229)
at jline.console.ConsoleReader.(ConsoleReader.java:221)
at jline.console.ConsoleReader.(ConsoleReader.java:209)
at org.apache.hadoop.hive.cli.CliDriver.setupConsoleReader(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:721)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
at jline.console.ConsoleReader.(ConsoleReader.java:230)
at jline.console.ConsoleReader.(ConsoleReader.java:221)
at jline.console.ConsoleReader.(ConsoleReader.java:209)
at org.apache.hadoop.hive.cli.CliDriver.setupConsoleReader(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:721)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
解决方式:
把 ${HADOOP_HOME}/share/hadoop/yarn/lib/的 jline 删了,换成hive自带的2.12版本的
rm ${HADOOP_HOME}/share/hadoop/yarn/lib/jline-0.9.94.jar
cp ${HIVE_HOME}/lib/jline-2.12.jar ${HADOOP_HOME}/share/hadoop/yarn/lib/
问题2:
[hadoop@hbaselx bin]$ ./hive
Logging initialized using configuration in file:/home/hadoop/app/apache-hive-1.2.2-bin/conf/hive-log4j.properties
hive> create table t1(
> id int,
> name string
> );
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)
解决方式:
数据库的字符集不对,应该是latin1。
mysql> show create database hivemeta\G
*************************** 1. row ***************************
Database: hivemeta
Create Database: CREATE DATABASE `hivemeta` /*!40100 DEFAULT CHARACTER SET utf8 */
1 row in set (0.00 sec)
mysql> drop database hivemeta;
Query OK, 22 rows affected (0.16 sec)
mysql> create database hivemeta;
Query OK, 1 row affected (0.00 sec)
mysql> show create database hivemeta\G
*************************** 1. row ***************************
Database: hivemeta
Create Database: CREATE DATABASE `hivemeta` /*!40100 DEFAULT CHARACTER SET latin1 */
1 row in set (0.00 sec)
然后重新启动metastore服务即可。