Hive安装及配置
安装Hive之前先安装MySQL,安装地址：http://blog.csdn.net/qq_33624294/article/details/53128202
装MySQL首先不要配置utf8编码，我配置以后导致后来出现hive删除表一直在处于卡死状态：http://blog.csdn.net/qq_33624294/article/details/53222266
本地模式安装如下：
先将Hive安装包apache-hive-0.13.1-bin.tar.gz和MySQL驱动包mysql-connector-java-5.1.26-bin.jar上传到Linux的/home/gznc目录下。
打开终端，输入cd,然后输入ls就能查看刚刚上传的文件，如下图所示
输入解压命令
[gznc@master ~]$ tar -xvf apache-hive-0.13.1-bin.tar.gz解压后输入ls命令会看到解压后的Hive
给Hive改名字，命令：
[gznc@master ~]$ cp -r apache-hive-0.13.1-bin hive-0.13.1
配置Hive环境，进入Hive配置界面，命令：
[gznc@master ~]$ vi /home/gznc/.bash_profile
启动命令让配置生效，命令：
source /home/gznc/.bash_profile进入hive-0.13.1/cnf配置界面，修改一些参数
[gznc@master ~]$ cd /home/gznc/hive-0.13.1/conf/
[gznc@master conf]$ ls
hive-default.xml.template hive-exec-log4j.properties.template
hive-env.sh.template hive-log4j.properties.template
[gznc@master conf]$ cp hive-default.xml.template hive-site.xml
[gznc@master conf]$ cp hive-env.sh.template hive-env.sh
[gznc@master conf]$ cp hive-log4j.properties.template hive-log4j.properties
[gznc@master conf]$ ls
hive-default.xml.template hive-exec-log4j.properties.template hive-site.xml
hive-env.sh hive-log4j.properties
hive-env.sh.template hive-log4j.properties.template
配置hive-env.sh，命令：
[gznc@master conf]$ gedit hive-env.sh
具体如下图所示：
配置hive-site.xml，命令：
[gznc@master conf]$ gedit hive-site.xml快捷键：ctrl + F，搜索hive.metastore.warehouse.dir
修改为如下界面
快捷键：ctrl + F，搜索hive.exec.scratchdir
修改为如下界面
快捷键：ctrl + F，搜索hive.querylog.location
修改为如下界面
快捷键：ctrl + F，搜索javax.jdo.option.ConnectionURL
修改为如下界面
快捷键：ctrl + F，搜索javax.jdo.option.ConnectionDriverName
修改为如下界面
快捷键：ctrl + F，搜索javax.jdo.option.ConnectionUserName
修改为如下界面
快捷键：ctrl + F，搜索javax.jdo.option.ConnectionPassword
修改为如下界面
快捷键：ctrl + F，搜索hive.metastore.ds.retry
修改为如下界面
配置hive-log4j.propertites，命令：
[gznc@master conf]$ gedit hive-log4j.properties快捷键：ctrl + F，搜索hive.log.dir
修改为如下界面
将MySQL驱动包mysql-connector-java-5.1.26-bin.jar上传到Hive的lib下，命令：
[gznc@master ~]$ cd
[gznc@master ~]$ ls
apache-hive-0.13.1-bin.tar.gz MySQL-client-5.5.52-1.linux2.6.x86_64.rpm
data.txt mysql-connector-java-5.1.26-bin.jar
Desktop MySQL-server-5.5.52-1.linux2.6.x86_64.rpm
Documents Pictures
Downloads pig-0.13.0
eclipse pig-0.13.0.tar.gz
hadoop-2.5.2 pig_1478425455921.log
hadoop-2.5.2.tar.gz pig_1478429632247.log
hadoopdata Public
hbase-0.98.7-hadoop2 Templates
hbase-0.98.7-hadoop2-bin.tar.gz Videos
hive-0.13.1 workspace
Music zookeeper
[gznc@master ~]$ mv mysql-connector-java-5.1.26-bin.jar /home/gznc/hive-0.13.1/lib/
[gznc@master ~]$ cd /home/gznc/hive-0.13.1/lib/
[gznc@master lib]$ ls
创建一个hive用户，用root用户进入mysql，命令
[gznc@master conf]$ cd
[gznc@master ~]$ mysql -u root -p123456
mysql> create user 'hive' identified by '123456';
Query OK, 0 rows affected (0.07 sec)
mysql> grant all privileges on *.* to 'hive'@'%' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.03 sec)
mysql> quitquit退出来后，用刚刚创建的hive用户去登陆，如下图，但是报错
上面错误的解决方法就是删除用户表中user=空，命令如下：
[gznc@master ~]$ mysql -u root -p123456
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.5.52-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test |
| tt |
+--------------------+
5 rows in set (0.00 sec)
mysql> use mysql;
Database changed
mysql> select user ,host from user;
+------+-----------+
| user | host |
+------+-----------+
| hive | % |
| root | 127.0.0.1 |
| root | ::1 |
| | localhost |
| root | localhost |
| | master |
| root | master |
+------+-----------+
7 rows in set (0.03 sec)
mysql> delete from user where user='';
Query OK, 2 rows affected (0.05 sec)
mysql> select user ,host from user;
+------+-----------+
| user | host |
+------+-----------+
| hive | % |
| root | 127.0.0.1 |
| root | ::1 |
| root | localhost |
| root | master |
+------+-----------+
5 rows in set (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)
mysql> quit
然后用hive用户登录mysql，命令：
[gznc@master ~]$ mysql -u hive -p123456
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.5.52-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> 
启动hive，命令如下：
[gznc@master ~]$ hive
Logging initialized using configuration in file:/home/gznc/hive-0.13.1/conf/hive-log4j.properties
hive> create table hh (id int);
OK
Time taken: 1.868 seconds
hive> show tables;
OK
hh
Time taken: 0.222 seconds, Fetched: 1 row(s)如果在hive中建立表报如下错误
hive> create table hh (id int);
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: An exception was thrown while adding/validating class(es) : Specified key was too long; max key length is 767 bytes
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4190)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4122)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2812)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2761)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:894)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:732)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:254)
at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(AbstractTable.java:760)解决办法如下：
gznc@master ~]$ mysql -u hive -p123456
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 51
Server version: 5.5.52-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> alter database hive character set latin1;
Query OK, 1 row affected (0.20 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.12 sec)
mysql> 测试hive
hive> create table student(id int,name string) row format delimited fields terminated by ',';
hive> show tables;
OK
student
hive> load data local inpath '/home/gznc/Desktop/student.txt' overwrite into table student;
hive> select * from student;
OK
1 zhangsan
2 lisi
3 wangwu
4 zhaoliu进入mysql
gznc@master ~]$ mysql -u hive -p123456
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 51
Server version: 5.5.52-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive |
| mysql |
| performance_schema |
| test |
| tt |
+--------------------+
6 rows in set (0.07 sec)
mysql> use hive
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_hive |
+---------------------------+
| BUCKETING_COLS |
| CDS |
| COLUMNS_V2 |
| DATABASE_PARAMS |
| DBS |
| FUNCS |
| FUNC_RU |
| GLOBAL_PRIVS |
| IDXS |
| INDEX_PARAMS |
| PARTITIONS |
| PARTITION_KEYS |
| PARTITION_KEY_VALS |
| PARTITION_PARAMS |
| PART_COL_STATS |
| PART_PRIVS |
| ROLES |
| SDS |
| SD_PARAMS |
| SEQUENCE_TABLE |
| SERDES |
| SERDE_PARAMS |
| SKEWED_COL_NAMES |
| SKEWED_COL_VALUE_LOC_MAP |
| SKEWED_STRING_LIST |
| SKEWED_STRING_LIST_VALUES |
| SKEWED_VALUES |
| SORT_COLS |
| TABLE_PARAMS |
| TAB_COL_STATS |
| TBLS |
| TBL_COL_PRIVS |
| TBL_PRIVS |
| VERSION |
+---------------------------+
34 rows in set (0.00 sec)
mysql> select * from TBLS;
+--------+-------------+-------+------------------+-------+-----------+-------+----------+---------------+--------------------+--------------------+
| TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER | RETENTION | SD_ID | TBL_NAME | TBL_TYPE | VIEW_EXPANDED_TEXT | VIEW_ORIGINAL_TEXT |
+--------+-------------+-------+------------------+-------+-----------+-------+----------+---------------+--------------------+--------------------+
| 1 | 1479387915 | 1 | 0 | gznc | 0 | 1 | student | MANAGED_TABLE | NULL | NULL |
+--------+-------------+-------+------------------+-------+-----------+-------+----------+---------------+--------------------+--------------------+
2 rows in set (0.00 sec)以上本地模式就配置好了
若要将hive的表数据存储到hdfs上面，可以在hdfs上创建相应的文件夹，如：
[gznc@master ~]$ hadoop fs -mkdir /user/gznc/hive
[gznc@master ~]$ hadoop fs -mkdir /user/gznc/hive/warehouse
[gznc@master ~]$ hadoop fs -mkdir /user/gznc/hive/scratchdir
$hadoop dfs -chmod -R 777 /user/hadoop/hive/(更改hive目录下所有文件及文件夹的权限)
然后更改hive-site.xml文件中的第一项和第二项的地址
hive.metastore.warehouse.dir=hdfs://master:9000/user/gznc/hive/warehouse
hive.exec.scratchdir=hdfs://master:9000/user/gznc/hive/scratchdir
hive> create table student(id int,name string) row format delimited fields terminated by '\t';
OK
Time taken: 0.894 seconds
hive> show tables;
OK
student
Time taken: 0.1 seconds, Fetched: 1 row(s)
hive> load data local inpath '/home/gznc/Desktop/student.txt' overwrite into table student;
Copying data from file:/home/gznc/Desktop/student.txt
Copying file: file:/home/gznc/Desktop/student.txt
Loading data to table default.student
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted hdfs://master:9000/user/gznc/hive/warehouse/student
Table default.student stats: [numFiles=1, numRows=0, totalSize=39, rawDataSize=0]
OK
Time taken: 1.521 seconds
hive> select * from student;
OK
1 张三
2 xiaohong
3 gang
4 jun
5 mei
[gznc@master ~]$ hadoop fs -ls /user/gznc/hive/warehouse/student
16/11/18 20:33:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r-- 1 gznc supergroup 39 2016-11-18 20:31 /user/gznc/hive/warehouse/student/student.txt
[gznc@master ~]$ hadoop fs -ls /user/gznc/hive/warehouse/student/student.txt
16/11/18 20:33:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r-- 1 gznc supergroup 39 2016-11-18 20:31 /user/gznc/hive/warehouse/student/student.txt
[gznc@master ~]$ hadoop fs -cat /user/gznc/hive/warehouse/student/student.txt
16/11/18 20:33:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1 张三
2 xiaohong
3 gang
4 jun
5 mei