Hive 将元数据存储在 RDBMS 中,有三种模式可以连接到数据库:
1)ingle User Mode: 此模式连接到一个 In-memory 的数据库 Derby,一般用于 Unit Test。
2)Multi User Mode:通过网络连接到一个数据库中,是最经常使用到的模式。
3)Remote Server Mode:用于非 Java 客户端访问元数据库,在服务器端启动一个 MetaStoreServer,客户端利用 Thrift 协议通过 MetaStoreServer 访问元数据库。
Hive默认是采用Derby来存储其Meta信息的,
现在我们需要修改为mysql
1.在mysql专门为hive添加用户
mysql>grant all PRIVILEGES on *.* to hive@'123.123.123.123' identified by '123456';
mysql> flush privileges;
2.修改配置文件conf/hive-default.xml 中的配置
<property> <name>hive.metastore.local</name> <value>true</value> <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hivepasswd</value> <description>password to use against metastore database</description> </property> <property> <name>datanucleus.autoCreateSchema</name> <value>false</value> </property> <property> <name>datanucleus.fixedDatastore</name> <value>true</value> </property> <property> <name>hive.support.concurrency</name> <value>true</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>node1,node2,node3</value> <description>The list of zookeeper servers to talk to. This is only needed for read/write locks.</description> </property>
datanucleus.fixedDatastore 选项设置主要是解决并发造成的错误:
Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'dataoven_prod_hadoop.DELETEME1309959999747' doesn't exist
hive.support.concurrency 设置为支持并发.
3.添加jdbc的jar包
wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.11.tar.gz/from/http://mysql.he.net/
tar -xvzf mysql-connector-java-5.1.11.tar.gz
cp mysql-connector-java-5.1.11/*.jar /data/soft/hive/lib
4.启动hive
bin/hive
hive> show tables;
When using MySQL as a metastore I see the error "com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes".
* This is a known limitation of MySQL 5.0 and UTF8 databases. One option is to use another character set, such as 'latin1', which is known to work.
这个问题是因为hive对mysql的UTF-8编码方式有限制,修改一下mysql的编码方式即可:alter database name character set latin1;
FAILED: Error in metadata: javax.jdo.JDOException: Couldnt obtain a new sequence (unique id) : Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT'
因为,READ-COMMITTED需要把bin-log以mixed方式来记录,用以下命令来修改:
set global binlog_format='MIXED';
5. 使用hive分析日志作业很多的时候,需要修改mysql的默认连接数
修改方法 打开/etc/my.cnf文件
在[mysqld] 中添加 max_connections=1000
重启mysql服务 service mysqld restart
查看设置是否成功 mysql -uroot -p
mysql>show variables like '%max_connections%';
查看当前mysql的连接数方法
mysqladmin -uroot -p status
其中,Uptime:mysqld运行时间,单位秒。 Threads: 当前连接数。 Questions: 从启动以来的查询数目。 Slow queries:慢查询数目。 Opens: 从启动以来打开过的表数目。 Flush tables: 从启动以来flush的表数目。 Open tables: 当前打开的表数目。 Queries per second avg:平均每秒查询数目。
分别查看mysqld的运行情况和活动连接列表 mysqladmin -u root -p processlist status