Hortonwork hdp3.1平台配置使用hplsql及测试
一、hplsql安装配置
Hortonwork hdp3.0平台自带的hive3已集成了hplsql，无需安装，但是需要手工新建配置文件hplsql-site.xml，并更新对应的hive-hpsql-xxx.jar文件(xxx是版本号）。
-
配置文件的样例
主要是修改hplsql的默认连接串hplsql.conn.default、hive的连接配置信息hplsql.conn.hive2conn，还有后面测试用到的mysql数据库的连接信息hplsql.conn.mysqlconn。
hplsql.conn.default
hive2conn
The default connection profile
hplsql.conn.hiveconn
org.apache.hive.jdbc.HiveDriver;jdbc:hive2://
HiveServer2 JDBC connection (embedded mode)
hplsql.conn.init.hiveconn
Statements for execute after connection to the database
hplsql.conn.convert.hiveconn
true
Convert SQL statements before execution
hplsql.conn.hive1conn
org.apache.hadoop.hive.jdbc.HiveDriver;jdbc:hive://
Hive embedded JDBC (not requiring HiveServer)
hplsql.conn.hive2conn
org.apache.hive.jdbc.HiveDriver;jdbc:hive2://hd4:10000;hive;hive
HiveServer2 JDBC connection
hplsql.conn.init.hive2conn
Statements for execute after connection to the database
hplsql.conn.convert.hive2conn
true
Convert SQL statements before execution
hplsql.conn.db2conn
com.ibm.db2.jcc.DB2Driver;jdbc:db2://localhost:50001/dbname;user;password
IBM DB2 connection
hplsql.conn.tdconn
com.teradata.jdbc.TeraDriver;jdbc:teradata://localhost/database=dbname,logmech=ldap;user;password
Teradata connection
hplsql.conn.mysqlconn
com.mysql.jdbc.Driver;jdbc:mysql://hd5/cpdds_etl;root;root123
MySQL connection
hplsql.dual.table
default.dual
Single row, single column table for internal operations
hplsql.insert.values
native
How to execute INSERT VALUES statement: native (default) and select
hplsql.onerror
exception
Error handling behavior: exception (default), seterror and stop
hplsql.temp.tables
native
Temporary tables: native (default) and managed
hplsql.temp.tables.schema
Schema for managed temporary tables
hplsql.temp.tables.location
/tmp/plhql
LOcation for managed temporary tables in HDFS
-
更新hive-hpsql-xxx.jar文件
jar uf hive-hplsql-3.1.0.3.0.1.0-187.jar hplsql-site.xml
二、hplsql测试
-
建立函数
--Create a function without parameters:
CREATE FUNCTION hello()
RETURNS STRING
BEGIN
RETURN 'Hello, world';
END;
-- Call the function
PRINT hello();
--Create a function with a parameter:
CREATE FUNCTION hello2(text STRING)
RETURNS STRING
BEGIN
RETURN 'Hello, ' || text || '!';
END;
-- Call the function
PRINT hello2('world');-
建立存储过程
CREATE PROCEDURE set_message(IN name STRING, OUT result STRING)
BEGIN
SET result = 'Hello, ' || name || '!';
END;
-- Now call the procedure and print the results
DECLARE str STRING;
CALL set_message('world', str);
PRINT str;
Result:
--
Hello, world!-
建立执行文件
INCLUDE set_message.sql
DECLARE str STRING;
CALL set_message('world', str);
PRINT str;
Result:
--
Hello, world!-
通过shell执行获得sql结果
MDATE=$(hplsql -e "NVL(MIN_PARTITION_DATE(sales, local_dt, code='A'), '1970-01-01')")
START=$(hplsql -e 'CURRENT_DATE - 1')-
执行sql文件
Hplsql –f script.sql
--可以通过 –main 参数指定要执行的部分，上面的命令会执行script.sql全部的内容
Hplsql –f script.sql –main funcation_name-
表映射，记录日志到mysql的日志表
先要拷贝mysql驱动到lib目录下。
cp /usr/share/java/mysql-connector-java.jar .
——logtomysql.sql内容
MAP OBJECT log TO cpdds_etl.hive_task_log AT mysqlconn;
DECLARE
start_time VARCHAR(20);
end_time VARCHAR(20);
return_code INT;
BEGIN
start_time = SYSDATE || '';
SELECT COUNT(1) FROM ods_test.tb_mail;
return_code = SQLCODE;
end_time = SYSDATE || '';
INSERT INTO log (`proc_name`,`start_time`,`end_time`,`return_code`)
VALUES(start_time,end_time,return_code);
EXCEPTION WHEN OTHERS THEN
return_code = SQLCODE;
end_time = SYSDATE || '';
INSERT INTO log (`proc_name`,`start_time`,`end_time`,`return_code`)
VALUES('testlog',start_time,end_time,return_code);
DBMS_OUTPUT.PUT_LINE('SQL execute error,return code : ' || return_code);
END——执行文件
hplsql -f logtomysql.sql
-
游标使用
FOR item IN (
SELECT v_mail_code, d_clct_date
FROM ods_test.tb_mail
ORDER BY v_mail_code)
LOOP
DBMS_OUTPUT.PUT_LINE('v_mail_code = ' || item.v_mail_code || ', d_clct_date = ' || item.d_clct_date);
END LOOP;三、参考文档
关于hplsql的更多用法可参考http://www.hplsql.org/