CREATE TEMPORARY FUNCTION function_name AS class_name;
DROP TEMPORARY FUNCTION [IF EXISTS] function_name;
在hive0.13之后可以将功能函数注册到元数据中,无需每次创建session重新创建临时功能函数。
CREATE FUNCTION [db_name.]function_name AS class_name
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
DROP FUNCTION [IF EXISTS] function_name;
RELOAD FUNCTION;
SHOW FUNCTIONS "a.*";
import com.google.common.hash.Hashing;
import org.apache.hadoop.hive.ql.exec.UDF;
public class Text2md5 extends UDF {
public String evaluate(final String s) {
if (s == null) { return null; }
return Hashing.md5().hashUnencodedChars(s.trim()).toString();
}
}
create temporary function tmp_md5 as 'com.example.hive.udf.Text2md5';
create function test_hive.tmp_md5 as 'com.example.hive.udf.Text2md5';
指定jar路径
hive>add jar /home/acm/jar/demo-1.0.jar;
查看jar
hive>list jars
创建函数
CREATE TEMPORARY FUNCTION test_hive.tmp_md5 as 'com.example.hive.udf.Text2md5';
hive local模式可以直接在本地文件使用。
hive no-local 模式时,先将jar上传至HDFS后再使用。
CREATE TEMPORARY FUNCTION test_hive.test_md5 AS 'com.demo.hive.Text2md5' USING JAR 'hdfs:///tmp/test2/demo-1.0.jar';
(TEMPORARY为临时函数,关闭失去hive连接后临时函数自动去除;不带TEMPORARY则为注册永久函数)
SHOW FUNCTIONS 'tes.*';
USE test_hive;
select id,test_hive.test_md5(name) from test limit 5;
问题:创建udf
CREATE FUNCTION my_testmd5 AS ‘com.demo.hive.Text2md5’ USING JAR ‘/home/acm/jar/demo-1.0.jar’;
报错:FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask. Hive warehouse is non-local, but /home/acm/jar/demo-1.0.jar specifies file on local filesystem. Resources on non-local warehouse should specify a non-local scheme/path
先将jar上传到hdfs
hadoop fs -copyFromLocal 本地jar 目标jar
问题:udf无法使用
Caused by: java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashUnencodedChars(Ljava/lang/CharSequence;)Lcom/google/common/hash/HashCode;
at com.demo.hive.Text2md5.evaluate(Text2md5.java:18)
… 23 more
问题:执行权限不足
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
解决:换hive用户
问题:FAILED: SemanticException Temporary function cannot be created with a qualified name.
CREATE TEMPORARY FUNCTION test_hive.t_md52 AS ‘com.demo.hive.Text2md5’ USING JAR ‘hdfs:///tmp/test-h/demo-1.0.jar’;
解决:CREATE TEMPORARY FUNCTION test_hive.t_md52 AS ‘com.demo.hive.Text2md5’ USING JAR ‘hdfs:///tmp/test-h/demo-1.0.jar’;
去掉 test_hive.
问题:/home/acm/jar/demo-1.0.jar does not exist
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. /home/acm/jar/demo-1.0.jar does not exist
解决:上穿到hdfs上