hive day05（调优）

目录

1.Function

  1.内置function（内置时间函数）

1.时间函数

 1.打印当前时间

2.打印时间戳

2.日期转换成秒值

3.秒值转换日期格式

4.返回时间值

1.to_date 返回年月日的值

2.year ：返回年的值

3.month ：返回月的值

4.day ：返回日的值

5.hour ：返回小时的值

6.minute ：返回分钟的值

5.计算时间

1.增加时间

1.add_months

2.date_add

2.减少时间

3. months_between

4.datediff 

5.last_day

7.日期格式

 2.算数相关函数

1.round  四舍五入取整

2.ceil 向上取整

3.floor 向下取整

4.rand  随机数

 3.字符串相关函数

1.upper  修改字母为大写

2.lower  修改字母为小写

3.length

4.trim  去除前后空格

5.补齐

1. lpad 左边补齐

2. rpad   右边补齐

6. regexp_replace  正则表达式

7.截取 substr

8.拼接  concat 

1.concat

2.concat_ws

9.分割   split

4.json数据处理

---

1.Function

  1.内置function（内置时间函数）

• String
  • upper
  • lower

• 1.时间函数

  •  1.打印当前时间

    • current_date 
    • 命令：select current_date;
      

  • 2.打印时间戳

    • current_timestamp 
    • 命令：select current_timestamp;
      

• 2.日期转换成秒值

  • select unix_timestamp('2022-12-28 00:00:00')
    
    有错，没找到
  • select unix_timestamp('2022-12-28','YYYY-MM-DD');
     

• 3.秒值转换日期格式

   

  • from_unixtime
  • SELECT from_unixtime(1,'yyyy-MM-dd HH:mm:ss');
    

• 4.返回时间值

  • 1.to_date 返回年月日的值

    • select to_date('2022-12-28 10:20:55');
      

  • 2.year ：返回年的值

  • 3.month ：返回月的值

  • 4.day ：返回日的值

    • select day('2022-12-28 10:20:55');
      

  • 5.hour ：返回小时的值

  • 6.minute ：返回分钟的值

• 5.计算时间

  • 1.增加时间

    • 1.add_months

      • 增加月份
      • select add_months('2022-12-28',1);
        

    • 2.date_add

      • 增加天数
      • select date_add('2022-12-28',1);
        

  • 2.减少时间

    • date_sub
    • select date_sub('2022-12-28',1);
      

  • 3. months_between

    • 求时间相差
    • select months_between('2022-12-28','2006-05-6');
      

  • 4.datediff 

    • 日期相差天数
    • select datediff("2022-12-30","2020-11-08");
      

  • 5.last_day

    • 求本月最后一天
    • select last_day('2022-12-03');
      

• 7.日期格式

  • date_format
  • select date_format('2022-12-28','y');
    

 2.算数相关函数

• 1.round  四舍五入取整

  • select round(3.14);
     

• 2.ceil 向上取整

  • select ceil(3.14);
    

• 3.floor 向下取整

  • select floor(3.94);
    

• 4.rand  随机数

  • 取随机数（0~1）
    • select rand();
      
  • 取随机数（1~10）
    • select round(10*rand());
      

 3.字符串相关函数

• 1.upper  修改字母为大写

  • SELECT upper('tinatian');
    

• 2.lower  修改字母为小写

  • select lower('TIANTIAN');
    

• 3.length

  • select length('dl2262');
    

• 4.trim  去除前后空格

  • select trim('  a  b cd   ');
    

• 5.补齐

  • 1. lpad 左边补齐

    • select lpad("dl2262",10,'_');
      

  • 2. rpad   右边补齐

    • select rpad("dl2262",10,'_');
      

• 6. regexp_replace  正则表达式

  • select regexp_replace('dl2262','[0-9]','num');
    
  • select regexp_replace('dl2262','[a-z]','num');
    
  • select regexp_replace('dl2262','[0-9||a-z]','num');
    

• 7.截取 substr

  • select substr('Facebook',5,4);
    

• 8.拼接  concat 

  • 1.concat

    • select concat('abc','def','456');
      

  • 2.concat_ws

    • 用 , 将1 2 3 进行拼接
    • select concat_ws(',',"1","2","3");
      

• 9.分割   split

  • 对 . 进行分割
  • select split("192.168.41.211",'\\.');
    

4.json数据处理

• 方式
  • 1.get_json_object
  • 2.json_tuple
  • 3.udf函数
• 案例：电影评分
  • 基本
    • 数据内容

      {"movie":"914","rate":"3","time":"978301968","userid":"1"}
      {"movie":"3408","rate":"4","time":"978300275","userid":"1"}
      {"movie":"2355","rate":"5","time":"978824291","userid":"1"}
      {"movie":"1197","rate":"3","time":"978302268","userid":"1"}
      {"movie":"1287","rate":"5","time":"978302039","userid":"1"}
      {"movie":"2804","rate":"5","time":"978300719","userid":"1"}
      {"movie":"594","rate":"4","time":"978302268","userid":"1"}
      {"movie":"919","rate":"4","time":"978301368","userid":"1"}
      {"movie":"595","rate":"5","time":"978824268","userid":"1"}
      {"movie":"938","rate":"4","time":"978301752","userid":"1"}
      {"movie":"2398","rate":"4","time":"978302281","userid":"1"}
      {"movie":"2918","rate":"4","time":"978302124","userid":"1"}
      {"movie":"1035","rate":"5","time":"978301753","userid":"1"}
      {"movie":"2791","rate":"4","time":"978302188","userid":"1"}
      {"movie":"2687","rate":"3","time":"978824268","userid":"1"}
      {"movie":"2018","rate":"4","time":"978301777","userid":"1"}
      {"movie":"3105","rate":"5","time":"978301713","userid":"1"}
      {"movie":"2797","rate":"4","time":"978302039","userid":"1"}
      

        {"movie":"1836","rate":"5","time":"978300172","userid":"1"}
        电影编号  评分  打分时间  打分用户id  

    • 创建hive表

      create table move_json(
      json string
      );

    • 插入数据

      load data local inpath "/home/hadoop/tmp/json/move_json" into table move_json;

        
  • 思路;需要json数据解析出来
    • 1.普通的json 
    • 2.嵌套json： 
      • 1.数组 ：多个元素+struct
      • 2.struct：kv
    • 3.不规则json  : udf 
  • 数据处理
    • 1.get_json_object

      •  一次只能解决一个字段

        select 
        get_json_object(json,'$.movie') as movie
        from move_json limit 10;

          
         

    • 2.json_tuple

      select  
      json_tuple(json,'movie','rate','time','userid') as (movie,rate,`time`,userid)
      from move_json limit 10;

        

       

    • 3.嵌套json
      • 数据

        {"store":{"fruit":[{"weight":8,"type":"apple"},{"weight":9,"type":"pear"}],"bicycle":{"price":19.95,"color":"red"}},"email":"amy@only_for_json_udf_test.net","owner":"amy"}

         路径：/home/hadoop/tmp/json
         [hadoop@bigdata14 json]$ vim json02.json

      • 建表

        create table json02(
        json string 
        );

      • 插入数据

        load data local inpath "/home/hadoop/tmp/json/json02.json" into table json02;
        

      • 解决简单字段

        select 
        json_tuple(json,'email','owner') as (email,owner)
        from json02;

          

      • 解决复杂字段

        select 
        json_tuple(json,'store','email','owner') as (store,email,owner)
        from json02;

          

         

      • 解决store字段 查看价格

        select 
        get_json_object(json,'$.store.bicycle.price') as movie
        from json02 limit 10;

          

         

    • 4..parse_url_tuple  => 解析url会用到 
      • 案例：https://cwiki.apache.org/course?a=b
                   协议   domain/hostname  path 参数
      • 解决

        select parse_url_tuple('https://cwiki.apache.org/course?a=b&c=d',"HOST","PATH","QUERY")