陌陌聊天数据分析 (一)

陌陌聊天数据分析(一)

目标

  • 基于Hadoop和Hive实现聊天数据统计分析,构建聊天数据分析报表

需求

  • 统计今日总消息量
  • 统计今日每小时消息量,发送和接收用户数量
  • 统计今日各地区发送消息数据量
  • 统计今日发送消息和接收消息用户数
  • 统计今日发送消息最多的用户前几名
  • 统计今日接收消息最多的用户前几名
  • 统计发送人手机型号分布情况
  • 统计发送人设备系统分布情况

数据来源

  • 聊天业务系统导出2021/11/01一天24小时用户聊天数据,以TSV文本形式存储在文件中
    • 数据大小:两个文件共14万条数据
    • 列分隔符:\t

数据集及所需文件

  • 链接:https://pan.baidu.com/s/1ToTanDrFRhAVsFTb2uclFg
    提取码:rkun

基于Hive数仓实现需求开发

⚽建库建表 加载数据

  • 建库建表
--创建数据库
create database db_msg;
--切换数据库
use db_msg;

--建表
create table db_msg.tb_msg_source(
  msg_time             string  comment "消息发送时间"
  , sender_name        string  comment "发送人昵称"
  , sender_account     string  comment "发送人账号"
  , sender_sex         string  comment "发送人性别"
  , sender_ip          string  comment "发送人IP"
  , sender_os          string  comment "发送人操作系统"
  , sender_phonetype   string  comment "发送人手机型号"
  , sender_network     string  comment "发送人网络类型"
  , sender_gps         string  comment "发送人GPS定位"
  , receiver_name      string  comment "接收人昵称"
  , receiver_ip        string  comment "接收人IP"
  , receiver_account   string  comment "接收人账号"
  , receiver_os        string  comment "接收人操作系统"
  , receiver_phonetype string  comment "接收人手机型号"
  , receiver_network   string  comment "接收人网络类型"
  , receiver_gps       string  comment "接收人GPS定位"
  , receiver_sex       string  comment "接收人性别"
  , msg_type           string  comment "消息类型"
  , distance           string  comment "双方距离"
  , message            string  comment "消息内容"
)
--指定分隔符为制表符
row format delimited fields terminated by '\t';

  • 加载数据
#上传数据到node1服务器本地文件系统(HS2服务所在机器)
[root@node1 hivedata]# pwd
/root/hivedata
[root@node1 hivedata]# ll
total 54104
-rw-r--r-- 1 root root 28237023 Jun 13 20:24 data1.tsv
-rw-r--r-- 1 root root 27161148 Jun 13 20:24 data2.tsv

--加载数据入表
load data local inpath '/root/hivedata/data1.tsv' into table db_msg.tb_msg_source;
load data local inpath '/root/hivedata/data2.tsv' into table db_msg.tb_msg_source;
  • 查询表,查看数据是否导入成功
--查询表
select * from tb_msg_source limit 5;

在这里插入图片描述

⚾ETL数据清洗

数据问题

  • 当前数据,一些数据字段为空,不是合法数据。
  • 需求需要统计每天每个小时消息量,但数据中没有天和小时字段,只有整体时间字段,不好处理。
  • 需求中,GPS对经纬度在同一字段,不好处理。

ETL需求

  • 对字段为空的不合法数据进行过滤
    • where过滤
  • 通过时间字段构建天和小时字段
    • substr函数
  • 从GPS经纬度提取经纬度
    • split函数
  • 将ETL以后的结果保存到一张新的Hive表中
    • create table …as select…
create table db_msg.tb_msg_etl as
select *,
       substr(msg_time, 0, 10)   as dayinfo,    --获取天
       substr(msg_time, 12, 2)   as hourinfo,   --获取小时
       split(sender_gps, ",")[0] as sender_lng, --经度
       split(sender_gps, ",")[1] as sender_lat  --纬度
from db_msg.tb_msg_source
--过滤字段为空数据
where length(sender_gps) > 0;
select
    msg_time,dayinfo,hourinfo,sender_gps,sender_lng,sender_lat
from db_msg.tb_msg_etl
limit 5;
--查询数据

陌陌聊天数据分析 (一)_第1张图片

需求指标SQL

  • 解读需求
  • 确定待查询数据表 from
  • 分析维度 group by
  • 找出计算指标 聚合
  • 细节 过滤 排序
  1. 统计今日消息总量

    --需求:统计今日总消息量
    create table if not exists tb_rs_total_msg_cnt
    comment "今日消息总量"
    as
    select
      dayinfo,
      count(*) as total_msg_cnt
    from db_msg.tb_msg_etl
    group by dayinfo;
    
    --查询
    select * from tb_rs_total_msg_cnt ;
    
    +------------------------------+------------------------------------+
    | tb_rs_total_msg_cnt.dayinfo  | tb_rs_total_msg_cnt.total_msg_cnt  |
    +------------------------------+------------------------------------+
    | 2021-11-01                   | 139062                             |
    +------------------------------+------------------------------------+
    
    
  2. 统计今日每小时消息量,发送/接收用户数

    create table tb_rs_hour_msg_cnt
    comment "每小时消息量趋势"
    as
    select
        dayinfo,
        hourinfo,
        count(*) as total_msg_cnt,
        count(distinct sender_account) as sender_usr_cnt,
        count(distinct receiver_account)as receiver_usr_cnt
    from db_msg.tb_msg_etl
    group by dayinfo,hourinfo;
    
    select * from tb_rs_hour_msg_cnt limit 5;
    
    +-----------------------------+------------------------------+-----------------------------------+------------------------------------+--------------------------------------+
    | tb_rs_hour_msg_cnt.dayinfo  | tb_rs_hour_msg_cnt.hourinfo  | tb_rs_hour_msg_cnt.total_msg_cnt  | tb_rs_hour_msg_cnt.sender_usr_cnt  | tb_rs_hour_msg_cnt.receiver_usr_cnt  |
    +-----------------------------+------------------------------+-----------------------------------+------------------------------------+--------------------------------------+
    | 2021-11-01                  | 00                           | 4349                              | 3520                               | 3558                                 |
    | 2021-11-01                  | 01                           | 2892                              | 2524                               | 2537                                 |
    | 2021-11-01                  | 02                           | 882                               | 842                                | 838                                  |
    | 2021-11-01                  | 03                           | 471                               | 463                                | 460                                  |
    | 2021-11-01                  | 04                           | 206                               | 202                                | 205                                  |
    +-----------------------------+------------------------------+-----------------------------------+------------------------------------+--------------------------------------+
    
    
  3. 统计今日各地区发送消息数据量

    create table tb_rs_loc_cnt
    comment "今日各地区发送总消息量"
    as select
      dayinfo,
      sender_gps,
      cast(sender_lng as double) as longitude,
      cast(sender_lat as double) as latitude,
      count(*) as total_msg_cnt
    from tb_msg_etl
    group by dayinfo, sender_gps, sender_lng,sender_lat;
    
    
    select * from tb_rs_loc_cnt limit 5;
    
    
    +------------------------+---------------------------+--------------------------+-------------------------+------------------------------+
    | tb_rs_loc_cnt.dayinfo  | tb_rs_loc_cnt.sender_gps  | tb_rs_loc_cnt.longitude  | tb_rs_loc_cnt.latitude  | tb_rs_loc_cnt.total_msg_cnt  |
    +------------------------+---------------------------+--------------------------+-------------------------+------------------------------+
    | 2021-11-01             | 100.297355,24.206808      | 100.297355               | 24.206808               | 1397                         |
    | 2021-11-01             | 100.591712,24.004148      | 100.591712               | 24.004148               | 1406                         |
    | 2021-11-01             | 101.62196,36.782187       | 101.62196                | 36.782187               | 1439                         |
    | 2021-11-01             | 102.357852,23.801165      | 102.357852               | 23.801165               | 1399                         |
    | 2021-11-01             | 102.357852,25.682909      | 102.357852               | 25.682909               | 1431                         |
    +------------------------+---------------------------+--------------------------+-------------------------+------------------------------+
    
    
  4. 统计今日发送消息和接受消息用户数

    create table tb_rs_usr_cnt
    comment "今日发送消息人数、接受消息人数"
    as
    select
      dayinfo,
      count(distinct sender_account) as sender_usr_cnt,
      count(distinct receiver_account) as receiver_usr_cnt
    from db_msg.tb_msg_etl
    group by dayinfo;
    
    select * from tb_rs_usr_cnt ;
    
    +------------------------+-------------------------------+---------------------------------+
    | tb_rs_usr_cnt.dayinfo  | tb_rs_usr_cnt.sender_usr_cnt  | tb_rs_usr_cnt.receiver_usr_cnt  |
    +------------------------+-------------------------------+---------------------------------+
    | 2021-11-01             | 10008                         | 10005                           |
    +------------------------+-------------------------------+---------------------------------+
    
    
  5. 统计今日发送消息最多的Top10用户

    create table tb_rs_susr_top10
    comment "发送消息条数最多的Top10用户"
    as
    select
      dayinfo,
      sender_name as username,
      count(*) as sender_msg_cnt
    from db_msg.tb_msg_etl
    group by dayinfo,sender_name
    order by sender_msg_cnt desc
    limit 10;
    
    select * from tb_rs_susr_top10;
    
    +---------------------------+----------------------------+----------------------------------+
    | tb_rs_susr_top10.dayinfo  | tb_rs_susr_top10.username  | tb_rs_susr_top10.sender_msg_cnt  |
    +---------------------------+----------------------------+----------------------------------+
    | 2021-11-01                | 茹鸿晖                        | 1466                             |
    | 2021-11-01                | 卢高达                        | 1464                             |
    | 2021-11-01                | 犁彭祖                        | 1460                             |
    | 2021-11-01                | 沐范                         | 1459                             |
    | 2021-11-01                | 夫潍                         | 1452                             |
    | 2021-11-01                | 烟心思                        | 1449                             |
    | 2021-11-01                | 称子瑜                        | 1447                             |
    | 2021-11-01                | 麻宏放                        | 1442                             |
    | 2021-11-01                | 邴时                         | 1439                             |
    | 2021-11-01                | 养昆颉                        | 1431                             |
    +---------------------------+----------------------------+----------------------------------+
    
    
  6. 统计今日接受消息最多的Top10用户

    create table tb_rs_rusr_top10
    comment "接受消息条数最多的Top10用户"
    as
    select
      dayinfo,
      receiver_name as username,
      count(*) as receiver_msg_cnt
    from db_msg.tb_msg_etl
    group by dayinfo,receiver_name
    order by receiver_msg_cnt desc
    limit 10;
    
    select * from tb_rs_rusr_top10 limit 3;
    
    +---------------------------+----------------------------+------------------------------------+
    | tb_rs_rusr_top10.dayinfo  | tb_rs_rusr_top10.username  | tb_rs_rusr_top10.receiver_msg_cnt  |
    +---------------------------+----------------------------+------------------------------------+
    | 2021-11-01                | 畅雅柏                        | 1539                               |
    | 2021-11-01                | 春纯                         | 1491                               |
    | 2021-11-01                | 邝琨瑶                        | 1469                               |
    +---------------------------+----------------------------+------------------------------------+
    
    
  7. 统计发送人手机型号分布情况

    create table if not exists tb_rs_sender_phone
    comment "发送人的手机型号分布"
    as
    select
      dayinfo,
      sender_phonetype,
      count(distinct sender_account) as cnt
    from tb_msg_etl
    group by dayinfo,sender_phonetype;
    
    select * from tb_rs_sender_phone limit 3;
    
    +-----------------------------+--------------------------------------+-------------------------+
    | tb_rs_sender_phone.dayinfo  | tb_rs_sender_phone.sender_phonetype  | tb_rs_sender_phone.cnt  |
    +-----------------------------+--------------------------------------+-------------------------+
    | 2021-11-01                  | Apple iPhone 10                      | 6749                    |
    | 2021-11-01                  | Apple iPhone 11                      | 3441                    |
    | 2021-11-01                  | Apple iPhone 7                       | 2424                    |
    +-----------------------------+--------------------------------------+-------------------------+
    
    
  8. 统计发送人设备操作系统分布情况

    create table tb_rs_sender_os
    comment "发送人的OS分布"
    as
    select
      dayinfo,
      sender_os,
      count(distinct sender_account) as cnt
    from tb_msg_etl
    group by dayinfo,sender_os;
    
    select * from tb_rs_sender_os;
    
    +--------------------------+----------------------------+----------------------+
    | tb_rs_sender_os.dayinfo  | tb_rs_sender_os.sender_os  | tb_rs_sender_os.cnt  |
    +--------------------------+----------------------------+----------------------+
    | 2021-11-01               | Android 5.1                | 5750                 |
    | 2021-11-01               | Android 6                  | 8514                 |
    | 2021-11-01               | Android 6.0                | 9398                 |
    | 2021-11-01               | Android 7.0                | 9181                 |
    | 2021-11-01               | Android 8.0                | 8594                 |
    | 2021-11-01               | IOS 10.0                   | 1289                 |
    | 2021-11-01               | IOS 12.0                   | 8102                 |
    | 2021-11-01               | IOS 9.0                    | 8760                 |
    +--------------------------+----------------------------+----------------------+
    
    

️FineBI实现可视化报表

官网

https://www.finebi.com/

配置数据源及数据准备

官方文档

https://help.fanruan.com/finebi/doc-view-301.html

  • 使用FineBI连接Hive,读取Hive数据表,需要在FineBI中添加Hive驱动jar包

  • 将Hive驱动jar包放入FineBI的lib目录下

  • 找到提供文件的HiveConnectDrive

陌陌聊天数据分析 (一)_第2张图片

  • 放入安装路径下的 webapps\webroot\WEB-INF\lib

陌陌聊天数据分析 (一)_第3张图片

插件安装

  • 我们自己Hive驱动包会与FineBI自带驱动包冲突,导致FineBI无法识别我们自己的驱动
  • 安装FineBI官方提供驱动包隔离插件

隔离插件:fr-plugin-hive-driver-loader-3.0.zip

  • 安装插件

    陌陌聊天数据分析 (一)_第4张图片

  • 重启FineBI

你可能感兴趣的:(#,大数据,数据分析,hadoop,hive)