基于hadoop的气象数据可视化分析

目 录
摘 要 I
Abstract III
1绪论 1
1.1选题背景及意义 1
1.2研究现状及趋势 1
1.3研究主要内容 2
2相关技术简介 3
2.1开发工具 3
2.1.1 JDK1.7 3
2.1.2 eclipse luna 3
2.1.3 Hadoop 2.7.2 3
2.1.4 hbase 1.1.3 3
2.1.5 hive 1.2.1 3
2.1.6 zookeeper 3.4.8 4
2.1.7 mysql 5.5 4
2.1.8 swing 4
2.1.9 VMware Workstation 12 Pro 4
2.1.10其他辅助开发工具包 5
2.2开发环境搭建 6
2.2.1 三台Linux虚拟机的搭建 6
2.2.2 配置静态ip 7
2.2.3 SSH安装及免密码登陆 8
2.2.4修改hosts文件 9
2.2.5安装jdk 10
2.2.6 Hadoop安装与配置 11
2.2.7 Zookeeper安装与配置 13
2.2.8 Hbase的安装与配置 15
4.2.9 Hive的安装与配置 16
2.2.10 mysql的安装与配置 17
4.2.11 Hive与Hbase的整合 19
2.2.12 Hadoop-eclipse-plugin插件安装 19
2.2.13 WindowBuilder插件 19
3 需求分析 21
3.1可行性分析 21
3.1.1可行性研究介绍 21
3.1.2可行性分析 21
3.2需求分析 21
3.2.1需求分析作用 21
3.2.2功能需求 22
4 可视化设计 23
4.1界面设计 23
4.1.1前台界面设计 23
4.2模块划分 24
4.2.1用户注册与登录模块 24
4.2.2上传气象数据模块 24
4.2.3数据分析模块 25
4.2.4定时更新气象数据模块 25
4.2.5历史气象数据查询模块 25
4.2.6今日气象数据动态显示模块 27
4.2.7历史气象走势与极端气象查询模块 27
4.2.8数据库设计模块 27
4.3数据库设计E-R图 27
5 可视化界面 31
5.1界面实现 31
5.1.1主窗体的创建与总体布局 31
5.1.2 界面主要控件实现 32
5.1.3界面动态图形展示实现 33
5.2各功能模块实现 35
5.2.1用户注册与登录模块实现 35
5.2.2上传数据模块实现 37
5.2.3数据分析模块实现 38
5.2.4定时在线更新与分析模块实现 39
5.2.5历史气象数据查询模块实现 41
5.2.6今日气象数据动态显示模块实现 42
5.2.7历史气象走势模块实现 42
5.2.8数据库模块实现 42
6 可视化展示 45
6.1用户注册与登录功能展示 45
6.1.1用户注册 45
6.1.2用户登录 45
6.2上传数据功能展示 46
6.3数据分析与定时更新功能展示 47
6.3.1数据分析 47
6.3.2定时更新 48
6.4数据库内容展示 49
6.5今日气象展示 49
6.6历史气象展示 50
6.7历史气象走势展示 53
7 总结与展望 55
7.1总结 55
7.2未来展望 55
参考文献 56
致 谢 59
4 可视化设计
本系统大体上分为前台界面和后台Hadoop集群两部分。通过将系统的前台与后台划分开来使本来看起来复杂庞大的可视化设计简单清晰了很多，便于开发时根据开发的部分适时调整自己的定位与开发方法。
4.1界面设计
4.1.1前台界面设计
前台主要界面有：
1）用户注册界面
提供用户名，密码，二次输入密码等用户注册的基本功能。
2）用户登录界面
提供用户名，密码，管理员权限输入等功能。
3）管理员界面分为以下6个部分：
（1）历史气象数据查询界面
提供查询历史信息的基本界面，如选择城市，时间。也提供了气象数据的查询方式，如查询当天气象情况，当月情况，当年情况等。查询结果以图形的形式展现。
（2）今日气象数据查询界面
展现全国各省会今日的气象概况，展现形式参考气象预报，提供以中国地图为背景动态展示气象情况的效果
（3）上传气象数据界面
提供选择城市，选择文件，点击上传等功能。
（4）数据分析界面
提供选择时间范围，选择分析数据方式，如：客户端分析，Hive分析等功能。
（5）定时数据更新界面
提供定时获取数据与分析数据的详细日志进行界面展现的功能。
（6）历史气象变迁查询界面
提供选择指定城市，展现选定时间范围内气象情况的走势情况，展现形式为折线图。
4）普通用户界面分为以下三个部分：
（1）历史气象数据查询界面：同管理员界面
（2）今日气象数据查询界面：同管理员界面
（3）历史气象变迁及恶劣气象情况查询界面：同管理员界面
4.2模块划分
将系统模块化使我们的设计思路更加清晰，层次分明。在详细设计时可以通过将模块分别开发来提高开发效率，也便于单元测试，保证程序的稳定性。
本系统主要分为八大模块：
1）用户注册与登录模块
2）上传气象数据模块
3）数据分析模块
4）定时更新气象数据模块
5）历史气象数据查询模块
6）今日气象数据动态显示模块
7）历史气象变迁模块
8）数据库设计模块
4.2.1用户注册与登录模块
模块主要功能：点击注册按钮进入到用户注册界面，输入用户名和密码等信息即可完成注册，注册成功之后将用户信息保存到数据库中，其中加入不符合条件的用户信息的前台简单判定：
1）输入判定：
（1）两次输入密码不相同
（2）用户名为空
（3）密码为空
2）用户信息判定：
1）用户注册：用户名是否存在的判定
2）用户登录：用户名，密码是否正确
以上不符合条件的注册与登录信息皆会以弹窗的形式告知用户
4.2.2上传气象数据模块
模块主要功能：
1）从指定的气象数据网站上下载气象数据文件，点击选择城市与选择文件按钮从本地选择下载好的数据进行上传。
2）将待上传的数据文件进行信息提取（气象，风速，云量，气象总体概况等）。
3）将提取后的文件上传到Hbase。
4）将源数据文件上传到HDFS，以备文件里的其他信息用于以后的功能扩展。
4.2.3数据分析模块
模块主要功能：本文转载自http://www.biyezuopin.vip/onews.asp?id=14114数据分析模块提供选择时间范围使用客户端分析或Hive分析Hbase所保存气象数据的功能。分析详细如下：
（1）分析每天全国各地最高/最低气象，最高/最低风速所在的城市
（2）分析全国各城市每个月的气象情况，如：当月平均气象，最高气象，最高风速情况等。
4.2.4定时更新气象数据模块
模块主要功能如下：
1）利用一些气象数据网站提供的API在线获取信息，将这些信息保存到Hbase.每天定时更新，方便快捷，便于城市气象数据的实时获取与维护。
2）定时分析当天获取到的气象数据，分析内容参照数据分析模块，并将分析进度展现出来。

/*
SQLyog Ultimate v12.08 (64 bit)
MySQL - 5.7.23-log : Database - userdb
*********************************************************************
*/


/*!40101 SET NAMES utf8 */;

/*!40101 SET SQL_MODE=''*/;

/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
CREATE DATABASE /*!32312 IF NOT EXISTS*/`userdb` /*!40100 DEFAULT CHARACTER SET utf8 */;

USE `userdb`;

/*Table structure for table `temperature` */

DROP TABLE IF EXISTS `temperature`;

CREATE TABLE `temperature` (
  `Tem_Date` varchar(10) NOT NULL,
  `Tem_Max` double DEFAULT NULL,
  `Tem_Min` double DEFAULT NULL,
  `Tem_Avg` double DEFAULT NULL,
  PRIMARY KEY (`Tem_Date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

/*Data for the table `temperature` */

insert  into `temperature`(`Tem_Date`,`Tem_Max`,`Tem_Min`,`Tem_Avg`) values ('01/01',-58,-362,-169.7674418604651),('01/02',-85,-387,-185.56944444444446),('01/03',-51,-391,-169.47222222222223),('01/04',-47,-375,-172),('01/05',-89,-368,-186.76279069767443),('01/06',-66,-378,-202.5720930232558),('01/07',-77,-392,-188.73953488372092),('01/08',-47,-363,-156.99065420560748),('01/09',-7,-327,-123.66976744186047),('01/10',-15,-338,-141.7906976744186),('01/11',10,-314,-144.75),('01/12',-23,-308,-147.8186046511628),('01/13',-37,-325,-155.6435185185185),('01/14',-42,-374,-150.67906976744186),('01/15',-37,-375,-167.41860465116278),('01/16',-34,-392,-185.6898148148148),('01/17',-60,-398,-206.0186046511628),('01/18',-68,-357,-179.8139534883721),('01/19',-31,-343,-156.11162790697674),('01/20',-20,-336,-157.33644859813083),('01/21',-2,-342,-142.29302325581395),('01/22',15,-269,-122.38888888888889),('01/23',-44,-310,-150.85185185185185),('01/24',-49,-298,-137.49302325581397),('01/25',-39,-322,-136.52558139534884),('01/26',6,-343,-119.9766355140187),('01/27',19,-295,-116.53917050691244),('01/28',-32,-317,-141.36320754716982),('01/29',21,-324,-122.05581395348837),('01/30',-9,-379,-158.16589861751152),('01/31',-59,-393,-172.56221198156683),('02/01',-19,-399,-167.89814814814815),('02/02',20,-354,-155.94495412844037),('02/03',-2,-407,-186.77064220183487),('02/04',-106,-420,-214.9813953488372),('02/05',-116,-428,-231.19444444444446),('02/06',-67,-437,-235.16589861751152),('02/07',-141,-387,-227.09259259259258),('02/08',-97,-358,-197.75813953488372),('02/09',-76,-380,-185.89767441860465),('02/10',-71,-394,-183.94907407407408),('02/11',-64,-380,-190.72222222222223),('02/12',-45,-374,-168.85581395348837),('02/13',-53,-369,-155.90232558139536),('02/14',-31,-360,-145.86511627906978),('02/15',-36,-344,-135.94930875576037),('02/16',8,-280,-97.75813953488372),('02/17',44,-295,-80.45161290322581),('02/18',62,-268,-63.646511627906975),('02/19',41,-275,-74.22325581395349),('02/20',47,-254,-64.30232558139535),('02/21',72,-237,-50.43255813953488),('02/22',90,-234,-47.972093023255816),('02/23',109,-199,-19.07906976744186),('02/24',68,-274,-55.87906976744186),('02/25',41,-282,-73.75462962962963),('02/26',57,-245,-60.585253456221196),('02/27',89,-218,-41.53488372093023),('02/28',84,-254,-44.055813953488375),('03/01',88,-228,-34.265116279069765),('03/02',124,-204,-16.455813953488374),('03/03',108,-209,-22.85185185185185),('03/04',113,-199,-10.744186046511627),('03/05',107,-254,-35.6221198156682),('03/06',52,-243,-43.73953488372093),('03/07',93,-187,-22.888372093023257),('03/08',153,-204,12.586046511627908),('03/09',79,-209,-35.61574074074074),('03/10',92,-155,-7.268867924528302),('03/11',97,-207,-26.50232558139535),('03/12',56,-240,-56.31018518518518),('03/13',42,-225,-48.92631578947368),('03/14',61,-240,-43.42325581395349),('03/15',43,-179,-42.300925925925924),('03/16',73,-184,-22.057894736842105),('03/17',156,-135,28.416666666666668),('03/18',182,-116,36.976851851851855),('03/19',181,-209,10.221198156682028),('03/20',78,-234,-38.273148148148145),('03/21',70,-236,-64.31506849315069),('03/22',105,-258,-54.252252252252255),('03/23',121,-175,-29.65765765765766),('03/24',135,-139,-0.8687782805429864),('03/25',122,-218,-17.368663594470046),('03/26',106,-270,-37.0462962962963),('03/27',112,-243,-35.81481481481482),('03/28',135,-247,-33.61395348837209),('03/29',161,-223,-22.046511627906977),('03/30',196,-214,-18.46330275229358),('03/31',163,-160,-19.28110599078341),('04/01',181,-220,-11.175115207373272),('04/02',189,-147,17.792626728110598),('04/03',212,-92,50.14418604651163),('04/04',210,-119,35.2),('04/05',205,-112,24.409302325581397),('04/06',156,-115,26.02777777777778),('04/07',110,-102,15.790697674418604),('04/08',116,-157,22.232558139534884),('04/09',145,-152,37.55348837209302),('04/10',167,-136,44.78604651162791),('04/11',175,-110,66.69302325581396),('04/12',200,-139,87.05581395348837),('04/13',202,-145,43.02325581395349),('04/14',193,-109,54.70048309178744),('04/15',234,-20,117.31162790697674),('04/16',276,-110,119.69907407407408),('04/17',245,-147,60.95813953488372),('04/18',173,-162,24.11627906976744),('04/19',208,-70,59),('04/20',181,-130,65.23720930232558),('04/21',180,-129,64.61395348837209),('04/22',261,-64,109.1767441860465),('04/23',251,-55,89.55760368663594),('04/24',190,-152,56.717592592592595),('04/25',162,-142,62.60648148148148),('04/26',183,-50,88.46046511627907),('04/27',229,-42,124.24651162790698),('04/28',228,-72,124.32093023255814),('04/29',225,18,128.84722222222223),('04/30',223,-77,112.23720930232558),('05/01',231,-55,92.32558139534883),('05/02',269,-19,125.94907407407408),('05/03',285,-7,155.7953488372093),('05/04',295,-104,108.25345622119816),('05/05',207,-80,62.39252336448598),('05/06',193,-22,101.39814814814815),('05/07',247,14,135.08333333333334),('05/08',246,-75,116.73953488372094),('05/09',272,0,139.36574074074073),('05/10',319,-29,161.50232558139535),('05/11',306,-12,155.10185185185185),('05/12',282,10,162.89252336448598),('05/13',266,7,118.07870370370371),('05/14',242,-32,113.36574074074075),('05/15',278,-14,155.25116279069766),('05/16',260,0,130.6435185185185),('05/17',259,2,143.65581395348838),('05/18',268,-5,123.56744186046511),('05/19',226,25,120.44651162790697),('05/20',247,45,107.09767441860465),('05/21',295,-57,110.2046511627907),('05/22',286,12,120.97183098591549),('05/23',350,57,176.51322751322752),('05/24',374,20,213.5305164319249),('05/25',328,-9,173.86511627906978),('05/26',300,-25,140.0139534883721),('05/27',218,-39,127.14953271028037),('05/28',229,-20,125.51869158878505),('05/29',269,15,148.906103286385),('05/30',285,-15,137.87906976744185),('05/31',252,-12,140.26699029126215),('06/01',266,-20,154.3860465116279),('06/02',276,6,162.76279069767443),('06/03',275,30,174.34418604651162),('06/04',277,21,187.06976744186048),('06/05',275,2,176.04651162790697),('06/06',307,37,185.53488372093022),('06/07',303,50,180.05581395348838),('06/08',299,17,189.33953488372094),('06/09',282,50,186.41203703703704),('06/10',306,75,196.99537037037038),('06/11',317,57,197.7813953488372),('06/12',302,70,200.3953488372093),('06/13',270,72,170.12037037037038),('06/14',258,1,141.3778801843318),('06/15',270,15,153.9181818181818),('06/16',282,26,171.7590909090909),('06/17',280,26,175.03240740740742),('06/18',325,85,196.89814814814815),('06/19',342,30,195.25925925925927),('06/20',315,40,176.70697674418605),('06/21',315,70,202.31627906976743),('06/22',331,107,218.27649769585253),('06/23',363,132,235.53917050691246),('06/24',340,96,235.9212962962963),('06/25',323,70,215.91627906976746),('06/26',301,75,197.6712962962963),('06/27',315,16,171.61111111111111),('06/28',328,12,163.25581395348837),('06/29',340,46,172.03255813953487),('06/30',327,35,184.8119266055046),('07/01',296,85,197.12442396313364),('07/02',305,60,191.65898617511522),('07/03',316,80,197.82027649769586),('07/04',320,20,201.0091743119266),('07/05',300,32,181.4884792626728),('07/06',311,67,202.3440366972477),('07/07',340,62,226.98617511520737),('07/08',327,107,241.14883720930231),('07/09',342,107,244.17592592592592),('07/10',326,102,227.21495327102804),('07/11',306,96,214.03773584905662),('07/12',313,105,215.04651162790697),('07/13',315,97,218.62325581395348),('07/14',328,97,218.3674418604651),('07/15',335,87,225.3860465116279),('07/16',316,90,217.353488372093),('07/17',331,110,224.46511627906978),('07/18',335,106,233.6),('07/19',326,168,243.1813953488372),('07/20',329,142,243.28372093023256),('07/21',321,120,230.01869158878506),('07/22',330,106,221.58139534883722),('07/23',307,122,212.53023255813955),('07/24',299,102,189.70697674418605),('07/25',280,65,186.75348837209302),('07/26',327,57,198.32407407407408),('07/27',302,128,221.2),('07/28',327,101,224.5906976744186),('07/29',322,120,237.2232558139535),('07/30',327,112,218.07906976744187),('07/31',316,100,220.2073732718894),('08/01',322,92,223.66511627906976),('08/02',337,115,235.0139534883721),('08/03',347,96,238.7314814814815),('08/04',370,86,247.80751173708921),('08/05',327,41,228.87962962962962),('08/06',310,75,202.6),('08/07',307,76,194.90277777777777),('08/08',325,82,190.0648148148148),('08/09',347,86,182.36915887850466),('08/10',257,100,173.88679245283018),('08/11',261,57,179.1860465116279),('08/12',295,75,194.12962962962962),('08/13',332,66,208.6139534883721),('08/14',345,37,209.37383177570092),('08/15',318,72,203.90232558139536),('08/16',267,86,180.67906976744186),('08/17',267,65,175.26728110599078),('08/18',246,37,168.44131455399062),('08/19',265,67,180.3287037037037),('08/20',282,115,183.52558139534884),('08/21',263,106,174.96744186046513),('08/22',297,90,180.58256880733944),('08/23',250,82,172.8479262672811),('08/24',254,75,174.86976744186046),('08/25',266,72,180.28240740740742),('08/26',287,82,185.53953488372093),('08/27',315,32,179.93119266055047),('08/28',332,45,175.28310502283105),('08/29',347,36,177.52534562211983),('08/30',347,35,186.38947368421051),('08/31',296,62,180.5462962962963),('09/01',302,-4,171.9953488372093),('09/02',280,55,177.3860465116279),('09/03',317,90,193.5860465116279),('09/04',328,106,212.08837209302325),('09/05',312,86,204.2325581395349),('09/06',322,87,207.74418604651163),('09/07',302,84,206.1860465116279),('09/08',279,17,172.7953488372093),('09/09',254,0,138.07943925233644),('09/10',225,-25,122.87441860465117),('09/11',282,20,145.5185185185185),('09/12',285,51,165.35813953488372),('09/13',227,-20,139.85340314136127),('09/14',247,-7,144.4),('09/15',256,-69,119.94418604651163),('09/16',213,-16,109.8232558139535),('09/17',222,-37,93.73148148148148),('09/18',217,-57,82.77251184834124),('09/19',257,-6,105.20370370370371),('09/20',267,-77,113.09302325581395),('09/21',287,-45,111.24537037037037),('09/22',296,-20,151.56744186046512),('09/23',311,-80,143.8186046511628),('09/24',307,-62,116.37209302325581),('09/25',300,-37,128.86976744186046),('09/26',300,6,152.62325581395348),('09/27',305,41,189.34562211981566),('09/28',276,-72,158.5767441860465),('09/29',240,-40,106.46261682242991),('09/30',290,-14,127.04608294930875),('10/01',298,-18,168.26388888888889),('10/02',284,-77,114.55440414507773),('10/03',150,-130,48.93023255813954),('10/04',165,-117,31.049180327868854),('10/05',190,-92,67.71627906976744),('10/06',216,-35,104.00925925925925),('10/07',236,-79,86.10232558139535),('10/08',237,-37,69.86976744186046),('10/09',229,-69,106.5925925925926),('10/10',221,-109,54.398148148148145),('10/11',210,-193,16.083333333333332),('10/12',189,-197,-1.6775700934579438),('10/13',189,-194,-0.33796296296296297),('10/14',197,-125,19.875),('10/15',192,-125,46.254629629629626),('10/16',169,-94,46.138888888888886),('10/17',180,-92,63.566820276497694),('10/18',181,-84,65.29439252336448),('10/19',189,-70,72.48387096774194),('10/20',192,-64,73.66197183098592),('10/21',163,-72,42.97674418604651),('10/22',191,-115,36.14883720930233),('10/23',192,-111,35.009049773755656),('10/24',184,-130,27.955156950672645),('10/25',156,-168,-2.939814814814815),('10/26',84,-150,-3.0324074074074074),('10/27',119,-96,12.89237668161435),('10/28',140,-134,26.219730941704036),('10/29',116,-152,9.991031390134529),('10/30',146,-122,41.300448430493276),('10/31',169,-144,32.860986547085204),('11/01',172,-195,15.702702702702704),('11/02',186,-237,-17.981900452488688),('11/03',102,-212,-24.800904977375566),('11/04',169,-125,7.745454545454545),('11/05',129,-228,-30.569506726457398),('11/06',30,-208,-61.593607305936075),('11/07',95,-238,-53.16279069767442),('11/08',73,-207,-53.13963963963964),('11/09',89,-189,-25.82511210762332),('11/10',101,-142,4.7207207207207205),('11/11',91,-243,-34.318385650224215),('11/12',36,-274,-72.79372197309416),('11/13',10,-326,-123.1447963800905),('11/14',26,-318,-143.34977578475338),('11/15',1,-319,-128.7072072072072),('11/16',-29,-290,-129.37387387387386),('11/17',-7,-273,-107.07239819004525),('11/18',-32,-202,-122.98206278026906),('11/19',18,-313,-142.76576576576576),('11/20',32,-311,-150.26696832579185),('11/21',47,-302,-111.1891891891892),('11/22',98,-254,-51.794520547945204),('11/23',83,-343,-104.60309278350516),('11/24',-35,-359,-151.07692307692307),('11/25',-40,-344,-130.46445497630333),('11/26',2,-344,-128.43093922651934),('11/27',-45,-345,-160.53365384615384),('11/28',-62,-342,-164.85096153846155),('11/29',-31,-333,-146.40865384615384),('11/30',-25,-294,-129.26923076923077),('12/01',-50,-313,-144.9047619047619),('12/02',-49,-307,-164.2095238095238),('12/03',-70,-323,-202.85849056603774),('12/04',-27,-387,-211.13888888888889),('12/05',7,-375,-191.7311320754717),('12/06',8,-350,-162.04651162790697),('12/07',-38,-372,-169.49302325581397),('12/08',-5,-374,-158.70232558139534),('12/09',-1,-334,-145.3440366972477),('12/10',5,-349,-153.97663551401868),('12/11',-16,-349,-174.69767441860466),('12/12',-11,-341,-144.4279069767442),('12/13',2,-323,-143.3813953488372),('12/14',-23,-334,-130.1627906976744),('12/15',7,-244,-115.40284360189574),('12/16',20,-354,-101.72093023255815),('12/17',-2,-379,-162.55348837209303),('12/18',-75,-392,-200),('12/19',-84,-393,-210.43867924528303),('12/20',-70,-399,-199.85581395348837),('12/21',-32,-405,-210.88837209302326),('12/22',-54,-408,-222.25233644859813),('12/23',-74,-405,-221.74883720930234),('12/24',-85,-399,-217.6602870813397),('12/25',-97,-402,-223.44600938967136),('12/26',-70,-400,-239.07246376811594),('12/27',-60,-399,-237.83091787439614),('12/28',-127,-410,-224.2378640776699),('12/29',-60,-417,-217.79),('12/30',-35,-415,-243.626213592233),('12/31',-35,-398,-227.03846153846155);

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;