Hadoop学习-基础篇

Hadoop 大数据平台与架构

功能与优势

Hadoop是大数据存储与分析的架构，是分布式存储和分布式计算平台

两个核心组成

• HDFS：分布式文件系统，用于存储海量的数据
• MapReduce：并行处理框架，实现任务的分解和调度

应用

可用于搭建数据仓库，分析统计数据

生态

• HIVE：SQL语句形式，转换为Hadoop任务去执行
• HBASE：存储结构化数据的分布式数据库
• zookeeper：服务注册、治理

HDFS概念

• 块（Block）：文件被切分成同大小的数据块，默认为64MB，是文件存储的逻辑单元
• NameNode：数据管理节点，包括文件与数据块映射表，数据块与数据节点映射表
• DataNode：工作节点，存放真正的数据块

HDFS数据管理与容错

每个数据块有3个副本，分布在两个机架上，如上图数据块

HDFS特点

• 数据冗余，硬件容错
• 流式数据访问
• 易于存储大文件

HDFS使用

配置好hadoop服务并启动后，执行 jps 命令，如看到这些服务即启动成功

[root@liyan ~]# jps
14258 Jps
5108 TaskTracker
4887 SecondaryNameNode
4634 NameNode
4970 JobTracker
4765 DataNode

执行 hadoop dfsadmin -report 可以看到整个hadoop的使用情况

[root@liyan ~]# hadoop dfsadmin -report
Configured Capacity: 42139451392 (39.25 GB)
Present Capacity: 34523619328 (32.15 GB)
DFS Remaining: 34523467776 (32.15 GB)
DFS Used: 151552 (148 KB)
DFS Used%: 0%
Under replicated blocks: 8
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 42139451392 (39.25 GB)
DFS Used: 151552 (148 KB)
Non DFS Used: 7615832064 (7.09 GB)
DFS Remaining: 34523467776(32.15 GB)
DFS Used%: 0%
DFS Remaining%: 81.93%
Last contact: Sun Jan 19 18:55:45 CST 2020

HDFS的三个基本操作命令

• 从本地linux上传文件到HDFS

  hadoop fs -put 文件  文件夹/
  

• 从HDFS查看文件：

  hadoop fs -cat 文件夹/文件
  

• 从HDFS下载文件到本地：

  hadoop fs -get 文件夹/文件  新文件
  

MapReduce原理

分而治之，将大任务拆分成多个小任务（map），并行执行，合并结果（reduce）

概念

• Job&Task：一个Job被拆分为多个Task
• JobTracker：作业调度、分配任务、监控任务执行进度、监控TaskTracker状态

体系结构

MapReduce容错机制

• 重复执行
• 推测执行：如果之前的TaskTracker计算很慢，则推测可能出现问题，新建一台TaskTracker同时与之计算，谁先执行完成取谁结果

Hadoop应用实践

按照官网推荐的WordCount来实践一下，输入文本文件，统计里面单词的出现次数

1. 先将wordcount.jar文件上传到Linux服务器本地

    scp wordcount.jar root@ip:/opt/examples
   

2.创建input文件夹，里面存放输入的文本文件
3.在HDFS里创建input_wordcount文件夹作为输入文件夹

    hadoop fs -mkdir input_wordcount/

4.将输入文件上传到HDFS的input_wordcount文件夹，使用 hadoop fs -ls 命令查看wordcount文件夹下是否已有文件

    [root@liyan examples]# hadoop fs -put input/* input_wordcount
    [root@liyan examples]# hadoop fs -ls input_wordcount/
    Found 2 items
    -rw-r--r--   3 root supergroup        147 2020-01-19 19:55 /user/root/input_wordcount/file1.text
    -rw-r--r--   3 root supergroup        153 2020-01-19 19:55 /user/root/input_wordcount/file2.text

5.执行jar文件，hadoop jar jar文件 main函数名称 输入文件夹 输出文件夹

    [root@liyan examples]# hadoop jar wordcount.jar WordCount input_wordcount output_wordcount
    20/01/19 20:35:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    20/01/19 20:35:13 INFO input.FileInputFormat: Total input paths to process : 2
    20/01/19 20:35:13 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    20/01/19 20:35:13 WARN snappy.LoadSnappy: Snappy native library not loaded
    20/01/19 20:35:13 INFO mapred.JobClient: Running job: job_202001192032_0002
    20/01/19 20:35:14 INFO mapred.JobClient:  map 0% reduce 0%
    20/01/19 20:35:22 INFO mapred.JobClient:  map 50% reduce 0%
    20/01/19 20:35:32 INFO mapred.JobClient:  map 100% reduce 0%
    20/01/19 20:35:33 INFO mapred.JobClient:  map 100% reduce 33%
    20/01/19 20:35:35 INFO mapred.JobClient:  map 100% reduce 100%
    20/01/19 20:35:36 INFO mapred.JobClient: Job complete: job_202001192032_0002
    20/01/19 20:35:36 INFO mapred.JobClient: Counters: 29
    20/01/19 20:35:36 INFO mapred.JobClient:   Map-Reduce Framework
    20/01/19 20:35:36 INFO mapred.JobClient:     Spilled Records=54
    20/01/19 20:35:36 INFO mapred.JobClient:     Map output materialized bytes=331
    20/01/19 20:35:36 INFO mapred.JobClient:     Reduce input records=27
    20/01/19 20:35:36 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=5852057600
    20/01/19 20:35:36 INFO mapred.JobClient:     Map input records=8
    20/01/19 20:35:36 INFO mapred.JobClient:     SPLIT_RAW_BYTES=246
    20/01/19 20:35:36 INFO mapred.JobClient:     Map output bytes=503
    20/01/19 20:35:36 INFO mapred.JobClient:     Reduce shuffle bytes=331
    20/01/19 20:35:36 INFO mapred.JobClient:     Physical memory (bytes) snapshot=411611136
    20/01/19 20:35:36 INFO mapred.JobClient:     Reduce input groups=19
    20/01/19 20:35:36 INFO mapred.JobClient:     Combine output records=27
    20/01/19 20:35:36 INFO mapred.JobClient:     Reduce output records=19
    20/01/19 20:35:36 INFO mapred.JobClient:     Map output records=51
    20/01/19 20:35:36 INFO mapred.JobClient:     Combine input records=51
    20/01/19 20:35:36 INFO mapred.JobClient:     CPU time spent (ms)=980
    20/01/19 20:35:36 INFO mapred.JobClient:     Total committed heap usage (bytes)=292954112
    20/01/19 20:35:36 INFO mapred.JobClient:   File Input Format Counters
    20/01/19 20:35:36 INFO mapred.JobClient:     Bytes Read=300
    20/01/19 20:35:36 INFO mapred.JobClient:   FileSystemCounters
    20/01/19 20:35:36 INFO mapred.JobClient:     HDFS_BYTES_READ=546
    20/01/19 20:35:36 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=156082
    20/01/19 20:35:36 INFO mapred.JobClient:     FILE_BYTES_READ=325
    20/01/19 20:35:36 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=147
    20/01/19 20:35:36 INFO mapred.JobClient:   Job Counters
    20/01/19 20:35:36 INFO mapred.JobClient:     Launched map tasks=3
    20/01/19 20:35:36 INFO mapred.JobClient:     Launched reduce tasks=1
    20/01/19 20:35:36 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=11869
    20/01/19 20:35:36 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
    20/01/19 20:35:36 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=13320
    20/01/19 20:35:36 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
    20/01/19 20:35:36 INFO mapred.JobClient:     Data-local map tasks=3
    20/01/19 20:35:36 INFO mapred.JobClient:   File Output Format Counters
    20/01/19 20:35:36 INFO mapred.JobClient:     Bytes Written=147

6.查看执行结果

    [root@liyan examples]# hadoop fs -ls output_wordcount/
    Found 3 items
    -rw-r--r--   3 root supergroup          0 2020-01-19 20:35 /user/root/output_wordcount/_SUCCESS
    drwxr-xr-x   - root supergroup          0 2020-01-19 20:35 /user/root/output_wordcount/_logs
    -rw-r--r--   3 root supergroup        147 2020-01-19 20:35 /user/root/output_wordcount/part-r-00000

part-r-00000文件中记录了统计结果

    [root@liyan examples]# hadoop fs -cat output_wordcount/part-r-00000
    five    2
    four    2
    hadoop  7
    help    3
    liyan   6
    love    7
    script  1
    six 1
    tacker  2
    tracker 4
    wend    5
    window  2
    word    3

来源我的个人博客