-大数据入门-1-Hadoop-hdfs伪分布式部署

Hadoop 大象

广义: 以apache hadoop软件为主的生态圈(hive sqoop spark flink…)
狭义: apache hadoop软件

官网

APAche:www.apache.org
CDH:http://archive.cloudera.com/cdh5/cdh/5/
Hadoop
1.x 企业不用
2.x 主流
3.x 很少敢用 坑 只能自己踩
但是CDH(www.cloudera.com) 版本兼容 统一的部署管理 很受企业欢迎 HDP
拿apache hadoop源代码 ，开源免费，封装成自己的hadoop版本 ，且自己打补丁
hadoop-2.6.0-cdh5.7.0-changes.log 作了变更，打了补丁，未来在公司，遇到问题，就在这里找，为了未来升级作准备
hadoop-2.6.0-cdh5.7.0-src.tar.gz src源代码
hadoop-2.6.0-cdh5.7.0.tar.gz 编译之后的 bin包
慎用CDH5.11.0 bug

Hadoop软件

1.存储 hdfs 分布式文件系统 ext3 ext4 需要部署 (hdfs、hive 、hbase、 kudu)
2.计算 mapreduce 分布式计算 job ,mapreduce 不用部署 (hive sql、spark、flink)
3.资源(memory cpu)和作业调度 yarn 需要部署

hadoop部署

1. wget 链接或rz上传
2. 创建hadoop用户

useradd hadoop
[root@hadoop001 ~]# su - hadoop
[hadoop@hadoop001 ~]$ pwd
/home/hadoop

3. JDK部署

[hadoop@hadoop001 ~]$ which java
/usr/java/jdk1.8.0_45/bin/java       目录:务必 /usr/java/
[hadoop@hadoop001 ~]$ ll          用户用户组修正
total 16
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:56 app
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 data
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 logs
drwxrwxr-x 3 hadoop hadoop 4096 Jul  8 10:52 software
[root@hadoop001 ~]# vi /etc/profile   

export JAVA_HOME=/usr/java/jdk1.8.0_45
export MYSQL_HOME=/usr/local/mysql 
export HADOOP_HOME=/usr/home/hadoop            全局环境变量配置 置前
PATH=${HADOOP_HOME}/bin:${MYSQL__HOME}/bin:${JAVA_HOME}/bin:$PATH        
export PATH

[root@hadoop001 ~]# source /etc/profile    

4. 生产上目录规划

[hadoop@hadoop001 ~]$ mkdir app software data logs
[hadoop@hadoop001 ~]$ ll
total 16
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 app
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 data
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 logs
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 software
[hadoop@hadoop001 ~]$ pwd       
/home/hadoop

5. 解压

mv tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz /home/hadoop/software
tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[root@hadoop001 hadoop]# ll software
total 304288
drwxr-xr-x 14 hadoop hadoop      4096 Mar 24  2016 hadoop-2.6.0-cdh5.7.0
-rw-r--r--  1 hadoop hadoop 311585484 Jul  1 10:46 hadoop-2.6.0-cdh5.7.0.ta
[root@hadoop001 hadoop]# ln -s /home/hadoop/software/hadoop-2.6.0-cdh5.7.0 /home/hadoop/app/hadoop
[root@hadoop001 hadoop]# ll app
total 0
lrwxrwxrwx 1 root root 43 Jul  8 10:56 hadoop -> /home/hadoop/software/hadoop-2.6.0-cdh5.7.0
[root@hadoop001 ~]# chown -R hadoop:hadoop /home/hadoop/app/hadoop
[root@hadoop001 ~]# cd /home/hadoop/app/     用户变更
[root@hadoop001 app]# ll
total 0
lrwxrwxrwx 1 hadoop hadoop 43 Jul  8 10:56 hadoop -> /home/hadoop/software/hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 app]$ cd hadoop/
[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop
[hadoop@hadoop001 hadoop]$ ll
total 76
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin   可执行脚本文件夹
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc     配置文件夹
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib      jar包存储文件夹
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
-rw-r--r--  1 hadoop hadoop 17087 Mar 24  2016 LICENSE.txt
-rw-r--r--  1 hadoop hadoop   101 Mar 24  2016 NOTICE.txt
-rw-r--r--  1 hadoop hadoop  1366 Mar 24  2016 README.txt
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin     Hadoop组件 启动 停止
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src

6. 官网

[hadoop@hadoop001 app]$ cd hadoop
[hadoop@hadoop001 hadoop]$ ll
total 76
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
-rw-r--r--  1 hadoop hadoop 17087 Mar 24  2016 LICENSE.txt
-rw-r--r--  1 hadoop hadoop   101 Mar 24  2016 NOTICE.txt
-rw-r--r--  1 hadoop hadoop  1366 Mar 24  2016 README.txt
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src
[hadoop@hadoop001 hadoop]$ cd etc/
[hadoop@hadoop001 etc]$ ll
total 16
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24  2016 hadoop
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24  2016 hadoop-mapreduce1
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24  2016 hadoop-mapreduce1-pseudo
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24  2016 hadoop-mapreduce1-secure
[hadoop@hadoop001 etc]$ cd /hadoop
-bash: cd: /hadoop: No such file or directory
[hadoop@hadoop001 etc]$ cd hadoop/
[hadoop@hadoop001 hadoop]$ ll
total 152
-rw-r--r-- 1 hadoop hadoop  4436 Mar 24  2016 capacity-scheduler.xml
-rw-r--r-- 1 hadoop hadoop  1335 Mar 24  2016 configuration.xsl
-rw-r--r-- 1 hadoop hadoop   318 Mar 24  2016 container-executor.cfg
-rw-r--r-- 1 hadoop hadoop   774 Mar 24  2016 core-site.xml
-rw-r--r-- 1 hadoop hadoop  3670 Mar 24  2016 hadoop-env.cmd
-rw-r--r-- 1 hadoop hadoop  4224 Mar 24  2016 hadoop-env.sh
-rw-r--r-- 1 hadoop hadoop  2598 Mar 24  2016 hadoop-metrics2.properties
-rw-r--r-- 1 hadoop hadoop  2490 Mar 24  2016 hadoop-metrics.properties
-rw-r--r-- 1 hadoop hadoop  9683 Mar 24  2016 hadoop-policy.xml
-rw-r--r-- 1 hadoop hadoop   775 Mar 24  2016 hdfs-site.xml
-rw-r--r-- 1 hadoop hadoop  1449 Mar 24  2016 httpfs-env.sh
-rw-r--r-- 1 hadoop hadoop  1657 Mar 24  2016 httpfs-log4j.properties
-rw-r--r-- 1 hadoop hadoop    21 Mar 24  2016 httpfs-signature.secret
-rw-r--r-- 1 hadoop hadoop   620 Mar 24  2016 httpfs-site.xml
-rw-r--r-- 1 hadoop hadoop  3523 Mar 24  2016 kms-acls.xml
-rw-r--r-- 1 hadoop hadoop  1611 Mar 24  2016 kms-env.sh
-rw-r--r-- 1 hadoop hadoop  1631 Mar 24  2016 kms-log4j.properties
-rw-r--r-- 1 hadoop hadoop  5511 Mar 24  2016 kms-site.xml
-rw-r--r-- 1 hadoop hadoop 11291 Mar 24  2016 log4j.properties
-rw-r--r-- 1 hadoop hadoop   938 Mar 24  2016 mapred-env.cmd
-rw-r--r-- 1 hadoop hadoop  1383 Mar 24  2016 mapred-env.sh
-rw-r--r-- 1 hadoop hadoop  4113 Mar 24  2016 mapred-queues.xml.template
-rw-r--r-- 1 hadoop hadoop   758 Mar 24  2016 mapred-site.xml.template
-rw-r--r-- 1 hadoop hadoop    10 Mar 24  2016 slaves
-rw-r--r-- 1 hadoop hadoop  2316 Mar 24  2016 ssl-client.xml.example
-rw-r--r-- 1 hadoop hadoop  2268 Mar 24  2016 ssl-server.xml.example
-rw-r--r-- 1 hadoop hadoop  2237 Mar 24  2016 yarn-env.cmd
-rw-r--r-- 1 hadoop hadoop  4567 Mar 24  2016 yarn-env.sh
-rw-r--r-- 1 hadoop hadoop   690 Mar 24  2016 yarn-site.xml
[hadoop@hadoop001 hadoop]$ rm -f *.cmd  删除cmd win的，我们不需要
[hadoop@hadoop001 hadoop]$ ll
total 140
-rw-r--r-- 1 hadoop hadoop  4436 Mar 24  2016 capacity-scheduler.xml
-rw-r--r-- 1 hadoop hadoop  1335 Mar 24  2016 configuration.xsl
-rw-r--r-- 1 hadoop hadoop   318 Mar 24  2016 container-executor.cfg
-rw-r--r-- 1 hadoop hadoop   774 Mar 24  2016 core-site.xml
-rw-r--r-- 1 hadoop hadoop  4224 Mar 24  2016 hadoop-env.sh
-rw-r--r-- 1 hadoop hadoop  2598 Mar 24  2016 hadoop-metrics2.properties
-rw-r--r-- 1 hadoop hadoop  2490 Mar 24  2016 hadoop-metrics.properties
-rw-r--r-- 1 hadoop hadoop  9683 Mar 24  2016 hadoop-policy.xml
-rw-r--r-- 1 hadoop hadoop   775 Mar 24  2016 hdfs-site.xml
-rw-r--r-- 1 hadoop hadoop  1449 Mar 24  2016 httpfs-env.sh
-rw-r--r-- 1 hadoop hadoop  1657 Mar 24  2016 httpfs-log4j.properties
-rw-r--r-- 1 hadoop hadoop    21 Mar 24  2016 httpfs-signature.secret
-rw-r--r-- 1 hadoop hadoop   620 Mar 24  2016 httpfs-site.xml
-rw-r--r-- 1 hadoop hadoop  3523 Mar 24  2016 kms-acls.xml
-rw-r--r-- 1 hadoop hadoop  1611 Mar 24  2016 kms-env.sh
-rw-r--r-- 1 hadoop hadoop  1631 Mar 24  2016 kms-log4j.properties
-rw-r--r-- 1 hadoop hadoop  5511 Mar 24  2016 kms-site.xml
-rw-r--r-- 1 hadoop hadoop 11291 Mar 24  2016 log4j.properties
-rw-r--r-- 1 hadoop hadoop  1383 Mar 24  2016 mapred-env.sh
-rw-r--r-- 1 hadoop hadoop  4113 Mar 24  2016 mapred-queues.xml.template
-rw-r--r-- 1 hadoop hadoop   758 Mar 24  2016 mapred-site.xml.template
-rw-r--r-- 1 hadoop hadoop    10 Mar 24  2016 slaves
-rw-r--r-- 1 hadoop hadoop  2316 Mar 24  2016 ssl-client.xml.example
-rw-r--r-- 1 hadoop hadoop  2268 Mar 24  2016 ssl-server.xml.example
-rw-r--r-- 1 hadoop hadoop  4567 Mar 24  2016 yarn-env.sh
-rw-r--r-- 1 hadoop hadoop   690 Mar 24  2016 yarn-site.xml
[hadoop@hadoop001 hadoop]$ vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_45

Now you are ready to start your Hadoop cluster in one of the three supported modes:三种启动Hadoop的模式

Local (Standalone) Mode 0进程 1台
Pseudo-Distributed Mode 伪分布式 多个进程 1台
Fully-Distributed Mode 集群 多个进程 n台

7. hdfs伪分布式部署

[hadoop@hadoop001 hadoop]$ vi core-site.xml









        fs.defaultFS
        hdfs://localhost:9000
    

[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml









        dfs.replication
        1
    
                                                                  

8. 设置无密码ssh
   可参考官网进行设置

Now check that you can ssh to the localhost without a passphrase:

  $ ssh localhost
If you cannot ssh to localhost without a passphrase, execute the following commands:

  $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  $ chmod 0600 ~/.ssh/authorized_keys

[hadoop@hadoop001 hadoop]$ cd 
[hadoop@hadoop001 ~]$ ll -a
total 48
drwx------   7 hadoop hadoop 4096 Jul  9 01:39 .
drwxr-xr-x. 30 root   root   4096 Jul  8 10:13 ..
drwxrwxr-x   2 hadoop hadoop 4096 Jul  8 10:56 app
-rw-------   1 hadoop hadoop  475 Jul  9 00:31 .bash_history
-rw-r--r--   1 hadoop hadoop   18 Mar 23  2017 .bash_logout
-rw-r--r--   1 hadoop hadoop  176 Mar 23  2017 .bash_profile
-rw-r--r--   1 hadoop hadoop  124 Mar 23  2017 .bashrc
drwxrwxr-x   2 hadoop hadoop 4096 Jul  8 10:33 data
drwxrwxr-x   2 hadoop hadoop 4096 Jul  8 10:33 logs
drwxrwxr-x   3 hadoop hadoop 4096 Jul  8 10:52 software
drwx------   2 hadoop hadoop 4096 Jul  9 01:39 .ssh
-rw-------   1 hadoop hadoop 3583 Jul  9 01:39 .viminfo
[hadoop@hadoop001 ~]$ cd .ssh
[hadoop@hadoop001 .ssh]$ ll
total 4
-rw-r--r-- 1 hadoop hadoop 391 Jul  9 01:39 known_hosts
[hadoop@hadoop001 .ssh]$ more known_hosts
localhost ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Fjip5f+cmSZZICJ5Txy9d2jwJZjs
F7NzRRuDR6c2BGa7SKaqdcMd2jqbL42kLZ8r/6HFL3qZDlGEq9eHJnWt0Yc41cWygNzCZW9Kfoj
l4y7n2YXp6tZ7B4GpOcT/bQ/S7apci044RWuD34F25pTIHuQ98XMSDTbJfs9BLiqaBKuGlxuce7
XYdjf1s81/DfLfuu51AcJNu1nBt2TD5ru9VdZu0O0y4RA7dmOcrlkWSFChUGFKFraQxdMdLHBqE
tu+dQXTdtYYHNBoBMAaX0nQ5+m89oC7GxICk9LWgvoiTivitR2YZxD07bVV2/IxmxNV/UhI5+Yx
sZEMatpwir0Sw==
[hadoop@hadoop001 .ssh]$ cd ../
[hadoop@hadoop001 ~]$ rm -rf .ssh
[hadoop@hadoop001 ~]$ ll -a
total 44
drwx------   6 hadoop hadoop 4096 Jul  9 01:45 .
drwxr-xr-x. 30 root   root   4096 Jul  8 10:13 ..
drwxrwxr-x   2 hadoop hadoop 4096 Jul  8 10:56 app
-rw-------   1 hadoop hadoop  475 Jul  9 00:31 .bash_history
-rw-r--r--   1 hadoop hadoop   18 Mar 23  2017 .bash_logout
-rw-r--r--   1 hadoop hadoop  176 Mar 23  2017 .bash_profile
-rw-r--r--   1 hadoop hadoop  124 Mar 23  2017 .bashrc
drwxrwxr-x   2 hadoop hadoop 4096 Jul  8 10:33 data
drwxrwxr-x   2 hadoop hadoop 4096 Jul  8 10:33 logs
drwxrwxr-x   3 hadoop hadoop 4096 Jul  8 10:52 software
-rw-------   1 hadoop hadoop 3583 Jul  9 01:39 .viminfo

也可ssh-keygen

[hadoop@hadoop001 ~]$ which ssh-keygen
/usr/bin/ssh-keygen
[hadoop@hadoop001 ~]$ ll /usr/bin/ssh*
-rwxr-xr-x 1 root root   450728 Sep  1  2017 /usr/bin/ssh
-rwxr-xr-x 1 root root   158920 Sep  1  2017 /usr/bin/ssh-add
-rwxr-sr-x 1 root nobody 141384 Sep  1  2017 /usr/bin/ssh-agent
-rwxr-xr-x 1 root root     1425 Sep  1  2017 /usr/bin/ssh-copy-id
-rwxr-xr-x 1 root root   199880 Sep  1  2017 /usr/bin/ssh-keygen
-rwxr-xr-x 1 root root   253752 Sep  1  2017 /usr/bin/ssh-keyscan
[hadoop@hadoop001 ~]$ ssh-keygen       三次回车
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
0b:64:be:3c:54:d7:9c:56:b3:5f:8d:93:4a:d6:44:1e hadoop@hadoop001
The key's randomart image is:
+--[ RSA 2048]----+
|             .E  |
|           o * *.|
|      o . . B B o|
|     + . . + . o.|
|      + S   .   .|
|     o o .       |
|      + .        |
|       .         |
|                 |
+-----------------+
[hadoop@hadoop001 ~]$ cd .ssh
[hadoop@hadoop001 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Jul  9 01:49 id_rsa      私钥
-rw-r--r-- 1 hadoop hadoop  398 Jul  9 01:49 id_rsa.pub你    公钥
[hadoop@hadoop001 .ssh]$ cat id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA2KCZBYaMiDxuIy9S0XwGiu4X2HLSCOy28ph0BcOSS52VyumT
rG5zr0LmUZL3spv3O9rkz9KqAOWtYBHL+nYMPOU8wDmbrC4PdrMNo56Wkdt1IyOQ
BGX1xk7hBQPBXBiTvO3umLB6iLtmIfxM1fixjHirq3k0B9rLyh+GVcgUPGq24hQa
gep8W7YUlDvTtSReal5ymhJ0zlpx1xeH/Aeq++aI4KLkpU3QR9ju6awgWhWeE9KG
luK1e0oYm2Jx6O3JOUc2vXEUI7IIpOp2pLeZwrdH6inS1znE57ukXeQhqy9ijiXb
wrC9nosLJEHmQJi25vwZHyuV9KmBx5U/Ghsg0QIBIwKCAQBQdio1QJqZAICt+6Jr
EM866rhmVoiG+NY82YouBs/wMzejI46fIbXwo9HjyN+ovZZYD3mPBRqSnnOYticx
JIgz4B3g/3RV5TjxkvZv+RNavzoi/pST61tJqETzO+i7zp1GLH0xZh7iVDvgvMwU
9fjNxmumFxNTX+VLEwYCmsWwCbIBcFCymjWI+Soal7NCGlHSvYagWERaqIu9IxYd
sN/QmLJ6LEKo3q4/WGs4iwj2uaRnB8bOCKYkzOVJC3/PCJw4ajp85UODyCN5C05r
GFb/HJQEmEEYHin82s9NlVxsKDUVKqA/xk1t/0KuxSyeexBmMUYoeDjg5/ui+9C3
ZudnAoGBAPg4kl83RLxRlcqtPgl/rjIifiDqJGrt6mIWAvhk2o3g/QyvirpkZoRl
Ya3nHewjANQp88Eojmxrt0CrJdOiYMuDM23WYBHQ2ndTG87cDy98qmAwkvnUEM0Z
IVmkIDGYwk5yPqtSjsitcL/jQM5Wpjyaxuy9zaSFI7NL1QLcY2/HAoGBAN9qjmvf
8uEjMrEpPftIamOuCXET7LRO0as6BN3kxxFLxIf73BRS3opy/1tqUvSyg3HAydWS
ZGJdufh6WLxUOruUWD8gp8K0989asDceepO+mJRKwdr9Tek5iEP8lEYC7SK7iB6J
WtW1Q6tqeBXwf8RMxDRqkGv+UFBifLuLBRqnAoGBAOoJdBCh0xf1JtUJviY2jkyG
6/Ml59KIjIhd5YsdQxgOpXJcXjq2b0mharnoie1FkxEu3oLruXwrC93jMksyswkG
rNVGd9ZIk3fSBEayOjQWdMEX2wjPQwqFaJZY6ydyxdTvX6jYz8SGRbw8qtEtIGUG
9hJp0IUtEwgi7WkY8ApNAoGBAJkzH9TxVha5DNFA2gtk2z0JoBMGWSsuu6ic0CMg
iIDjf3MwXGW1KuKX/48kViQio0awFV8/z85sJ8BT5RNs9RLpYRVYOIWDaBkoPlGu
f/BILhyLCJYqCYn7kKOl4gQfQ4WH6E90IQeSPQfT+pKzis+/nHu2yW6fwgs8OEYW
L2K7AoGBAPRYLDGUzJZ/aJm+zMNLQ/GDU8yuZkArJ4rGrg6MPAAqTGQXHM+/m8/d
Kra4ecq9VsGHPSTWzeu8onJeSy5q6REhnNPT01tnU410lxcxPe2OzU7iFGHsFZS5
Ydadpk+4NlEXdrsnl9sgsy/SwGWUWsQu+DSH8CICpyEDKjH1S048
-----END RSA PRIVATE KEY-----
[hadoop@hadoop001 .ssh]$ cat id_rsa.pub   真正需要的是公钥
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2KCZBYaMiDxuIy9S0XwGiu4X2HLSCOy28ph0BcOSS52VyumTrG5zr0LmUZL3spv3O9rkz9KqAOWtYBHL+nYMPOU8wDmbrC4PdrMNo56Wkdt1IyOQBGX1xk7hBQPBXBiTvO3umLB6iLtmIfxM1fixjHirq3k0B9rLyh+GVcgUPGq24hQagep8W7YUlDvTtSReal5ymhJ0zlpx1xeH/Aeq++aI4KLkpU3QR9ju6awgWhWeE9KGluK1e0oYm2Jx6O3JOUc2vXEUI7IIpOp2pLeZwrdH6inS1znE57ukXeQhqy9ijiXbwrC9nosLJEHmQJi25vwZHyuV9KmBx5U/Ghsg0Q== hadoop@hadoop001
hadoop@hadoop001                                       用户@机器

[hadoop@hadoop001 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Jul  9 01:49 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Jul  9 01:49 id_rsa.pub
[hadoop@hadoop001 .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@hadoop001 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop  398 Jul  9 02:05 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul  9 01:49 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Jul  9 01:49 id_rsa.pub

[hadoop@hadoop001 .ssh]$ ssh localhost date  
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is d4:0e:26:f5:f6:2c:12:17:ba:9d:19:c4:10:fe:d8:2a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
hadoop@localhost's password: 
Permission denied, please try again.
hadoop@localhost's password: 
Permission denied, please try again.
hadoop@localhost's password: 
Permission denied (publickey,password).
[hadoop@hadoop001 .ssh]$ ll
total 16
-rw-rw-r-- 1 hadoop hadoop  398 Jul  9 02:05 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul  9 01:49 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Jul  9 01:49 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  391 Jul  9 02:07 known_hosts      还是不行，因为我们配置的是Hadoop001，不是localhost
[hadoop@hadoop001 .ssh]$ 

[hadoop@hadoop001 .ssh]$ ssh hadoop date
ssh: Could not resolve hostname hadoop: Name or service not known
[hadoop@hadoop001 .ssh]$ ssh hadoop001 data
The authenticity of host 'hadoop001 (106.75.103.178)' can't be established.
RSA key fingerprint is d4:0e:26:f5:f6:2c:12:17:ba:9d:19:c4:10:fe:d8:2a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop001,106.75.103.178' (RSA) to the list of known hosts.
hadoop@hadoop001's password: 
Permission denied, please try again.
hadoop@hadoop001's password: 
Permission denied, please try again.
hadoop@hadoop001's password: 
Permission denied (publickey,password).       Hadoop001还是不行，权限问题，apache官网有关于此处解决方法，cdh无
[hadoop@hadoop001 .ssh]$ 

[hadoop@hadoop001 .ssh]$ pwd
/home/hadoop/.ssh
[hadoop@hadoop001 .ssh]$ chmod 600 /home/hadoop/.ssh/authorized_keys    按照官网配置600权限
[hadoop@hadoop001 .ssh]$ ll
total 16
-rw------- 1 hadoop hadoop  398 Jul  9 02:05 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul  9 01:49 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Jul  9 01:49 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  797 Jul  9 02:11 known_hosts

[hadoop@hadoop001 .ssh]$ ssh localhost date   成功后测试
Tue Jul  9 02:15:20 CST 2019
[hadoop@hadoop001 .ssh]$ ssh 10.9.6.136 date     sh登录localhost机器执行date命令 返回结果 不会做机器切换
Tue Jul  9 02:18:54 CST 2019
[hadoop@hadoop001 .ssh]$ ssh hadoop001 date
Tue Jul  9 02:19:56 CST 2019

[hadoop@hadoop001 .ssh]$ ssh hadoop001   shh登录Hadoop001机器作了机器切换
[hadoop@hadoop001 ~]$ 

远程连接

ssh root@ip
输入密码
ssh root@ip date    不会远程切换到ip

9. 正式开始

[hadoop@hadoop001 ~]$ ll
total 16
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:56 app
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 data
drwxrwxr-x 2 hadoop hadoop 4096 Jul  8 10:33 logs
drwxrwxr-x 3 hadoop hadoop 4096 Jul  8 10:52 software
[hadoop@hadoop001 ~]$ cd app/hadoop
[hadoop@hadoop001 hadoop]$ ll
total 76
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
-rw-r--r--  1 hadoop hadoop 17087 Mar 24  2016 LICENSE.txt
-rw-r--r--  1 hadoop hadoop   101 Mar 24  2016 NOTICE.txt
-rw-r--r--  1 hadoop hadoop  1366 Mar 24  2016 README.txt
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src
[hadoop@hadoop001 hadoop]$ bin/hdfs namenode -format  格式化

/dfs/name has been successfully formatted.      这个代表成功
19/07/09 17:15:18 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
19/07/09 17:15:18 INFO util.ExitUtil: Exiting with status 0
19/07/09 17:15:18 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at 10-9-6-136/127.0.0.1

sbin/start-dfs.sh  启动
[hadoop@hadoop001 sbin]$ jps
897 SecondaryNameNode
1090 Jps
712 DataNode
617 NameNode

[hadoop@hadoop001 ~]$ cd app
[hadoop@hadoop001 app]$ ll
total 0
lrwxrwxrwx 1 hadoop hadoop 43 Jul  8 10:56 hadoop -> /home/hadoop/software/hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 app]$ cd hadoop
[hadoop@hadoop001 hadoop]$ ll
total 80
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
-rw-r--r--  1 hadoop hadoop 17087 Mar 24  2016 LICENSE.txt
drwxrwxr-x  2 hadoop hadoop  4096 Jul  9 02:40 logs
-rw-r--r--  1 hadoop hadoop   101 Mar 24  2016 NOTICE.txt
-rw-r--r--  1 hadoop hadoop  1366 Mar 24  2016 README.txt
drwxr-xr-x  3 hadoop hadoop  4096 Jul  9 02:45 sbin
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src
[hadoop@hadoop001 hadoop]$ ll logs    日志
total 112
-rw-rw-r-- 1 hadoop hadoop 28240 Jul  9 02:40 hadoop-hadoop-datanode-hadoop001.log               有问题tail这里定位查看
-rw-rw-r-- 1 hadoop hadoop   729 Jul  9 02:40 hadoop-hadoop-datanode-hadoop001.out
-rw-rw-r-- 1 hadoop hadoop 43895 Jul  9 02:56 hadoop-hadoop-namenode-hadoop001.log
-rw-rw-r-- 1 hadoop hadoop   729 Jul  9 02:40 hadoop-hadoop-namenode-hadoop001.out
-rw-rw-r-- 1 hadoop hadoop 28316 Jul  9 02:41 hadoop-hadoop-secondarynamenode-hadoop001.log
-rw-rw-r-- 1 hadoop hadoop   729 Jul  9 02:40 hadoop-hadoop-secondarynamenode-hadoop001.out
-rw-rw-r-- 1 hadoop hadoop     0 Jul  9 02:40 SecurityAuth-hadoop.audit

[root@hadoop001 ~]# netstat -nlp|grep 50070  端口号
tcp        0      0 0.0.0.0:50070               0.0.0.0:*   

再次格式化后如何出现问题解决

问题
[hadoop@hadoop001 hadoop]$ sbin/start-dfs.sh
19/07/09 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop001.out
localhost: starting datanode, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop001.out
Starting secondary namenodes [pastebin.com]       ？？？

pastebin.com: starting secondarynamenode, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop001.out
19/07/09 17:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ jps
8440 NameNode    ？？
8235 NameNode    ？？
8716 SecondaryNameNode
8831 Jps
[hadoop@hadoop001 hadoop]$ 

解决方案
kill -9 $(pgrep -f hadoop)

rm -rf /tmp/hadoop*

然后开始重新格式化  启动

[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir /user
19/07/09 18:01:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /
19/07/09 18:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-07-09 18:01 /user
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /user/
19/07/09 18:45:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-07-09 18:44 /user/hadoop

[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -put etc/hadoop input        把Linux源端的拷贝hadoop的终端
19/07/09 18:48:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /user/hadoop/    默认在当前命令的目录下面
19/07/09 19:07:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-07-09 18:48 /user/hadoop/input
    
    [hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /user/hadoop/input/
    19/07/09 19:09:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Found 26 items
    -rw-r--r--   1 hadoop supergroup       4436 2019-07-09 18:48 /user/hadoop/input/capacity-scheduler.xml
    -rw-r--r--   1 hadoop supergroup       1335 2019-07-09 18:48 /user/hadoop/input/configuration.xsl
    -rw-r--r--   1 hadoop supergroup        318 2019-07-09 18:48 /user/hadoop/input/container-executor.cfg
    -rw-r--r--   1 hadoop supergroup        880 2019-07-09 18:48 /user/hadoop/input/core-site.xml
    -rw-r--r--   1 hadoop supergroup       4233 2019-07-09 18:48 /user/hadoop/input/hadoop-env.sh
    -rw-r--r--   1 hadoop supergroup       2490 2019-07-09 18:48 /user/hadoop/input/hadoop-metrics.properties
    -rw-r--r--   1 hadoop supergroup       2598 2019-07-09 18:48 /user/hadoop/input/hadoop-metrics2.properties
    -rw-r--r--   1 hadoop supergroup       9683 2019-07-09 18:48 /user/hadoop/input/hadoop-policy.xml
    -rw-r--r--   1 hadoop supergroup        863 2019-07-09 18:48 /user/hadoop/input/hdfs-site.xml
    -rw-r--r--   1 hadoop supergroup       1449 2019-07-09 18:48 /user/hadoop/input/httpfs-env.sh
    -rw-r--r--   1 hadoop supergroup       1657 2019-07-09 18:48 /user/hadoop/input/httpfs-log4j.properties
    -rw-r--r--   1 hadoop supergroup         21 2019-07-09 18:48 /user/hadoop/input/httpfs-signature.secret
    -rw-r--r--   1 hadoop supergroup        620 2019-07-09 18:48 /user/hadoop/input/httpfs-site.xml
    -rw-r--r--   1 hadoop supergroup       3523 2019-07-09 18:48 /user/hadoop/input/kms-acls.xml
    -rw-r--r--   1 hadoop supergroup       1611 2019-07-09 18:48 /user/hadoop/input/kms-env.sh
    -rw-r--r--   1 hadoop supergroup       1631 2019-07-09 18:48 /user/hadoop/input/kms-log4j.properties
    -rw-r--r--   1 hadoop supergroup       5511 2019-07-09 18:48 /user/hadoop/input/kms-site.xml
    -rw-r--r--   1 hadoop supergroup      11291 2019-07-09 18:48 /user/hadoop/input/log4j.properties
    -rw-r--r--   1 hadoop supergroup       1383 2019-07-09 18:48 /user/hadoop/input/mapred-env.sh
    -rw-r--r--   1 hadoop supergroup       4113 2019-07-09 18:48 /user/hadoop/input/mapred-queues.xml.template
    -rw-r--r--   1 hadoop supergroup        758 2019-07-09 18:48 /user/hadoop/input/mapred-site.xml.template
    -rw-r--r--   1 hadoop supergroup         10 2019-07-09 18:48 /user/hadoop/input/slaves
    -rw-r--r--   1 hadoop supergroup       2316 2019-07-09 18:48 /user/hadoop/input/ssl-client.xml.example
    -rw-r--r--   1 hadoop supergroup       2268 2019-07-09 18:48 /user/hadoop/input/ssl-server.xml.example
    -rw-r--r--   1 hadoop supergroup       4567 2019-07-09 18:48 /user/hadoop/input/yarn-env.sh
    -rw-r--r--   1 hadoop supergroup        690 2019-07-09 18:48 /user/hadoop/input/yarn-site.xml
    [hadoop@hadoop001 hadoop]$ 
    
    [hadoop@hadoop001 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar grep input output 'dfs[a-z.]+'      相当于计算
    
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -get output output
19/07/09 19:16:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ cat output/*
4	dfs.class
4	dfs.audit.logger
3	dfs.server.namenode.
2	dfs.period
2	dfs.audit.log.maxfilesize
2	dfs.audit.log.maxbackupindex
1	dfsmetrics.log
1	dfsadmin
1	dfs.servers
1	dfs.replication
1	dfs.file
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /user/hadoop/
19/07/09 19:18:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2019-07-09 18:48 /user/hadoop/input
drwxr-xr-x   - hadoop supergroup          0 2019-07-09 19:13 /user/hadoop/output
[hadoop@hadoop001 hadoop]$ bin/hdfs dfs -ls /user/hadoop/output/
19/07/09 19:19:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2019-07-09 19:13 /user/hadoop/output/_SUCCESS
-rw-r--r--   1 hadoop supergroup        197 2019-07-09 19:13 /user/hadoop/output/part-r-00000
    [hadoop@hadoop001 hadoop]$ 

[hadoop@hadoop001 hadoop]$ sbin/stop-dfs.sh    停止
19/07/09 19:22:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [pastebin.com]
pastebin.com: stopping secondarynamenode
19/07/09 19:22:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 hadoop]$ 

bin/hdfs dfs 相当于help命令帮助

[hadoop@hadoop001 hadoop]$ bin/hdfs dfs
Usage: hadoop fs [generic options]
	[-appendToFile  ... ]
	[-cat [-ignoreCrc]  ...]
	[-checksum  ...]
	[-chgrp [-R] GROUP PATH...]
	[-chmod [-R]  PATH...]
	[-chown [-R] [OWNER][:[GROUP]] PATH...]
	[-copyFromLocal [-f] [-p] [-l]  ... ]
	[-copyToLocal [-p] [-ignoreCrc] [-crc]  ... ]
	[-count [-q] [-h] [-v]  ...]
	[-cp [-f] [-p | -p[topax]]  ... ]
	[-createSnapshot  []]
	[-deleteSnapshot  ]
	[-df [-h] [ ...]]
	[-du [-s] [-h]  ...]
	[-expunge]
	[-find  ...  ...]
	[-get [-p] [-ignoreCrc] [-crc]  ... ]
	[-getfacl [-R] ]
	[-getfattr [-R] {-n name | -d} [-e en] ]
	[-getmerge [-nl]  ]
	[-help [cmd ...]]
	[-ls [-d] [-h] [-R] [ ...]]
	[-mkdir [-p]  ...]
	[-moveFromLocal  ... ]
	[-moveToLocal  ]
	[-mv  ... ]
	[-put [-f] [-p] [-l]  ... ]
	[-renameSnapshot   ]
	[-rm [-f] [-r|-R] [-skipTrash]  ...]
	[-rmdir [--ignore-fail-on-non-empty]  ...]
	[-setfacl [-R] [{-b|-k} {-m|-x } ]|[--set  ]]
	[-setfattr {-n name [-v value] | -x name} ]
	[-setrep [-R] [-w]   ...]
	[-stat [format]  ...]
	[-tail [-f] ]
	[-test -[defsz] ]
	[-text [-ignoreCrc]  ...]
	[-touchz  ...]
	[-usage [cmd ...]]

Generic options supported are
-conf      specify an application configuration file
-D             use value for given property
-fs       specify a namenode
-jt     specify a ResourceManager
-files     specify comma separated files to be copied to the map reduce cluster
-libjars     specify comma separated jar files to include in the classpath.
-archives     specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]