hadoop.笔记

HDFS

NameNode
Secondary(与NameNode同一台机)
主机名：nn01 ip:192.168.1.10

DateNode(三台机)
主机名：
node1 ip: 192.168.1.11
node2 ip: 192.168.1.12
node3 ip: 192.168.1.13

安装
Datenode：
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel

vim /etc/hosts
192.168.1.10 nn01
192.168.1.11 node1
192.168.1.12 ndoe2
192.168.1.13 node3

for i in {1..3}; do; scp /etc/hosts 192.168.1.1$i:/etc/hosts; done

datanode:
yum -y install java-1.8.0-openjdk-devel

修改配置文件(修改Datenode)
设置环境变量
vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=”/usr/lib/jvm/java-1.8.0openjdk-1.8.0.131-11.b12.el7.x86_64/jre”
export HADOOP_CONF_DIR=”/usr/local/hadoop/etc/hadoop”

分析单词出现的次数
cd /usr/local/hadoop
mkdir oo
cp LICENSE.txt NOTICE.txt README.txt oo
./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount oo xx

在Datenode上配置免密登陆(登陆node三台机不用密码)
ssh-keygen
ssh-copy-ip nn01 node1 node2 node3
远程SSH时不输入YES(指提示输入密码之前，四台机全部设置)
Vim /etc/ssh/ssh_config
60行添加 StrictHostKeyChecking no

修改其他配置文件(nn01上修改)
Cd /usr/local/hadoop/etc/

配置 core-site.xml


fs.defaultFS
hdfs://nn01:9000


hadoop.tmp.dir
/var/hadoop

ALL: 创建文件夹 /var/hadoop

配置 hdfs-site.xml


dfs.namenode.http-address
nn01:50070


dfs.namenode.secondary.http-address
nn01:50090


dfs.replication
2

配置 slaves
node1
node2
node3

将hadoop同步到所有的主机上(4台)
For i in {1..3}; do; scp /usr/local/hadoop node$i; done

Nn01: 格式化 namenode (没有err和warning，只有infor)
./bin/hdfs namenode -format

Nn01: 启动集群(显示启动node1 node2 node3)
./sbin/start-dfs.sh
停止集群可以使用 ./sbin/stop-dfs.sh

所有: 验证角色 jps(nn01:显示jps namenode secondaynamenode 其他:显示jps datanode)

Nn01: 验证集群是否组建成功(显示live datanode (3))
./bin/hdfs dfsadmin -report

服务日志路径 /usr/local/hadoop/logs(查看是否生成日志文件，nn01上，也可以用于排错)

./bin/hadoop fs -mkdir /oo
./bin/hadoop fs -ls /oo
./bin/hadoop fs -put *.txt /oo
./bin/hadoop fs -ls /oo
./bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount /oo /xx
./bin/hadoop fs -ls /
./bin/hadoop/ fs -cat /xx/*

./bin/hadoop fs -rm -f /xx/*
./bin/hadoop fs -rmdir /xx

HDFS增加节点
node4 192.168.1.14
ssh-copy-id [email protected] nn01
vim /etc/hosts nn01
192.168.1.14 node4
scp vim /etc/hosts 192.168.1.14:/etc/hosts nn01
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel node4
vim /usr/local/hadop/etc/hadoop/slaves
node4
for i in {1..3};do;scp /usr/local/hadop/etc/hadoop/slaves node$i:/usr/local/hadop/etc/hadoop/
scp -r /usr/local/hadoop node4:/usr/local/
/usr/local/hadoop/sbin/hadoop-daemon.sh start datanode node4
设置同步带宽 nn01

./bin/hdfs dfsadmin -setBalancerBandwidth 60000000
./sbin/start-balancer.sh
检测

HDFS删除节点 (node4节点删除)
cd /usr/local/hadoop
vim etc/hadoop/hdfs-site.xml

dfs.hosts.exclude
/usr/local/hadoop/etc/hadoop/exclude

touch etc/hadoop/exclude
vim etc/hadoop/exclude
添加要移除的节点：node4
./bin/hdfs dfsadmin -refreshNodes 开始导出数据，把node4中的数据导出来

./bin/hdfs dfsadmin -report 查看状态
Normal 正常状态
Decommissioned in Program 数据正在迁移
Decommissioned 数据迁移完成
vim etc/hadoop/exclude (去掉node4)
vim /etc/hosts (去掉192.168.1.14 node4)
for i in {1..3}; do scp /etc/hosts node i:/etc/;donevimetc/hadoop/slaves(去掉node4)foriin1..3;doscpetc/hadoop/slavesnode i : / e t c / ; d o n e v i m e t c / h a d o o p / s l a v e s ( 去 掉 n o d e 4 ) f o r i i n 1..3 ; d o s c p e t c / h a d o o p / s l a v e s n o d e i:/usr/local/hadoop/etc/hadoop/; done

[root@nn01 hadoop]# /usr/local/hadoop/sbin/stop-all.sh
[root@nn01 hadoop]# /usr/local/hadoop/sbin/start-dfs.sh

下线完成。。。。。。。。。。

NFS网关
node4
groupadd -g 500 nsd180
useradd -u 500 -g 500 nsd1804

nn01
groupadd -g 500 nsd180
useradd -u 500 -g 500 nsd1804
vim /etc/hosts
192.168.1.14 node4
for i in {1..3};do scp /etc/hosts node i:/etc/;done/usr/local/hadoop/sbin/stop−all.shvim/usr/local/hadoop/etc/hadoop/core−site.xmlhadoop.proxyuser.nsd1804.groups∗hadoop.proxyuser.nsd1804.hosts∗foriin1..4;doscp/usr/local/hadoop/etc/hadoop/core−site.xmlnode i : / e t c / ; d o n e / u s r / l o c a l / h a d o o p / s b i n / s t o p − a l l . s h v i m / u s r / l o c a l / h a d o o p / e t c / h a d o o p / c o r e − s i t e . x m l h a d o o p . p r o x y u s e r . n s d 1804. g r o u p s ∗ h a d o o p . p r o x y u s e r . n s d 1804. h o s t s ∗ f o r i i n 1..4 ; d o s c p / u s r / l o c a l / h a d o o p / e t c / h a d o o p / c o r e − s i t e . x m l n o d e i:/usr/local/hadoop/etc/hadoop/ ;done
/usr/local/hadoop/sbin/start-dfs.sh
/usr/local/hadoop/bin/hdfs dfsadmin-report

node4
vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

nfs.exports.allowed.hosts
* rw


nfs.dump.dir
/var/nfstmp

NFSGW: 创建转储目录，并给用户 nsd1804 赋权
mkdir /var/nfstmp
chown nsd1804:nsd1804 /var/nfstmp

NFSGW: 给 /usr/local/hadoop/logs 赋权
setfacl -m u:nsd1804:rwx

创建数据根目录 /var/hadoop
mkdir /var/hadoop

必须用 root 启动，必须先启动 ！！！
./sbin/hadoop-daemon.sh –script ./bin/hdfs start portmap

必须用代理用户启动，必须后启动 ！！！
./sbin/hadoop-daemon.sh –script ./bin/hdfs start nfs3

node5(新建，)
Client: 安装 nfs-utils
查看注册服务
rpcinfo -p 192.168.1.14
查看共享目录
showmount -e 192.168.1.14

mount 共享目录
mount -t nfs -o vers=3,proto=tcp,nolock,noatime,sync,noacl 192.168.1.14:/ /mnt/
查看
ls /mnt

192.168.1.10 nn01 #namenode,secondarynamenode,ResourceManager
192.168.1.11 node1 #datanode,NodeManager
192.168.1.12 node2 #datanode,NodeManager
192.168.1.13 node3 #datanode,NodeManager