hadoop2.6.0集群复制因子更改(四)

一、查看当前文件副本个数


[hadoop@master hadoop]$ hdfs dfs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:39 /system
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 10:33 /tmp
-rw-r--r--   2 hadoop supergroup          6 2017-03-16 10:33 /tmp/a.txt   《===当前副本为2
[hadoop@master hadoop]$


二、查看hdfs-sizt.xml配置文件


[hadoop@master hadoop]$ cat hdfs-site.xml

  dfs.replication
  2
 



三、检查/目录下文件系统状态


[hadoop@master hadoop]$ hdfs fsck /
Connecting to namenode via http://master:50070
FSCK started by hadoop (auth:SIMPLE) from /192.168.123.10 for path / at Thu Mar 16 23:53:10 EDT 2017
.Status: HEALTHY
 Total size:    6 B
 Total dirs:    3
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 6 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2       《===当前复制因子是2
 Average block replication:     2.0

 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Thu Mar 16 23:53:10 EDT 2017 in 23 milliseconds



The filesystem under path '/' is HEALTHY

[hadoop@master hadoop]$


四、修改hdfs-site.xml文件中的复制因子为3

[hadoop@master hadoop]$ vi hdfs-site.xml 


  dfs.replication
  3
 



五、 再次检查文件系统


[hadoop@master hadoop]$ hdfs fsck /
Connecting to namenode via http://master:50070
FSCK started by hadoop (auth:SIMPLE) from /192.168.123.10 for path / at Thu Mar 16 23:56:20 EDT 2017
.Status: HEALTHY
 Total size:    6 B
 Total dirs:    3
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 6 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2               《====复制因子还是2,但是当我新上传一个文件后,副本是3,以下有实验
 Average block replication:     2.0

 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Thu Mar 16 23:56:20 EDT 2017 in 3 milliseconds


The filesystem under path '/' is HEALTHY


六、 新上传一个文件


[hadoop@master ~]$ hdfs dfs -put b.txt /tmp
[hadoop@master ~]$ hdfs dfs -lsr /

lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:39 /system
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:57 /tmp
-rw-r--r--   2 hadoop supergroup          6 2017-03-16 10:33 /tmp/a.txt     《==修改复制因子之前的旧文件,还是2,需要手工还成3
-rw-r--r--   3 hadoop supergroup         21 2017-03-16 23:57 /tmp/b.txt   《==新上传b.txt,副本是3



七、检查文件系统

[hadoop@master ~]$ hdfs fsck /
Connecting to namenode via http://master:50070
FSCK started by hadoop (auth:SIMPLE) from /192.168.123.10 for path / at Thu Mar 16 23:59:09 EDT 2017
..Status: HEALTHY
 Total size:    27 B
 Total dirs:    3
 Total files:   2
 Total symlinks:                0
 Total blocks (validated):      2 (avg. block size 13 B)
 Minimally replicated blocks:   2 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2                   《===重启后复制因子会变成3
 Average block replication:     2.5            《===这里变为2.5了

 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Thu Mar 16 23:59:09 EDT 2017 in 4 milliseconds


The filesystem under path '/' is HEALTHY
[hadoop@master ~]$



八、 重启hdfs,检查文件系统及复制因子


[hadoop@master hadoop]$ stop-dfs.sh 
Stopping namenodes on [master]
master: stopping namenode
slave2: stopping datanode
slave1: stopping datanode
Stopping secondary namenodes [master]
master: stopping secondarynamenode
[hadoop@master hadoop]$ 

[hadoop@master ~]$ start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-namenode-master.out
hadoop04: starting datanode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-hadoop04.out
slave1: starting datanode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-slave1.out
slave2: starting datanode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-slave2.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /opt/hadoop-2.6.0/logs/hadoop-hadoop-secondarynamenode-master.out
[hadoop@master ~]$ 
[hadoop@master ~]$ jps
5552 NameNode
5743 SecondaryNameNode
5895 Jps
3757 ResourceManager
[hadoop@master ~]$ hdfs fsck /
Connecting to namenode via http://master:50070
FSCK started by hadoop (auth:SIMPLE) from /192.168.123.10 for path / at Fri Mar 17 01:51:01 EDT 2017
..Status: HEALTHY
 Total size:    27 B
 Total dirs:    3
 Total files:   2
 Total symlinks:                0
 Total blocks (validated):      2 (avg. block size 13 B)
 Minimally replicated blocks:   2 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3                《===hdfs重启后复制因子变为3
 Average block replication:     2.5

 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Mar 17 01:51:01 EDT 2017 in 19 milliseconds



The filesystem under path '/' is HEALTHY
[hadoop@master ~]$


九、手工调整副本数


[hadoop@master ~]$ hdfs dfs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:39 /system
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:57 /tmp
-rw-r--r--   2 hadoop supergroup          6 2017-03-16 10:33 /tmp/a.txt  《===手工改之前
-rw-r--r--   3 hadoop supergroup         21 2017-03-16 23:57 /tmp/b.txt
[hadoop@master ~]$ hadoop fs -setrep -w 3 -R /tmp 
setrep: `-R': No such file or directory
Replication 3 set: /tmp/a.txt
Replication 3 set: /tmp/b.txt
Waiting for /tmp/a.txt .... done
Waiting for /tmp/b.txt ... done
[hadoop@master ~]$ hdfs dfs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:39 /system
drwxr-xr-x   - hadoop supergroup          0 2017-03-16 23:57 /tmp
-rw-r--r--   3 hadoop supergroup          6 2017-03-16 10:33 /tmp/a.txt  《===手工改之后
-rw-r--r--   3 hadoop supergroup         21 2017-03-16 23:57 /tmp/b.txt

[hadoop@master ~]$

注意:在本次实验中就改了dfs.replication的值为3后,只对新上传的文件起作用,对于旧的文件不起作用,需要手工更改

你可能感兴趣的:(hadoop)