Hadoop群集环境搭建-- 伪分布式模式

Hadoop群集环境搭建

伪分布式模式

介绍

操作系统：redhat5.6

NameNode：           172.16.40.180

DateNode:            172.16.40.201

DateNode:             172.16.40.108

software:

JDK version:  jdk1.6.0_27

Download  address:

http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html

Hadoop version: hadoop-0.20.2

Download  address:

http://download.filehat.com/apache/hadoop/core/hadoop-0.20.2/

JDK

[root@dlxa180 tools] # chmod +x jdk-6u27-linux-x64.bin

[root@dlxa180 tools] #./jdk-6u27-linux-x64.bin

[root@dlxa201 tools]#rm –rf  jdk-6u27-linux-x64.bin

[root@dlxa180 tools]#vi  /etc/profile

 

export JAVA_HOME=/home/tools/jdk

export JAVA_BIN=/home/tools/jdk/bin

export HADOOP_HOME=/usr/local/hadoop-0.20.3-dev

export ANT_HOME=/home/tools/apache-ant-1.8.2

PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$ANT_HOME/bin:$PATH

 

[root@dlxa180 tools] ## java -version

java version "1.6.0_27"

Java(TM) SE Runtime Environment (build 1.6.0_27-b07)

Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)

说明：JDK安装参考，环境变量设置等

    

 

HOSTS and IPTABLES

 

[root@dlxa180 tools] #vi /etc/hosts

 

说明：将各个节点都需要追加上去

[root@dlxa180 tools] # service  iptables  stop

[root@dlxa180 tools] #chkconfig  iptables  off

SSH

 

[root@dlxa180 ~] #ssh-keygen -t rsa                                   

[root@dlxa180  .ssh]#cat id_rsa.pub >> authorized_keys   

      // 权限设置

[root@dlxa180 .ssh]#chmod 644 authorized_keys              

 //将 authorized_keys复制到其他远程机器上面 ,

 [root@dlxa180 .ssh]# scp authorized_keys 172.16.40.108:/root/.ssh       

说明：

namenode访问所有数据节点无密码，namenode访问自身无密码。

Hadoop

[root@dlxa180 tools] #tar –zxvf  hadoop-0.20.2.tar.gz

[root@dlxa180 tools] #rm –rf  hadoop-0.20.2.tar.gz

[root@dlxa180 tools] # vi  /etc/profile

export JAVA_HOME=/home/tools/jdk

export JAVA_BIN=/home/tools/jdk/bin

export HADOOP_HOME=/usr/local/hadoop-0.20.3-dev

export ANT_HOME=/home/tools/apache-ant-1.8.2

PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$ANT_HOME/bin:$PATH

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME JAVA_BIN ANT_HOME PATH CLASSPATH

[root@dlxa180 tools] # source /etc/profile

 

hadoop/conf

hadoop-env.sh

#追加

Export JAVA_HOME=/home/tools/jdk

Mastters

 

172.16.40.180

Slaves：

 

172.16.40.108

172.16.40.201 

core-site.xml：

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://172.16.40.180:54310</value>

<description>The name of the default file system.  A URI whose

  scheme and authority determine the FileSystem implementation.  The

  uri's scheme determines the config property (fs.SCHEME.impl) naming

  the FileSystem implementation class.  The uri's authority is used to

  determine the host, port, etc. for a filesystem.</description>

</property>

<property>

<name> dfs.replication</name>

<value>1</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/HadoopTemp</value>

</property>

</configuration>

hdfs-site.xml:

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

 <description>Default block replication.

  The actual number of replications can be specified when the file is created.

  The default is used if replication is not specified in create time.

  </description>

</property>

</configuration>

Start hadoop

Master: 首先格式化在启动

[root@dlxa180 ~]#bin/hadoop namenode -format

[root@dlxa180 ~]#bin/start-all.sh

站在大象的背上说hello！

HDFS

显示文件

[root@dlxa180 hadoop-0.20.3-dev]#bin/hadoop dfs -ls

Found 8 items

drwxr-xr-x   - root supergroup          0 2011-11-09 13:55 /user/root/aa

…

drwxr-xr-x   - root supergroup          0 2011-11-09 14:02 /user/root/wbk

 

上传文件到HDFS

 

[root@dlxa180 hadoop-0.20.3-dev]# echo "hello my hadoop hdfs" > test1

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfs -put test1 test

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfs -ls test

Found 1 items

-rw-r--r--   1 root supergroup         21 2011-11-09 14:12 /user/root/test

 

将HDFS上的文件复制到本地

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfs -get test getin

[root@dlxa180 hadoop-0.20.3-dev]# cat getin

hello my hadoop hdfs

 

删除HDFS上的文件

 

 

查看HDFS下的某个文件

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfs -cat aa/*

hello word

hello hadoop

管理与更新

#查看HDFS的基本信息

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfsadmin –report

#离开安全模式

[root@dlxa180 hadoop-0.20.3-dev]#bin/hadoop dfsadmin -safemode leave 

#进入安全模式

[root@dlxa180 hadoop-0.20.3-dev]#bin/hadoop dfsadmin –safemode enter

 

添加节点

可以将namenode的hadoop目录复制到新的数据节点上，并修改master和slave

负载均衡：

[root@dlxa180 hadoop-0.20.3-dev]#bin/start-balancer.sh

starting balancer, logging to /usr/local/hadoop-0.20.3-dev/logs/hadoop-root-balancer-dlxa180.out

Time Stamp        Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved

The cluster is balanced. Exiting...

Balancing took 273.0 milliseconds

MapReduce

Wordcount

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfs -put put in

[root@dlxa180 hadoop-0.20.3-dev]#bin/hadoop jar hadoop-0.20.2-examples.jar wordcount in out

11/11/09 14:40:54 INFO input.FileInputFormat: Total input paths to process : 2

11/11/09 14:40:55 INFO mapred.JobClient: Running job: job_201111091357_0006

11/11/09 14:40:56 INFO mapred.JobClient:  map 0% reduce 0%

11/11/09 14:41:04 INFO mapred.JobClient:  map 100% reduce 0%

11/11/09 14:41:16 INFO mapred.JobClient:  map 100% reduce 100%

11/11/09 14:41:18 INFO mapred.JobClient: Job complete: job_201111091357_0006

11/11/09 14:41:18 INFO mapred.JobClient: Counters: 18

11/11/09 14:41:18 INFO mapred.JobClient:   Job Counters

11/11/09 14:41:18 INFO mapred.JobClient:     Launched reduce tasks=1

11/11/09 14:41:18 INFO mapred.JobClient:     Rack-local map tasks=1

11/11/09 14:41:18 INFO mapred.JobClient:     Launched map tasks=2

11/11/09 14:41:18 INFO mapred.JobClient:     Data-local map tasks=1

11/11/09 14:41:18 INFO mapred.JobClient:   FileSystemCounters

11/11/09 14:41:18 INFO mapred.JobClient:     FILE_BYTES_READ=54

11/11/09 14:41:18 INFO mapred.JobClient:     HDFS_BYTES_READ=24

11/11/09 14:41:18 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=178

11/11/09 14:41:18 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=24

11/11/09 14:41:18 INFO mapred.JobClient:   Map-Reduce Framework

11/11/09 14:41:18 INFO mapred.JobClient:     Reduce input groups=3

11/11/09 14:41:18 INFO mapred.JobClient:     Combine output records=4

11/11/09 14:41:18 INFO mapred.JobClient:     Map input records=2

11/11/09 14:41:18 INFO mapred.JobClient:     Reduce shuffle bytes=60

11/11/09 14:41:18 INFO mapred.JobClient:     Reduce output records=3

11/11/09 14:41:18 INFO mapred.JobClient:     Spilled Records=8

11/11/09 14:41:18 INFO mapred.JobClient:     Map output bytes=40

11/11/09 14:41:18 INFO mapred.JobClient:     Combine input records=4

11/11/09 14:41:18 INFO mapred.JobClient:     Map output records=4

11/11/09 14:41:18 INFO mapred.JobClient:     Reduce input records=4

 

[root@dlxa180 hadoop-0.20.3-dev]# bin/hadoop dfs -get out output

[root@dlxa180 hadoop-0.20.3-dev]# cat output/*

cat: output/_logs: Is a directory

hadoop  1

hello   2

word    1