一、HADOOP安装配置步骤
1.1、服务器的准备
四台Redhat服务器
192.168.130.170 master 192.168.130.168 dd1 192.168.130.162 dd2 192.168.130.248 dd31.2、安装和配置JDK环境
[hadoop@master conf]$ java -version java version “1.6.0_37″ Java(TM) SE Runtime Environment (build 1.6.0_37-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)1.2.2、设置环境变量
vi /etc/profile export JAVA_HOME=/usr/java/jdk1.6.0_26 export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin export JAVA_HOME CLASSPATH PATH1.2.3、编辑完/etc/profile文件后,保存好之后还需输入下面这个命令使其马上生效
其他服务器之间以此类推
1.4.7、验证各服务器是否能正常ssh通讯
ssh dd1(第一次会提示是否继续连接,输入yes即可),直接可以登录到对应的节点上说明成功。
1.5、编辑/etc/hosts文件
在四台服务器的hosts文件中添加各服务器的主机名。
192.168.130.170 master 192.168.130.168 dd1 192.168.130.162 dd2 192.168.130.248 dd31.6、配置hadoop
export JAVA_HOME=/usr/java/jdk1.6.0_26(2)配置conf/core-site.xml文件
<configuration> <property> <name>fs.default.name</name> <value>hdfs://master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/data/zqhadoop/data</value> </property> </configuration>(3)配置conf/hdfs-site.xml文件
<configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.name.dir</name> <value>/home/zqhadoop/HDFS/Namenode</value> </property> <property> <name>dfs.data.dir</name> <value>/home/zqhadoop/HDFS/Datanode</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </property> </configuration>(4)配置conf/mapred-site.xml文件
<configuration> <property> <name>mapred.job.tracker</name> <value>master:9001</value> </property> </configuration>(5)配置masters和slaves文件
masterslaves:
dd1 dd2 dd31.6.3、使用scp命令将配置好的hadoop拷贝到其他服务器上
export HADOOP_HOME=/home/hadoop/hadoop export PATH=$PATH:$HADOOP_HOME/bin1.6.5、格式化hadoop namenode
Configured Capacity: 95862853632 (89.28 GB)
Present Capacity: 83737403392 (77.99 GB)
DFS Remaining: 83736358912 (77.99 GB)
DFS Used: 1044480 (1020 KB)
DFS Used%: 0%
Under replicated blocks: 1
Blocks with corrupt replicas: 0
Missing blocks: 0
————————————————-
Datanodes available: 3 (3 total, 0 dead)
Name: 192.168.130.248:50010
Decommission Status : Normal
Configured Capacity: 31954284544 (29.76 GB)
DFS Used: 348160 (340 KB)
Non DFS Used: 3638841344 (3.39 GB)
DFS Remaining: 28315095040(26.37 GB)
DFS Used%: 0%
DFS Remaining%: 88.61%
Last contact: Tue Feb 19 10:42:48 CST 2013
Name: 192.168.130.168:50010
Decommission Status : Normal
Configured Capacity: 31954284544 (29.76 GB)
DFS Used: 348160 (340 KB)
Non DFS Used: 3718184960 (3.46 GB)
DFS Remaining: 28235751424(26.3 GB)
DFS Used%: 0%
DFS Remaining%: 88.36%
Last contact: Tue Feb 19 10:42:49 CST 2013
Name: 192.168.130.162:50010
Decommission Status : Normal
Configured Capacity: 31954284544 (29.76 GB)
DFS Used: 348160 (340 KB)
Non DFS Used: 4768423936 (4.44 GB)
DFS Remaining: 27185512448(25.32 GB)
DFS Used%: 0%
DFS Remaining%: 85.08%
Last contact: Tue Feb 19 10:42:48 CST 2013
2.3、使用web访问验证hadoop
通过访问web界面访问hadoop的群集概况和数据节点概况, 可以通过以下两种方式进行访问, 其中master-IP是指命名主机的IP地址。
https://master-IP:50030 (查看命名空间情况)
https://master-IP:50070 (查看整个分布式文件系统的状态,浏览分布式文件系统中的文件以及 log)
三、zookeeper安装配置步骤:
3.1、解压zookeeper压缩包
tar -zxvf zookeeper-3.4.5.tar.gz
mv zookeeper-3.4.5 zookeeper
3.2、配置zoo.cfg文件
进入zookeeper/conf目录,执行以下命令
cp zoo_sample.cfg zoo.cfg
修改zoo.cfg中的配置如下:
dataDir=/home/zqhadoop/zookeeper_data dataLogDir=/home/zqhadoop/zookeeper_log clientPort=2181 initLimit=10 syncLimit=5 tickTime=2000 server.1=192.168.130.170:2888:3888 server.2=192.168.130.168:2888:3888 server.3=192.168.130.162:2888:3888 server.4=192.168.130.248:2888:38883.3、创建dataDir目录(这里指的是“/home/zqhadoop/zookeeper_data”),且在该目录下创建名为myid的文件
scp -r zookeeper_data hadoop@dd1:~/
scp -r zookeeper_data hadoop@dd2:~/
scp -r zookeeper_data hadoop@dd3:~/
依照zoo.cfg修改zookeeper_data 目录中myid的值
3.5、启动zookeeper
单独启动的zookeeper需要分别在每台服务器上执行以下命令
~/home/hadoop/zookeeper/bin/zkServer.sh start
四、验证zookeeper安装
4.1、使用jps检查服务是否启动
[hadoop@master conf]$ jps
763 SecondaryNameNode
866 JobTracker
9203 Jps
1382 QuorumPeerMain
554 NameNode
可以看到jps会多出这样一个进程QuorumPeerMain
4.2、通过输入“sh /jz/zookeeper-3.3.1/bin/zkServer.sh status”检查是否启动
[zqhadoop@master bin]$ ./zkServer.sh status
JMX enabled by default
Using config: /home/zqhadoop/zookeeper/bin/../conf/zoo.cfg
Mode: follower
五、HBASE安装配置
5.1、解压hbase安装包
tar -zxvf hbase-0.94.5.tar.gz
mv hbase-0.94.5 hbase
5.2、配置hbase配置文件
(1)设置conf/hbase-site.xml文件
<configuration> <property> <name>hbase.rootdir</name> <value>hdfs://master:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>dd1,dd2,dd3</value> </property> <property> <name>hbase.zookeeper.session.timeout</name> <value>60000</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> <property> <name>hbase.master</name> <value>master</value> </property> <property> <name>hbase.regionserver.lease.period</name> <value>60000</value> </property> <property> <name>hbase.rpc.timeout</name> <value>60000</value> </property> <property> <name>hbase.master.maxclockskew</name> <value>180000</value> </property> </configuration>
dd1 dd2 dd3(3)配置conf/hbase-env.xml文件
export JAVA_HOME=/usr/java/jdk1.6.0_37 export HBASE_MANAGES_ZK=flase(4)配置hadoop中的hdfs-site.xml,添加配置
<property> <span style="white-space:pre"> </span><name>dfs.datanode.max.xcievers</name> <span style="white-space:pre"> </span><value>4096</value> </property>该参数限制了datanode所允许同时执行的发送和接受任务的数量,缺省为256,hadoop-defaults.xml中通常不设置这个参数。这个限制看来实际有些偏小,高负载下,DFSClient 在put数据的时候会报 could not read from stream 的 Exception。
错误记录:
情况1:
2012-12-19 22:11:46,018 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section ‘Client’ could not
be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-12-19 22:11:46,024 WARN org.apache.zookeeper.ClientCnxn: Session 0×0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
解决方法:
修改/etc/hosts
注释掉加粗的一行
<strong>#127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 </strong> ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.130.170 master 192.168.130.168 dd1 192.168.130.162 dd2 192.168.130.248 dd3 192.168.130.164 dd4六、验证hbase运行
hbase(main):001:0>
6.3、通过web验证hbase是否正常运行