Hadoop 离线安装 CDH5.1 第一章:离线环境部署
配置主机映射关系 [hadoop@slave03 ~]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 10.15.5.200 master.hadoop 10.15.5.201 slave01.hadoop 10.15.5.202 slave02.hadoop 10.15.5.203 slave03.hadoop
如上有4台主机，每台hosts都已经编辑过。所有的主机都已经编辑过/etc/ssh/sshd_config
RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile .ssh/authorized_keys
所有的authorized_keys都已经过hadoop用户ssh-keygen -t rsa生成过空密码，并写入成功如下
[hadoop@slave03 ~]$ cat .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsZbjTNyyW4ZPZ2cCbUk/m3F5LGTsLKzv/TzuG0LJZ9E/UN2dV4j7dhHbN1olbsvFp4oyGGKhFxn036PljJRfEWhdY0dDagWklH3oijRYr+UzBTMf8sIR9FxKD540rhC0NCEP+KaNZXAhndB7BTZTIkytMlfCboOcbRjt1XU+3Yhi7Dlp6Z6cu2oSgYo4iYpX1anHUMRuMLl5/mQ1INeoRMerBTzGfIYAIQF5Du2bE9HRSClyUal2O6QkKhZGHRa0+3jjFY/vER0R+ruLJZLVfQQkJffQJEz1qk4e/V2GEg5ScjAPyojKg+R/mpWNAwivbsJ8pe4YWnUxBNQFphB/7w== [email protected] ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwk3DPdzRD41+saaLwnqLkMnLVf5m5br2hb6lCNDF+3vEiezIKpoxp2lzqOp7kUOMrubcJRVcdXJWgrzbp bmUzkqF1YyaXqZ1OaIR+by20glTnYemkTE3ljknVgrQXXyVUxYDw8TGx3sJ/sw5EgrUEM3xB98Yz9j9JUL7lkGzahjYO02JcaKU83YfqUjCt9QIhnLiIo 7BPs/NVK6UP6Srn0fCvsVXS2XqkHzjPhIjiQAiyDINMz3ZCHxnFsOQm9I//R61YYrHLfNPQKOnBMRV4/Q2FPGXa5wgx022i3VaUiXOSGlTV2ugoC22Q1n 422q9P3kWevhL4nDGizn5rEOCEQ== [email protected] ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3FI6rK31HE5MGpqQS/60w8lHruIHSomjdWhmiN+FXKoTyZzqHwxyRS7j1Y8mplIpoUzAOlfPWbgm2AQto jFbLJfOtN4vvBkniw6OEcvjojyOQdyHsEr907L9K9GRq0VtFkjo/YxvYQf823I+14sOdZ8fCZrue0NmcJ0RMifLHrmMGeJ+LLN5fyWYygvDQIenF/DWNN KNgp15v/eVVaRokJ00sSooeikJlv7DL7EWK8UBc0M6Fum7Y2L4WXVmjavAoG5jxN2tJXQvwgbzFaJ1OWFzZzwsOlI+khVvb13ARGnSd0Lp1PDQ0V64Jyh Vj70TllSgvrhz2vkMFbe82TJM/w== [email protected] ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA20RPILfJJpN7QsiiGfGaKmcibuwDexslHob9CmzJUFJFxm8sMXtvOMuuipknCl4Aya5+U3AjO5xPh5ZcD rCMSgUrgerwi9ofF/2Od9efOCd4JMKV0V/nsAHdUtCxCeBGOyaPdZ9rEnEqKtu3dF8fW2zOl3UGJ3GrzgTeAvG1Rlet/+jL4RJ/ob+CzVwX3pZQ5YGrwU sPbQ70Sn4aZSM56VTjx9QaqWlXxQDiHgyI+FN2OCsfqB9kiDHSu3DZ2Jjyil+7kOeCUuoCOXZqjG3VUGURYVtguEIlRuHEOQV9hWc7zFVADt5AKpi0c4q i3Bo1P3gYVJ3/wYBqK8YrrHT8Jw== [email protected]
给每台主机安装Java, 如果已有Java 请先卸载，从oracle网站得到链接如下:
http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz
下载该文件到本地FTP，因为直接wget会被oracle拒绝
[hadoop@slave03 usr]$ cd java [hadoop@slave03 java]$ pwd /usr/java ftp> bin 200 Type set to I. ftp> get jdk-7u67-linux-x64.tar.gz local: jdk-7u67-linux-x64.tar.gz remote: jdk-7u67-linux-x64.tar.gz 227 Entering Passive Mode (10,15,15,63,19,48). 150 Opening data connection for jdk-7u67-linux-x64.tar.gz.
解压数据包到/usr/java目录下
[hadoop@slave03 java]$ sudo tar -zxvf jdk-7u67-linux-x64.tar.gz
修改系统变量，指定java路径，通过source命令 应用该配置,验证Java版本
[hadoop@slave03 jdk1.7.0_67]$ sudo vim /etc/profile # JAVA environment export JAVA_HOME=/usr/java/jdk1.7.0_67 export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin [hadoop@slave03 jdk1.7.0_67]$ source /etc/profile [hadoop@slave03 jdk1.7.0_67]$ java -version java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
把安装文件分发到其他主机,参照以上步骤，安装java并配置变量并验证。
[hadoop@slave03 java]$ scp jdk-7u67-linux-x64.tar.gz [email protected]:~/ [hadoop@slave03 java]$ scp jdk-7u67-linux-x64.tar.gz [email protected]:~/ [hadoop@slave03 java]$ scp jdk-7u67-linux-x64.tar.gz [email protected]:~/
配置NTP服务：集群的时间必须保持一致，否则同步会有问题。所以需要配置NTP服务保证集群正常运行，将master外联到ntp服务器，其他主机和master同步，或者在企业网中搭建专用的NTP服务器都可以,可以使用210.72.145.44 (国家授时中心服务器IP地址)
[hadoop@master ~]$ sudo ntpdate -u 65.55.56.206 21 Aug 23:30:04 ntpdate[2355]: step time server 65.55.56.206 offset 455.994813 sec
看到硬件时间和系统时间不一样，将时间写入硬件(执行同步后，不会自动写).
[hadoop@master ~]$ sudo hwclock -r Thu 21 Aug 2014 11:25:07 PM CST -0.736104 seconds [hadoop@master ~]$ sudo date Thu Aug 21 23:33:01 CST 2014 [hadoop@master ~]$ sudo hwclock -w [hadoop@master ~]$ sudo hwclock -r Thu 21 Aug 2014 11:34:36 PM CST -0.053046 seconds
配置NTP Server参数,默认的不用动 修改和添加下面两行即可，最终验证如下
[hadoop@master ~]$ sudo vim /etc/ntp.conf restrict default nomodify notra server 65.55.56.206 prefer [hadoop@master ~]$ sudo service ntpd start Starting ntpd: [ OK ] [hadoop@master ~]$ sudo chkconfig ntpd on [hadoop@master ~]$ sudo ntpstat synchronised to NTP server (202.112.31.197) at stratum 3 time correct to within 1053 ms polling server every 64 s
如下，将所有的datanode 配置NTP Client ，并同步 然后写入硬件时钟
[hadoop@slave01 ~]$ sudo vim /etc/ntp.conf server 10.15.5.200 [hadoop@slave03 ~]$ sudo ntpdate -u 10.15.5.200 22 Aug 00:00:03 ntpdate[2404]: step time server 10.15.5.200 offset 4964.547679 sec [hadoop@slave03 ~]$ sudo hwclock -r Thu 21 Aug 2014 10:37:35 PM CST -0.447982 seconds [hadoop@slave03 ~]$ sudo hwclock -w [hadoop@slave03 ~]$ sudo hwclock -r Fri 22 Aug 2014 12:00:30 AM CST -0.248008 seconds
安装mysql，这里参考了一个日本人的博客，使用日本那边大学的源：
http://changineer.info/server/mysql/mysql_installation_linux.html
[hadoop@master ~]$ sudo yum install http://ftp.jaist.ac.jp/pub/mysql/Downloads/MySQL-5.6/MySQL-server-5.6.20-1.el6.x86_64.rpm
出现了错误
file /usr/share/mysql/czech/errmsg.sys from install of MySQL-server-5.6.20-1.el6.x86_64 conflicts with file from package mysql-libs-5.1.73-3.el6_5.x86_64 #解决办法也是参照日本人的博客来做的移除掉mysql-libs然后重新yum安装# [hadoop@master ~]$ sudo yum remove mysql-libs [hadoop@master ~]$ sudo yum install mysql/Downloads/MySQL-5.6/MySQL-server-5.6.20-1.el6.x86_64.rpm Installed: MySQL-server.x86_64 0:5.6.20-1.el6 Complete!
启动mysql ,设置为开机启动（需要安装client来连接）
[hadoop@master ~]$ sudo service mysql start Starting MySQL.. SUCCESS! [hadoop@master ~]$ sudo chkconfig mysql on
MySQL使用随机密码，通过cat主目录下文件可以得到,进入mysql界面,修改root密码为hadoop
[root@master hadoop]# cat ~/.mysql_secret
# The random password set for the root user at Fri Aug 22 13:55:10 2014 (local time): TPQKIgffMJGcFfFG
[root@master hadoop]# mysql -u root -p
#复制到这里#
Enter password:
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> update user set password=password("hadoop") where user='root';
Query OK, 4 rows affected (0.00 sec)
Rows matched: 4 Changed: 4 Warnings: 0
建立数据库，分配库权限。
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | +--------------------+ 4 rows in set (0.00 sec) mysql> create databse amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | amon | | hive | | hue | | mysql | | performance_schema | | test | +--------------------+ 7 rows in set (0.00 sec) mysql> grant all on *.* to 'root'@'%' identified by 'hadoop' with grant option; Query OK, 0 rows affected (0.00 sec)
下载离线的cloudera manager安装包, 放置于/opt目录，并解压，改名
[hadoop@master opt]$ sudo tar -xzvf cloudera-manager-el5-cm5.1.0_x86_64.tar.gz [hadoop@master opt]$ sudo mv cm-5.1.0 cloudera-manager-5.1.0
本文出自 “The old artisan” 博客，谢绝转载！