hadoop cdh离线安装

转至元数据起始Page View StatisticsHelpClosehadoop集群安装总结准备环境  192.168.249.3   hadoop1192.168.249.4   hadoop2192.168.249.5   hadoop3192.168.249.6   hadoop4
免密登陆
在所有的服务器上运行ssh-keygen，然后一路回车。用xshell给每台机器上执行登陆hadoop1cat  /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keya没有authorized_keya文件的话自己创建即可把每台服务器 /root/.ssh/下的id_rsa.pub拷贝到hadoop1的服务器上并追加到authorized_keya中然后scp /root/.ssh/authorized_keya [email protected]:/root/.ssh/scp /root/.ssh/authorized_keya [email protected]:/root/.ssh/scp /root/.ssh/authorized_keya [email protected]:/root/.ssh/
修改所有主机hosts
echo 192.168.249.3 hadoop1 >> /etc/hostsecho 192.168.249.4 hadoop2 >> /etc/hostsecho 192.168.249.5 hadoop3 >> /etc/hostsecho 192.168.249.6 hadoop4 >> /etc/hostsscp /etc/hosts [email protected]:/etc/scp /etc/hosts [email protected]:/etc/scp /etc/hosts [email protected]:/etc/
关闭并禁用防火墙systemctl stop firewalldsystemctl disable firewalld.service
设置参数优化  每台都要做vim vi /etc/sysctl.confvm.swappiness = 0net.ipv6.conf.all.disable_ipv6 = 1net.ipv6.conf.default.disable_ipv6 = 1net.ipv4.ip_local_port_range = 1024 65000:wq 保存退出   vim /etc/rc.localecho never > /sys/kernel/mm/transparent_hugepage/defragecho never > /sys/kernel/mm/transparent_hugepage/enabled：wq 保存退出source /etc/rc.localchmod +x /etc/rc.d/rc.local   这个命令不能少，否则重启后/etc/rc.local不能自动执行reboot重启所有服务器验证参数修改getenforce   检查结果是否是Disabledfirewall-cmd --state 检查防火墙是否关闭cat /sys/kernel/mm/transparent_hugepage/defrag 看是否是选中了never。cat /proc/sys/vm/swappiness  看结果是否是设置的0，如果设置的是10，则结果显示是10。注意：如果centos是在vmware虚拟机安装的，这个参数可能会改不了，一直是30。物理机的centos没有问题。安装后面需要的一些工具  yum install wget -yyum install deltarpm -yyum install psmisc -y   http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html下载jdk 需要oracle的帐号系统64位的就下64位的  安装前检查本机是否安装有别的版本jdk 有的话卸载重新安装检查  rpm -qa | grep java卸载  rpm -e --nodeps安装  rpm -ivh jdk-8u231-linux-x64.rpm添加变量vim /etc/profile/export JAVA_HOME=/usr/java/defaultexport PATH=$JAV{A}_{H}OME/bin:$PATHexport CLASSPATH=.:$JAV{A}_{H}OME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:wq保存退出执行source /etc/profile检查java是否安装成功echo $JAVA_HOME查看版本java -version安装数据库测试环境我这就直接yum安装了  yum install mariadb mariadb-server -y
systemctl enable mariadb.service 设置开机启动
systemctl start mariadb.service 启动数据库
mysql_secure_installation  设置密码如果不能安装先把本机带的数据库卸载优化数据库vim /etc/my.cnf[mysqld]transaction-isolation = READ-COMMITTED# Disabling symbolic-links is recommended to prevent assorted security risks;# to do so, uncomment this line:# symbolic-links = 0key_buffer = 16Mkey_buffer_size = 32Mmax_allowed_packet = 32Mthread_stack = 256Kthread_cache_size = 64query_cache_limit = 8Mquery_cache_size = 64Mquery_cache_type = 1max_connections = 550#expire_logs_days = 10#max_binlog_size = 100M#log_bin should be on a disk with enough free space. Replace ‘/var/lib/mysql/mysql_binary_log’ with an appropriate path for your system#and chown the specified folder to the mysql user.log_bin=/var/lib/mysql/mysql_binary_logbinlog_format = mixedread_buffer_size = 2Mread_rnd_buffer_size = 16Msort_buffer_size = 8Mjoin_buffer_size = 8M# InnoDB settingsinnodb_file_per_table = 1innodb_flush_log_at_trx_commit  = 2innodb_log_buffer_size = 64Minnodb_buffer_pool_size = 4Ginnodb_thread_concurrency = 8innodb_flush_method = O_DIRECTinnodb_log_file_size = 512M[mysqld_safe]log-error=/var/log/mariadb/mariadb.logpid-file=/var/run/mariadb/mariadb.pid:wq保存退出优化参数根据自己服务器配置进行修改优化后重启数据库systemctl restart mariadb.service为后面的服务创建数据登陆数据库mysql -uroot -p
create database amon DEFAULT CHARACTER SET utf8;create database rman DEFAULT CHARACTER SET utf8;create database hive DEFAULT CHARACTER SET utf8;create database sentry DEFAULT CHARACTER SET utf8;create database nav DEFAULT CHARACTER SET utf8;create database navms DEFAULT CHARACTER SET utf8;create database oozie DEFAULT CHARACTER SET utf8;create database hue DEFAULT CHARACTER SET utf8;授权用户grant all on amon.* TO amon@’%’ IDENTIFIED BY ‘amon_password’;grant all on rman.* TO rman@’%’ IDENTIFIED BY ‘rman_password’;grant all on hive.* TO hive@’%’ IDENTIFIED BY ‘hive_password’;grant all on sentry.* TO sentry@’%’ IDENTIFIED BY ‘sentry_password’;grant all on nav.* TO nav@’%’ IDENTIFIED BY ‘nav_password’;grant all on navms.* TO navms@’%’ IDENTIFIED BY ‘navms_password’;grant all on oozie.* TO oozie@’%’ IDENTIFIED BY ‘oozie_password’;grant all on hue.* TO hue@’%’ IDENTIFIED BY ‘hue_password’;

安装时间服务器 再hadoop1yum install ntp -y所有服务器  
#在hadoop1节点，注释掉原有的server指向把时间服务器的域名地址指向一个靠谱可连的时间服务器，我选择的是ntp节点，注释掉原有的server指向把时间服务器的域名地址指向一个靠谱可连的时间服务器，我选择的是ntp.api.bz.# vim /etc/ntp.conf    ……     #server 0.centos.pool.ntp.org iburst    #server 1.centos.pool.ntp.org iburst    #server 2.centos.pool.ntp.org iburst    #server 3.centos.pool.ntp.org iburst    server ntp.api.gz iburst…

在其余的节点上，把时间服务器的server指向master namenode节点# vi /etc/ntp.conf     ……     #server 0.centos.pool.ntp.org iburst    #server 1.centos.pool.ntp.org iburst    #server 2.centos.pool.ntp.org iburst    #server 3.centos.pool.ntp.org iburst    server 192.168.249.3 iburst…

在所有节点systemctl start ntpd启动服务systemctl enable ntpd设置开机启动

安装Cloudera Manager Server 软件（都安装）下载地址https://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz（只再主节点进行操作 hadoop1）新建目录mkdir /opt/cloudera-manager上传并解压 Cloudera Manager Server包tar zxvf cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz创建CM Server系统用户账号useradd --system --home=/opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment “Cloudera SCM User” cloudera-scm创建CM Server的本地数据存储目录并更改权限mkdir -p /var/lib/cloudera-scm-serverchown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server配置所有节点上的Cloudera Manager Agentssed -i ‘s/server_host=localhost/server_host=hadoop1/’ /opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-agent/config.ini在主节点初始化数据库下载mysql驱动器 并上传到  /opt/cloudera-manager/cm-5.12.0/share/cmf/lib 所有节点都要上传/opt/cloudera-manager/cm-5.12.0/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -phadoop123 scm scm  hadoop123                                                                                   地址      用户     密码    库名 用户 密码执行会报错 因为scm用户前面我们没有授权给个授权就好grant all on scm.* TO ‘scm’@‘hadoop1’ IDENTIFIED BY ‘hadoop123’;查看配置vim /opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-server/db.propertiescom.cloudera.cmf.db.type=mysqlcom.cloudera.cmf.db.host=hadoop1com.cloudera.cmf.db.name=scmcom.cloudera.cmf.db.user=scmcom.cloudera.cmf.db.setupType=EXTERNALcom.cloudera.cmf.db.password=Hadoop123wq保存退出创建CDH Parcel资源目录在hadoop1mkdir -p /opt/cloudera/parcel-repocd /opt/cloudera/parcel-repo把我们需要的CDHparcel放入此目录下载地址http://archive.cloudera.com/cdh5/parcels/5.12.0/CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcelCDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.shamanifest.json下载时注意系统版本  给目录修改权限chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo在其他节点上创建资源目录mkdir -p /opt/cloudera/parcels更改权限chown cloudera-scm:cloudera-scm /opt/cloudera/parcels在所有节点上 给cloudera-manager更改下权限chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager启动所有节点hadoop1上启动主节点/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-server  starthadoop234上面启动/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-agent  start在浏览器输入192.168.249.3:7180用户名密码默认的都位 admin登陆之后在web页面选择我们的主机 在主机上安装CDH就ok安装完成后选择我们所需要的服务

注意NFS Gateway启动失败在要运行NFS Gateway的服务器上安装：rpcbindyum install rpcbindservice rpcbind startsystemctl enable rpcbind Oozie客户端不能访问少ext/opt/cloudera/parcels/CDH/lib/oozie/libext把ext-2.2.zip 解压缩后放到这个目录。Yarn部署客户端出错部署客户端出错：需要yum install perl -y
查看日志：Yarn启动失败
查看日志：不能创建目录hdfs://hadoop1:8020/user/history/解决办法，手动创建？ hdfshadoop fs -mkdir hdfs://hadoop1:8020/user/history/点开yarn后，有命令操作可以通过CDH的界面创建。
Hive启动失败把mysql-connector-java-5.1.45-bin.jar 拷贝到/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hive/lib创建hive需要的表：点Hive->操作->创建表，这个也是通过CDH的界面创建。Hue启动失败还需要运行命令yum install -y python-lxml创建目录mkdir -p /var/log/hue-httpd/touch /var/log/hue-httpd/error_log安装服务httpd，mod_sslyum install httpd mod_ssl -y安装sasl，否则打开hue站点的时候有错误提示yum install cyrus-sasl-develyum install cyrus-sasl-gssapiyum install krb5-server.x86_64  krb5-devel.x86_64出现报错请查看日志 根据日志信息进行处理