分布式集群离线配置
该配置方法适用于私有yum源部署完成
1、集群划分
一个master，两个node（目前考虑3个服务器）
2、服务器基本配置
- 网络配置
配置网络(root 用户)
Vi /etc/syconfig/network-script/ifcfg-eth0
ONBOOT=yes
BOOTTPROTO=static
IPADDR=192.168.124.100
DNS1=8.8.8.8
NETMASK=255.255.255.0
GWTEWAY=192.168.124.1
service network restart - 创建hadoop用户（root用户）
groupadd hadoop //创建用户组
useradd –m –g hadoop hadoop //新建hadoop用户并创建目录
passwd hadoop //创建密码 密码设置为admin - 关闭防火墙
防火墙配置（root 用户）
查看防火墙状态：service iptables status
关闭防火墙：service iptables stop
永久关闭防火墙：chkconfig iptables off - 配置selinux
配置selinux(root用户)
关闭selinux: vi /etc/selinux/config
SELIUNX=disabled - ssh 免密
参考地址：https://dongkelun.com/2018/04/05/sshConf/
3、配置jdk
创建jdk路径 mkdir /usr/local/jdk（root用户）
从base/java中拷贝jdk-8u181-linux-x64.tar.gz 到/usr/local/jdk下
解压安装： tar –zxvf jdk-8u181-linux-x64.tar.gz
配置环境变量：vim /etc/profile
最后添加：
JAVA_HOME=/usr/local/jdk/jdk1.8.0_181
CLASSPATH=.: J A V A H O M E / l i b / t o o l s . j a r : JAVA_HOME/lib/tools.jar: JAVAHOME/lib/tools.jar:JAVA_HOME/lib/dt.jar
PATH= J A V A H O M E / b i n : JAVA_HOME/bin: JAVAHOME/bin:HOME/bin: H O M E / . l o c a l / b i n : HOME/.local/bin: HOME/.local/bin:PATH
检查是否安装成功：java -version
修改jdk安全机制，vim /usr/local/jdk/jdk1.8.0.181/jre/lib/serurity/java.security
删除jdk.tls.dis….下的3DES_EDE_CBC
4、拷贝yum源
备份yum源 配置文件
cd /etc/yum.repo.d
rename .repo .repo.bak
base/yum.repo 拷贝 ambari.repo,CentOS.repo,hdp.repo 到/etc/yum.repo.d 下
5、安装mabari（类似在线安装）
yum install ambari-server
[root@yum hadoop]# yum install ambari-server
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Loading mirror speeds from cached hostfile
CentOS6.8 | 2.9 kB 00:00
HDP-2.6.3.0 | 2.9 kB 00:00
HDP-2.6.3.0/primary_db | 78 kB 00:00
HDP-UTILS-1.1.0.21 | 2.9 kB 00:00
HDP-UTILS-1.1.0.21/primary_db | 36 kB 00:00
ambari-2.6.0.0 | 2.9 kB 00:00
Resolving Dependencies
--> Running transaction check
---> Package ambari-server.x86_64 0:2.6.0.0-267 will be installed
--> Processing Dependency: postgresql-server >= 8.1 for package: ambari-server-2.6.0.0-267.x86_64
--> Running transaction check
---> Package postgresql-server.x86_64 0:8.4.20-6.el6 will be installed
--> Processing Dependency: postgresql-libs(x86-64) = 8.4.20-6.el6 for package: postgresql-server-8.4.20-6.el6.x86_64
--> Processing Dependency: postgresql(x86-64) = 8.4.20-6.el6 for package: postgresql-server-8.4.20-6.el6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-8.4.20-6.el6.x86_64
--> Running transaction check
---> Package postgresql.x86_64 0:8.4.20-6.el6 will be installed
---> Package postgresql-libs.x86_64 0:8.4.20-6.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================================
Installing:
ambari-server x86_64 2.6.0.0-267 ambari-2.6.0.0 712 M
Installing for dependencies:
postgresql x86_64 8.4.20-6.el6 CentOS6.8 2.6 M
postgresql-libs x86_64 8.4.20-6.el6 CentOS6.8 202 k
postgresql-server x86_64 8.4.20-6.el6 CentOS6.8 3.4 M
Transaction Summary
====================================================================================================================================================================
Install 4 Package(s)
Total size: 719 M
Total download size: 712 M
Installed size: 805 M
Is this ok [y/N]: y
Downloading Packages:
ambari-server-2.6.0.0-267.x86_64.rpm | 712 MB 00:30
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : postgresql-libs-8.4.20-6.el6.x86_64 1/4
Installing : postgresql-8.4.20-6.el6.x86_64 2/4
Installing : postgresql-server-8.4.20-6.el6.x86_64 3/4
Installing : ambari-server-2.6.0.0-267.x86_64 4/4
Verifying : postgresql-server-8.4.20-6.el6.x86_64 1/4
Verifying : ambari-server-2.6.0.0-267.x86_64 2/4
Verifying : postgresql-libs-8.4.20-6.el6.x86_64 3/4
Verifying : postgresql-8.4.20-6.el6.x86_64 4/4
Installed:
ambari-server.x86_64 0:2.6.0.0-267
Dependency Installed:
postgresql.x86_64 0:8.4.20-6.el6 postgresql-libs.x86_64 0:8.4.20-6.el6 postgresql-server.x86_64 0:8.4.20-6.el6
Complete!
6、初始化ambari-server服务
[root@yum hadoop]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (hadoop):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk/jdk1.8.0.181
ERROR: Exiting with exit code 1.
REASON: Downloading or installing JDK failed: 'Fatal exception: Java home path or java binary file is unavailable. Please put correct path to java home., exit code 1'. Exiting.
[root@yum hadoop]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (hadoop):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk/jdk1.8.0_181
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 1
Database admin user (postgres):
Database name (ambari):
Postgres schema (ambari):
Username (ambari):
Enter Database Password (bigdata):
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database: [ OK ]
About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
......ambari-admin-2.6.0.0.267.jar
.....
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
7、ambari分布式组件安装
登录：http://192.168.124.100:8080（为ambari服务所在IP）
登录ambari ：
用户名：admin
密码 ：admin
创建集群名：dgbc1
选择yum源：
参考:http://www.cnblogs.com/zlslch/p/6629249.html
获取目标ssh公钥方法：
登陆目标电脑：
cd /home/master/.ssh
ssh-keygen -t rsa
[master@localhost .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/master/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/master/.ssh/id_rsa.
Your public key has been saved in /home/master/.ssh/id_rsa.pub.
The key fingerprint is:
8e:ee:98:b1:c4:b9:a1:f6:70:4c:e4:d8:c2:c0:87:88 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| |
|+ . |
|Eo .. |
| o.= |
| + + S |
| = . o |
| . O . . |
| .= O |
| ...*.o |
+-----------------+
[master@localhost .ssh]$ ls
id_rsa id_rsa.pub
[master@localhost .ssh]$ cat id_rsa
配置节点告警解决
安装节点告警解决：
Ntpd
sudo service ntpd status
sudo service ntpd start
解决：Transparent Huge Pages(THP) enabled
在root用户下：echo never > /sys/kernel/mm/rehat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/rehat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/ transparent_hugepage / enabled
echo never > /sys/kernel/mm/ transparent_hugepage / defrag