使用ambari部署大数据平台
一.、版本介绍 使用的版本为ambari版本2.6.0.0，HDP版本2.6.3.0-235，HDP-UTILS-1.1.0.21及jdk1.8，mysql5.7，mysql-connector-java-5.1.45，另外系统内存和磁盘容量都要大一些。
二、环境准备
- 磁盘检查
~/bin/runRemoteCmd.sh “df -h” all
2.修改字符集
vim /etc/locale.conf
LANG=" en_US.UTF-8" 即为系统的语言（修改操作系统字符集）
修改文件保存退出之后要生效要执行如下命令才可生效
[root@ database01~]$ source vim /etc/locale.conf
3.关闭防火墙
查看状态： systemctl status firewalld
开机禁用 ： systemctl disable firewalld
4.关闭SeLinux
setenforce 0 临时生效
/etc/selinux/config SELINUX=disabled （重启后永久生效）
5.修改limits 参数
vi /etc/pam.d/login
//在其中加入如下内容：
session required /lib/security/pam_limits.so
vi /etc/security/limits.conf
//在文件中添加下面的内容
- soft nofile 65535
- hard nofile 65535
- soft nproc 65535
- hard nproc 65535
6.禁用IPV6
vi /etc/default/grub
GRUB_CMDLINE_LINUX=“ipv6.disable=1 rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet”
7.修改主机名
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=bigdata
重启生效。
8.配置hosts文件
vi /etc/hosts
192.168.234.135 bigdata
9.时钟同步
yum install ntp
完成后，配置开机启动：
chkconfig ntpd on
检查是否设置成功：
chkconfig --list ntpd
10.SSH互信配置
mkdir .ssh
ssh-keygen -t rsa//执行命令一路回车，生成秘钥
cd .ssh
cat id_rsa.pub >> authorized_keys //将公钥保存到authorized_keys认证文件中
cd …
chmod 700 .ssh
chmod 600 .ssh
ssh bigdata //第一次执行需要输入yes 第二次以后就可以直接访问
cat ~/.ssh/id_rsa.pub | ssh hadoop@ambari01 ‘cat >> ~/.ssh/authorized_keys’
scp -r authorized_keys hadoop@bigdata:~/.ssh/ scp文件到所有节点
11.相关服务的安装
yum -y install lrzsz
yum install -y openssh-clients
12.OPENJDK卸载和重新安装JDK
rpm -qa|grep java
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.191-2.6.15.5.el7.x86_64
tar -zxvf jdk-8u112-linux-x64.tar.gz -C /usr/local/jdk/
echo “export JAVA_HOME=/usr/local/jdk1.8.0_112” >> /etc/profile
echo “export PATH=$JAVA_HOME/bin:$PATH” >> /etc/profile
source /etc/profile
cat /etc/profile //查看jdk配置class路径
java -version
13.安装配置mysql
rpm -qa | grep mariadb 卸载掉mariadb包
rpm -e mariadb-libs-5.5.56-2.el7.x86_64 --nodeps
如果删不掉的话可以试试这条命令：yum remove mysql-libs
https://blog.csdn.net/qq_38417808/article/details/81291588
然后可以离线自己安装或者直接使用指令安装mysql。
mysql数据库，作为ambari的元数据数据库
通过
yum install mysql-community-server
安装mysql服务器。
systemctl enable mysqld.service
设置开机启动，并
systemctl start mysqld.service
获取root登录密码：
grep ‘password’ /var/log/mysqld.log
启动mysql服务，并根据提示设置root的初试密码：P;Gpd>L!i9hp
修改root密码
alter user ‘root’@‘localhost’ identified by ‘XXXX’
mysqladmin -u root password ‘xxxx’
grant all privileges on . to root@“localhost” Identified by “XXXX”;
grant all privileges on . to root@“127.0.0.1” Identified by “XXXX”;
grant all privileges on . to root@“主机名” Identified by “XXXX”;
mysql -uroot -pxxxx
进入mysql命令行，创建以下数据库：
#hive
create database hive default charset utf8 collate utf8_general_ci;
CREATE USER hive@‘localhost’ IDENTIFIED BY ‘XXXXX’;
CREATE USER hive@’%’ IDENTIFIED BY ‘XXXXX’;
grant all privileges on hive.* to ‘hive’@‘localhost’ identified by ‘XXXXX’;
grant all privileges on hive.* to ‘hive’@’%’ identified by ‘XXXXX’;
flush privileges;
#ambari
create database ambari default charset utf8 collate utf8_general_ci;
CREATE USER ambari@‘localhost’ IDENTIFIED BY ‘XXXXXX’;
CREATE USER ambari@’%’ IDENTIFIED BY ‘XXXXXX’;
grant all privileges on ambari.* to ‘ambari’@‘localhost’ identified by ‘XXXXXX’;
grant all privileges on ambari.* to ‘ambari’@’%’ identified by ‘XXXXXX’;
flush privileges;
#ranger
create database ranger default charset utf8 collate utf8_general_ci;
CREATE USER ranger@‘localhost’ IDENTIFIED BY ‘Xxxxx’;
CREATE USER ranger@’%’ IDENTIFIED BY ‘Xxxxx’;
grant all privileges on ranger.* to ‘ranger’@‘localhost’ identified by ‘Xxxxx’;
grant all privileges on ranger.* to ‘ranger’@’%’ identified by ‘Xxxxx’;
flush privileges;
设置root授权访问以上所有的数据库：
#授权root用户在主节点拥有所有数据库的访问权限
grant all privileges on . to root@"%" Identified by “XXXX”;
flush privileges;
grant all privileges on ambari.* to ambari@"%" Identified by “XXXX”;
grant all privileges on hive.* to hive@"%" Identified by “XXXX”;
grant all privileges on hive.* to ranger @"%" Identified by “XXXX”;
flush privileges;
测试mysql
mysql -uambari -pXxxxx@12#Qaz
use ambari;
create table aa( aa varchar(10));
show tables;
测试结果
mysql> show tables;
±-----------------+
| Tables_in_ambari |
±-----------------+
| aa |
±-----------------+
1 row in set (0.00 sec)
三、文件服务器搭建
1.下载 rpm下载软件 后续各个内网机器借助能连外网的机器下载rpm 包搭建http文件服务器
yum install yumdownloader
yumdownloader --resolve --destdir=/var/www/html/offlineRepo/other/ createrepo
yumdownloader --resolve --destdir=/var/www/html/offlineRepo/other/ httpd
2.安装httpd
yum安装apache
yum install httpd
随系统自启动
chkconfig httpd on
开启apache
service httpd start
在CentOS里Apache的默认文档路径的位置是在/var/www/html，配置文件的路径是/etc/httpd/conf/httpd.conf。其他的配置存储在/etc/httpd/conf.d/ 文件夹里。
建议删除/etc/httpd/conf.d/下的welcome.conf文件
删除指令 #rm -rf . （注释：*.*改成/etc/httpd/conf.d/welcome.conf）
vi /etc/httpd/conf/httpd.conf 修改此配置文件，把路径对应上
#
# This is the main Apache HTTP server configuration file. It contains the
# configuration directives that give the server its instructions.
# See
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do. They're here only as hints or reminders. If you are unsure
# consult the online docs. You have been warned.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path. If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so 'log/access_log'
# with ServerRoot set to '/www' will be interpreted by the
# server as '/www/log/access_log', where as '/log/access_log' will be
# interpreted as '/log/access_log'.
#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path. If you point
# ServerRoot at a non-local disk, be sure to specify a local disk on the
# Mutex directive, if file-based mutexes are used. If you wish to share the
# same ServerRoot for multiple httpd daemons, you will need to change at
# least PidFile.
#
ServerRoot "/etc/httpd"
#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 80
#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
Include conf.modules.d/*.conf
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User apache
Group apache
# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# definition. These values also provide defaults for
# any containers you may define later in the file.
#
# All of these directives may appear inside containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#
#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed. This address appears on some server-generated pages, such
# as error documents. e.g. [email protected]
#
ServerAdmin root@localhost
#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
#ServerName www.example.com:80
#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other
# blocks below.
#
#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#
#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"
#
# Relax access to content within /var/www.
#
AllowOverride None
# Allow open access:
Require all granted
# Further relax access to the default document root:
#
# Possible values for the Options directive are "None", "All",
# or any combination of:
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important. Please see
# http://httpd.apache.org/docs/2.4/mod/core.html#options
# for more information.
#
Options Indexes FollowSymLinks
#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
# Options FileInfo AuthConfig Limit
#
AllowOverride None
#
# Controls who can get stuff from this server.
#
Require all granted
#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
DirectoryIndex index.html
#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
Require all denied
#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a
# container, error messages relating to that virtual host will be
# logged here. If you *do* define an error logfile for a
# container, that host's errors will be logged there and not here.
#
ErrorLog "logs/error_log"
#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn
#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
# You need to enable mod_logio.c to use %I and %O
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a
# container, they will be logged here. Contrariwise, if you *do*
# define per- access logfiles, transactions will be
# logged therein and *not* in this file.
#
#CustomLog "logs/access_log" common
#
# If you prefer a logfile with access, agent, and referer information
# (Combined Logfile Format) you can use the following directive.
#
CustomLog "logs/access_log" combined
#
# Redirect: Allows you to tell clients about documents that used to
# exist in your server's namespace, but do not anymore. The client
# will make a new request for the document at its new location.
# Example:
# Redirect permanent /foo http://www.example.com/bar
#
# Alias: Maps web paths into filesystem paths and is used to
# access content that does not live under the DocumentRoot.
# Example:
# Alias /webpath /full/filesystem/path
#
# If you include a trailing / on /webpath then the server will
# require it to be present in the URL. You will also likely
# need to provide a section to allow access to
# the filesystem path.
#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the target directory are treated as applications and
# run by the server when requested rather than as documents sent to the
# client. The same rules about trailing "/" apply to ScriptAlias
# directives as to Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
AllowOverride None
Options None
Require all granted
#
# TypesConfig points to the file containing the list of mappings from
# filename extension to MIME-type.
#
TypesConfig /etc/mime.types
#
# AddType allows you to add to or override the MIME configuration
# file specified in TypesConfig for specific file types.
#
#AddType application/x-gzip .tgz
#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
#
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz
#
# If the AddEncoding directives above are commented-out, then you
# probably should define those extensions to indicate media types:
#
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
#AddHandler cgi-script .cgi
# For type maps (negotiated resources):
#AddHandler type-map var
#
# Filters allow you to process content before it is sent to the client.
#
# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default. To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
AddDefaultCharset UTF-8
#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type. The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
MIMEMagicFile conf/magic
#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#
#
# EnableMMAP and EnableSendfile: On systems that support it,
# memory-mapping or the sendfile syscall may be used to deliver
# files. This usually improves server performance, but must
# be turned off when serving from networked-mounted
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults if commented: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
EnableSendfile on
# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
3.准备HDP以及ambari 文件
分别解压到
/var/www/html/offlineRepo/ambari
/var/www/html/offlineRepo/HDP
/var/www/html/offlineRepo/HDP-UTILS
新建文件夹/var/www/html/offlineRepo/other，在/var/www/html/offlineRepo/ 目录下的各个文件夹中
createrepo ./
创建本地源repo文件
vi ambari.repo
[ambari]
name=HDP-AMBARI
baseurl=http://database01:18080/offlineRepo/ambari/
path=/
enabled=1
gpgcheck=0
[OTHER]
name=HDP-OTHER
baseurl=http://database01:18080/offlineRepo/other/
path=/
enabled=1
gpgcheck=0
同时后续在安装过程中如果发现却少什么包使用yumdownloader下载到other目录下。
4…访问文件服务器
文件服务器搭建完成后，访问http://database01:18080/offlineRepo/，文件服务器就搭好了
四、安装ambari
1…安装mysql驱动
MySql的官网下载JDBC驱动，http://dev.mysql.com/downloads/connector/j/，解压后，找到mysql-connector-java-5.1.47.jar，放到/usr/share/java中。
cp mysql-connector-java-5.1.47.jar /usr/share/java
2…安装服务端
yum install ambari-server
3…ambari基础表建立
进入mysql mysql -u root -p，进入ambari数据表，运行sql
mysql> use ambari
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
4…安装客户端
yum install ambari-agent
5…启动服务
service ambar-server restart
6./var/www/html/offlineRepo/HDP/centos7/2.6/hdp.repo
/var/www/html/ offlineRepo/HDP-Util/hdp-util.repo
/etc/yum.repos.d/ambari.repo
将上面几个repo分发到其它所有主机/etc/yum.repos.d
Ambari Server启动以后，就可以进行HDP的安装配置了,需要。
通过浏览器访问http://hadoop06:8080，默认的用户名和密码均为admin
剩下的就可以在页面上进行可视化安装需要的服务插件。