大数据环境搭建-CDH6.3.0安装

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html

安装impala 非CDH 版本，恶心的一逼，算了老夫 花了20分钟装了一个CDH。

准备工作

配置hostname

1. 设置hostname,一定要独一无二

   sudo hostnamectl set-hostname foo-1.example.com
   

2. 修改 /etc/hosts

   192.168.1.102 foo-1.example.com
   

3. Edit /etc/sysconfig/network with the FQDN of this host only:

   HOSTNAME=foo-1.example.com
   

关闭防火墙

RHEL 7 compatible:

sudo systemctl disable firewalld
sudo systemctl stop firewalld

设置 SELiunx 模式

1. 检查SELiunx状态，如果是disabled跳过此步骤

   getenforce
   

2. Open the /etc/selinux/config file (in some systems, the /etc/sysconfig/selinux file).

   [root@flinkhadoop java]# vi /etc/selinux/config
   
   SELINUX=disabled
   SELINUXTYPE=targeted 
   
   

3. 重启系统或者使用下面的命令立即生效

   setenforce 0
   

开启ntp（本机玩忽略）

可以将server配置到公网的。

安装python 2.7.x

python --version

配置ulimit

1.查看

ulimit -a

ulimit -u    # max user processes

ulimit -n    # open files

2. 编辑

   vi /etc/security/limits.conf

* soft nofile 65536      # open files  (-n)
* hard nofile 65536

* soft nproc 65565
* hard nproc 65565       # max user processes   (-u)

需要重新登录，或者重新打开ssh客户端连接，永久生效

安装

配置仓库

Type	Location (baseurl)	Repo File
RHEL 7 Compatible	https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/	cloudera-manager.repo
RHEL6 Compatible	https://archive.cloudera.com/cm6/6.3.0/redhat6/yum/	cloudera-manager.repo
SLES 12	https://archive.cloudera.com/cm6/6.3.0/sles12/yum/	cloudera-manager.repo
Ubuntu Xenial (18.04)	https://archive.cloudera.com/cm6/6.3.0/ubuntu1804/apt/	cloudera-manager.list
Ubuntu Xenial (16.04)	https://archive.cloudera.com/cm6/6.3.0/ubuntu1604/apt/	cloudera-manager.list

我这里是 centos7

sudo wget https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

Import the repository signing GPG key:

• RHEL 7 compatible:

  sudo rpm --import https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPM-GPG-KEY-cloudera
  

安装oracle jdk

1. 使用cloudera的镜像

sudo yum install oracle-j2sdk1.8

2. 也可以直接下载oracle jdk 1.8

tar xvfz /opt/jdk-8u212-linux-x64.tar.gz -C /usr/java/

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_212
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile

安装 Cloudera-Manager

如果有条件可以直接在线安装

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

下载

https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JMfSnEuy-1588774016879)(docs/images/1568471656433.png)]

构造本地yum源

## 
yum -y install createrepo
mkdir -p /opt/package
## package存放所有的包
cd /opt/package
## 创建镜像仓库关联关系哦
createrepo .

配置nginx

用于其它机器访问

# 配置nginx
location /cloudera/package {
    alias  /opt/package;
    index index.html;
}

其它机器 修改本地的cloudera.manager的源

#

vi /etc/yum.repos.d/cloudera-manager.repo

​```
[cloudera-manager]
name=cloudera-manager
baseurl=http://192.168.1.101/cloudera/package/
gpgkey=https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPM-GPG-KEY-cloudera
gpgcheck=0
enabled=1
autorefresh=0
type=rpm-md


​```

yum clean all

安装

yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

ps -ef|grep cloudera-manager 

配置Mysql

centos 安装 https://www.cnblogs.com/xiaopotian/p/8196464.html

wget http://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

yum -y install mysql57-community-release-el7-11.noarch.rpm

yum install mysql-community-server


sudo systemctl start mysqld

查看一下初始密码

grep "password" /var/log/mysqld.log

登录

mysql -uroot -p

修改密码
ALTER USER 'root'@'localhost' IDENTIFIED BY 'Cdh123456.';

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Cdh123456.' WITH GRANT OPTION;
FLUSH PRIVILEGES;

安装数据库

创建数据库 & 执行脚本

mysql

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Cdh123456.';
FLUSH PRIVILEGES;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Cdh123456.';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Cdh123456.';

安装数据库驱动

cd /usr/share/java/
 wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.48/mysql-connector-java-5.1.48.jar
ln mysql-connector-java-5.1.48.jar mysql-connector-java.jar

初始化数据库

# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm Cdh123456.
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.1.101 scm scm Cdh123456.

Enter SCM password:
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

配置 parcel 包

上传包到server节点的/opt/cloudera/parcel-repo

cd /opt/cloudera/parcel-repo

# 不修改会导致<对于此 Cloudera Manager 版本 (5.15.0) 太新的 CDH 版本不会显示>
mv CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha1 CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha

启动

# 启动服务
systemctl start cloudera-scm-server

# 查看日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

访问

http://node1:7180

admin/admin

添加一台新机器

scp /usr/java/jdk1.8.0_212 [email protected]:/usr/java/jdk1.8.0_212 

sudo yum install cloudera-manager-daemons cloudera-manager-agent

sudo vi  /etc/cloudera-scm-agent/config.ini  
server_host=cdh1 （这里我是以cdh1作为server节点）

## 重启
systemctl restart cloudera-scm-agent
## 查看日志
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

问题

1. ddr.arpa domain name pointer localhost.localdomain.

   [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-q3C3NsOK-1588774016885)(docs/images/1568476397190.png)]

   https://www.cnblogs.com/jins-note/p/9513455.html>