cloudera-manager安装+CDH5.4.1部署+mysql+python+impala安装
三台Red Hat Enterprise Linux Server release 6.3 (Santiago) ，搭建hadoop
[root@master01 cdh_install]# more /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.5.140 master01 192.168.5.141 slave01 192.168.5.142 slave02
--解压jdk
tar -xvf jdk-7u7-linux-i586.tar.gz
--配置jdk环境变量
export JAVA_HOME=/it/jdk1.7.0_07
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
--测试java环境
[root@master01 it]# source /etc/profile [root@master01 it]# java -version java version "1.7.0_07" Java(TM) SE Runtime Environment (build 1.7.0_07-b10) Java HotSpot(TM) Server VM (build 23.3-b01, mixed mode)
--上传安装包
[root@master01 cdh_install]# mkdir -p cdh_install [root@master01 cdh_install]# cd cdh_install/
[root@master01 cdh_install]# scp * root@slave01:/it/cdh_install/ root@slave01's password: CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel 100% 1215MB 46.7MB/s 00:26 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha 100% 41 0.0KB/s 00:00 cloudera-manager-el6-cm5.4.1_x86_64.tar.gz 100% 659MB 43.9MB/s 00:15 manifest.json 100% 41KB 41.5KB/s 00:00 Python-2.6.6.tgz 100% 13MB 12.7MB/s 00:00 Python-2.7.10.tgz 100% 16MB 16.0MB/s 00:01 [root@master01 cdh_install]# scp * root@slave02:/it/cdh_install/ root@slave02's password: CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel 100% 1215MB 40.5MB/s 00:30 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha 100% 41 0.0KB/s 00:00 cloudera-manager-el6-cm5.4.1_x86_64.tar.gz 100% 659MB 47.1MB/s 00:14 manifest.json 100% 41KB 41.5KB/s 00:00 Python-2.6.6.tgz 100% 13MB 12.7MB/s 00:00 Python-2.7.10.tgz 100% 16MB 16.0MB/s 00:01 [root@master01 cdh_install]#
--下面我们进行CM5.4.1的安装配置，在安装的过程中，官方文档有点累赘，看起来有点费劲（但不可否认，它确实很详细，最权威的文档）
--Installation Path C - Manual Installation Using Cloudera Manager Tarballs
--建立目录，拷贝文件(root用户下执行以下命令) sudo mkdir /opt/cloudera-manager tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager useradd --system --home=/opt/cloudera-manager/cm-5.4.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm mkdir -p /var/log/cloudera-scm-headlamp mkdir -p /var/log/cloudera-scm-firehose mkdir -p /var/log/cloudera-scm-alertpublisher mkdir -p /var/log/cloudera-scm-eventserver mkdir -p /var/lib/cloudera-scm-headlamp mkdir -p /var/lib/cloudera-scm-firehose mkdir -p /var/lib/cloudera-scm-alertpublisher mkdir -p /var/lib/cloudera-scm-eventserver mkdir -p /var/lib/cloudera-scm-server chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp mkdir -p /var/cm_logs/cloudera-scm-headlamp chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp mkdir -p /opt/cloudera/parcel-repo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo mkdir -p /opt/cloudera/parcels chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
--下面进行mysql的安装配置（rpm包）
--mysql安装
[root@master01 cdh_install]# rpm -ivh mysql-community-client-5.7.9-1.el6.x86_64.rpm mysql-community-com-embedded-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-devel-5.7.9-1.el6.x86_64.rpm mysql-community-lmunity-server-5.7.9-1.el6.x86_64.rpm warning: mysql-community-client-5.7.9-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: N Preparing... ########################################### [100%] file /usr/share/mysql/czech/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_64 file /usr/share/mysql/danish/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_6 file /usr/share/mysql/dutch/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_64
上面mysql安装报错，肯定是mysql有低版本的包，查看mysql已安装的包
[root@master01 cdh_install]# rpm -qa | grep -i mysql mysql-libs-5.1.61-4.el6.x86_64
--卸载已安装的mysql
[root@master01 cdh_install]# yum -y remove mysql-libs-5.1.61* Loaded plugins: product-id, refresh-packagekit, subscription-manager Updating certificate-based repositories. Unable to read consumer identity Setting up Remove Process Resolving Dependencies --> Running transaction check ---> Package mysql-libs.x86_64 0:5.1.61-4.el6 will be erased --> Processing Dependency: libmysqlclient.so.16()(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64 --> Processing Dependency: libmysqlclient.so.16(libmysqlclient_16)(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64 --> Processing Dependency: mysql-libs for package: 2:postfix-2.6.6-2.2.el6_1.x86_64 --> Running transaction check ---> Package postfix.x86_64 2:2.6.6-2.2.el6_1 will be erased --> Processing Dependency: /usr/sbin/sendmail for package: cronie-1.4.4-7.el6.x86_64 --> Running transaction check 省略。。。 Removed: mysql-libs.x86_64 0:5.1.61-4.el6 Dependency Removed: cronie.x86_64 0:1.4.4-7.el6 cronie-anacron.x86_64 0:1.4.4-7.el6 crontabs.noarch 0:1.10-33.el6 postfix.x86_64 2:2.6.6-2.2.el6_1 sysstat.x86_64 0:9.0.4-20.el6 Complete! [root@master01 cdh_install]#
--查看是否已卸载
[root@master01 cdh_install]# rpm -qa | grep -i mysql [root@master01 cdh_install]# 没有了
[root@master01 cdh_install]# rpm -ivh mysql-community-client-5.7.9-1.el6.x86_64.rpm mysql-community-common-5.7.9-1.el6.x86_64.rpm mysql-community-devel-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-devel-5.7.9-1.el6.x86_64.rpm mysql-community-libs-5.7.9-1.el6.x86_64.rpm mysql-community-libs-compat-5.7.9-1.el6.x86_64.rpm mysql-community-server-5.7.9-1.el6.x86_64.rpm warning: mysql-community-client-5.7.9-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY Preparing... ########################################### [100%] 1:mysql-community-common ########################################### [ 13%] 2:mysql-community-libs ########################################### [ 25%] 3:mysql-community-client ########################################### [ 38%] 4:mysql-community-devel ########################################### [ 50%] 5:mysql-community-embedde########################################### [ 63%] 6:mysql-community-embedde########################################### [ 75%] 7:mysql-community-server ########################################### [ 88%] 8:mysql-community-libs-co########################################### [100%] [root@master01 cdh_install]#
--重设置密码
5.7.9重设置密码 有人会问，为什么user没有password字段了。从5.6之后就没有了 会在 /var/log/mysqld.log 最后生成一个随机密码，供初始登录。 [root@master01 mysql]# more /var/log/mysqld.log 160203 14:40:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 2016-02-03T06:40:45.397816Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2016-02-03T06:40:45.399265Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.9) starting as process 36627 ... 2016-02-03T06:40:45.407902Z 0 [Note] InnoDB: PUNCH HOLE support available 2016-02-03T07:28:24.812608Z 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.7.9' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) 2016-02-03T07:50:54.238157Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2016-02-03T07:50:54.708935Z 0 [Warning] InnoDB: New log files created, LSN=45790 2016-02-03T07:50:54.780046Z 0 [Warning] InnoDB: Creating foreign key constraint system tables. 2016-02-03T07:50:54.840130Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: da97342a-ca4a-11e5- acfb-000c29c14e9c. 2016-02-03T07:50:54.841328Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened. 2016-02-03T07:50:54.845873Z 1 [Note] A temporary password is generated for root@localhost: C*s!.=y6<-OC [root@master01 mysql]# mysql -uroot -p password:
--或者直接跳过 权限控制，修改密码 （我通常采用这种）
[root@master01 mysql]# mysqld_safe --skip-grant-tables --user=mysql &
[1] 37043
[root@master01 mysql]# 160203 15:59:11 mysqld_safe Logging to '/var/log/mysqld.log'.
160203 15:59:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.9 MySQL Community Server (GPL)
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
mysql>
mysql> use mysql
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.
mysql> SET PASSWORD = PASSWORD('123456');
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql>
mysql>
mysql> exit
Bye
--我们登陆试试
[root@master01 mysql]# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 6 Server version: 5.7.9 MySQL Community Server (GPL) Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> exit 搞定。
--重启mysqld
[root@master01 mysql]# vi /etc/my.cnf --调整一些参数 [root@master01 mysql]# sudo /sbin/chkconfig mysqld on [root@master01 mysql]# sudo /sbin/chkconfig --list mysqld mysqld 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@master01 mysql]# sudo service mysqld start Starting mysqld: [ OK ]
--把jar包拷贝到对应目录下，改为为mysql-connector-java.jar
[root@master01 share]# mkdir -p /usr/share/java [root@master01 share]# sudo cp /it/cdh_install/mysql-connector-java-5.1.38-bin.jar /usr/share/java/mysql-connector-java.jar [root@master01 share]# cd /usr/share/java [root@master01 java]# ls mysql-connector-java.jar [root@master01 share]#
mysql> create database amon DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all PRIVILEGES on amon.* TO ' amon'@'%' IDENTIFIED BY ' 123456' WITH GRANT OPTION; Query OK, 1 row affected (0.00 sec) 省略（ 省略create database .... GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; GRANT ALL PRIVILEGES ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; grant all PRIVILEGES on metastore.* TO 'hive'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; grant all PRIVILEGES on sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; grant all PRIVILEGES on nav.* TO 'nav'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; grant all PRIVILEGES on navms.* TO 'navms'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; ） mysql> mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)
--备份mysql
[root@master01 java]# mysqldump -uroot -p mysql > /tmp/mysql-bakup.sql Enter password: [root@master01 java]# cd /tmp/ [root@master01 tmp]# ls hsperfdata_root mysql-bakup.sql pulse-ahmL5Yupk46M ssh-XKnrmy7U7AfM keyring-NkHAG2 orbit-root pulse-Nij3doz66iv1 yum.log
----建立scm
[root@master01 schema]# pwd /opt/cloudera-manager/cm-5.4.1/share/cmf/schema [root@master01 schema]# ./scm_prepare_database.sh mysql scm scm 123456 JAVA_HOME=/it/jdk1.7.0_07 Verifying that we can write to /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server Creating SCM configuration file in /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server Executing: /it/jdk1.7.0_07/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.4.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. Wed Feb 03 17:58:37 CST 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! [root@master01 schema]#
[root@master01 parcel-repo]# pwd /opt/cloudera/parcel-repo [root@master01 parcel-repo]# cp /it/cdh_install/CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel* manifest.json . [root@master01 parcel-repo]# ls -l total 1244320 -rw-r--r-- 1 root root 1274129311 Feb 3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel -rw-r--r-- 1 root root 41 Feb 3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha -rw-r--r-- 1 root root 42445 Feb 3 18:04 manifest.json [root@master01 parcel-repo]# ls -l total 1244276 -rw-r--r-- 1 root root 1274129311 Feb 3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel -rw-r--r-- 1 root root 41 Feb 3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha [root@master01 parcel-repo]# cd /opt/cloudera-manager/cm-5.4.1/etc/init.d/ [root@master01 init.d]# ls cloudera-scm-agent cloudera-scm-server cloudera-scm-server-db [root@master01 init.d]# ls -l total 20 -rwxr-xr-x 1 1106 4001 5241 May 9 2015 cloudera-scm-agent -rwxr-xr-x 1 1106 4001 6105 May 9 2015 cloudera-scm-server -rwxr-xr-x 1 1106 4001 2894 May 9 2015 cloudera-scm-server-db [root@master01 init.d]# ./cloudera-scm-server start Starting cloudera-scm-server: [ OK ] [root@master01 init.d]#
[root@master01 cloudera-scm-server]# tailf cloudera-scm-server.log 2016-02-03 18:02:51,753 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2016-02-03T10:02:51.753Z 2016-02-03 18:02:51,790 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2016-02-03T10:02:51.790Z 2016-02-03 18:02:51,801 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:114 2016-02-03 18:02:51,801 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2016-02-03T10:02:51.801Z 2016-02-03 18:02:51,838 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:126 2016-02-03 18:02:51,840 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2016-02-03T10:02:51.839Z 2016-02-03 18:02:52,118 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4 2016-02-03 18:02:52,120 INFO WebServerImpl:org.mortbay.log: Started [email protected]:7180 2016-02-03 18:02:52,120 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. CM-server就搞定了
先检查包 [root@master01 init.d]# rpm -q chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb chkconfig-1.3.49.3-2.el6.x86_64 python-2.6.6-29.el6_2.2.x86_64 package bind-utils is not installed psmisc-22.6-15.el6_0.1.x86_64 libxslt-1.1.26-2.el6.x86_64 zlib-1.2.3-27.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 package cyrus-sasl-plain is not installed package cyrus-sasl-gssapi is not installed package fuse is not installed package portmap is not installed package fuse-libs is not installed package redhat-lsb is not installed [root@master01 init.d]#
--yum安装所需要的包
[root@master01 init.d]# yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb Loaded plugins: product-id, refresh-packagekit, subscription-manager
--启动代理
[root@master01 init.d]# ./cloudera-scm-agent start Starting cloudera-scm-agent: [ OK ] [root@master01 init.d]#
--可以登录界面，默认用户密码为admin/admin
这里有140，141，142三台机器，我们这里先安装140=master01 141=slave01，
142等安装好后，作为新节点添加。（测试添加节点）
