hadoop搭建服务器

一、服务器配置说明

1.四台服务器

2.每台服务器内存4G

3./boot空间3072M

4.【可选择配置】/swap交互虚拟内存，4096

5./根目录空间57G

二、服务器网络选择配置

1.选择NAT链接网络/桥接

2.编辑~/.bashrc文件

vim ~/.bashrc

export all_proxy=socks5://192.168.197.2:21881

export http_proxy=http://192.168.197.2:21882

export https_proxy=http://192.168.197.2:21882

三、关闭防火墙

1.查看防火墙的状态

systemctl status firewalld.service

2.关闭防火墙

systemctl stop firewalld.service

3.永久关闭防火墙

systemctl disable firewalld.service

4.关闭每台虚拟机的SELinux

修改/etc/selinux/config 文件

将SELINUX=enforcing改为SELINUX=disabled

5.系统内核调优参考sysctl.conf

net.ipv4.ip_forward = 0

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 68719476736

kernel.shmall = 4294967296

net.ipv4.tcp_max_tw_buckets = 60000

net.ipv4.tcp_sack = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_rmem = 4096 87380 4194304

net.ipv4.tcp_wmem = 4096 16384 4194304

net.core.wmem_default = 8388608

net.core.rmem_default = 8388608

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.core.netdev_max_backlog = 262144

net.core.somaxconn = 262144

net.ipv4.tcp_max_orphans = 3276800

net.ipv4.tcp_max_syn_backlog = 262144

net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_synack_retries = 1

net.ipv4.tcp_syn_retries = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_mem = 94500000 915000000 927000000

net.ipv4.tcp_fin_timeout = 1

net.ipv4.tcp_keepalive_time = 1200

net.ipv4.tcp_max_syn_backlog = 65536

net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_synack_retries = 2

net.ipv4.tcp_syn_retries = 2

net.ipv4.tcp_tw_recycle = 1

#net.ipv4.tcp_tw_len = 1

net.ipv4.tcp_tw_reuse = 1

#net.ipv4.tcp_fin_timeout = 30

#net.ipv4.tcp_keepalive_time = 120

net.ipv4.ip_local_port_range = 1024  65535

四、建立用户

1.创建组和用户：bdadmin

# groupadd bdadmin

# useradd bdadmin-r -m -g bdadmin

设置密码：

# passwd bdadmin

New password:

Retype new password:

2.切换root与用户之间免密配置 /etc/sudoers

bdadmin ALL=(ALL) NOPASSWD: ALL

【注：为方便执行gp的一些命令，在切换用户时可以免密执行】

3.授予bdadmin数控权限

  确保取消注释包含 NOPASSWD 关键字的行。使用此命令将 bdadmin 用户添加到 wheel 组中。

# usermod -aG wheel bdadmin

五、配置SSH密钥以及服务器之间免密配置

1.开启公钥配置。 root用户下，修改配置文件 /etc/ssh/sshd_config

PubkeyAuthentication yes

PasswordAuthentication yes

2.在主节点配置私钥与公钥，切换到对应的用户上

ssh-keygen -t ed25519 -b 4096

cat id_dsa.pub >> authorized_keys

3.给服务器添加公钥

将主节点上的密钥分发给其他机器

ssh-copy-id -i /home/bdadmin/.ssh/authorized_keys bdadmin@segment1-server

ssh-copy-id -i /home/bdadmin/.ssh/authorized_keys bdadmin@segment2-server

ssh-copy-id -i /home/bdadmin/.ssh/authorized_keys bdadmin@segment3-server

4.验证

ssh bdadmin@segment1-server

六、安装JDK

可参考JAVA中的，Centos7 部署jdk

七、部署Scala

下载https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz

1.解压scala-2.11.8.tgz

2.配置环境变量 /etc/profile

export SCALA_HOME=/usr/local/src/scala-2.11.8

export PATH=$PATH:${SCALA_HOME}/bin

八、在master节点上安装mysql

1.检查centos 自带的mariadb，并卸载

rpm -qa|grep mariadb  (检查是否存在)

rpm -e --nodeps mariadb-libs-5.5.68-1.el7.x86_64 (卸载相应的内容)

2.下载

wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-libs-8.0.18-1.el7.x86_64.rpm

wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-common-8.0.18-1.el7.x86_64.rpm

wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-client-8.0.18-1.el7.x86_64.rpm

wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-server-8.0.18-1.el7.x86_64.rpm

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.18-1.el7.noarch.rpm

3.安装

rpm -ivh mysql-community-common-8.0.18-1.el7.x86_64.rpm

rpm -ivh mysql-community-libs-8.0.18-1.el7.x86_64.rpm

rpm -ivh mysql-community-client-8.0.18-1.el7.x86_64.rpm

rpm -ivh mysql-community-server-8.0.18-1.el7.x86_64.rpm

rpm -ivh mysql-connector-java-8.0.18-1.el7.noarch.rpm --force --nodeps

启动

systemctl start mysqld

查看启动状态

systemctl status mysqld

开机启动

systemctl enable mysqld

查看初始root密码

cat /var/log/mysqld.log | grep password

把密码记录下来，接下来登陆需要输入

登陆MySQL

mysql -u root -p

输入上面记录的密码

改变root的登陆密码

use mysql;

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Root@123';

允许非本机登陆

update user set host='%' where user='root';

grant all privileges on *.* to 'root' with grant option;

flush privileges;

添加hivedb用户

create user hivedb identified by 'Hivedb@123';

grant all privileges on *.* to 'hivedb' with grant option;

flush privileges;

查看用户：

select user,host from user;

配置文件 /etc/my.cnf

character_set_server=utf8mb4

collation-server=utf8mb4_general_ci

default-storage-engine=INNODB

九、zookeeper部署

下载zookeeper3.6.3版本，解压文件到/app路径下

下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz

1.建立对应的数据存放和日志目录

mkdir -p /data/zookeeper/zdata/data

mkdir -p /data/zookeeper/zdata/logs

2.修改配置

cd /app

ln -s apache-zookeeper-3.6.3-bin zookeeper

cd zookeeper/conf

cp zoo_sample.cfg zoo.cfg

# 修改zoo.cfg,以下为配置信息

dataDir=/data/zookeeper/zdata/data

dataLogsDir=/data/zookeeper/zdata/logs

# 副本数

autopurge.snapRetainCount=3

server.0=master-server:2888:3888

server.1=segment1-server:2888:3888

server.2=segment2-server:2888:3888

server.3=segment3-server:2888:3888

4lw.commands.whitelist=*

3.再dataDir的目录下创建myid文件，并在对应服务器添加服务编号

[master-server] # vim /data/zookeeper/zdata/data/myid

myid文件中添加对应服务器的编号：0

myid文件中添加对应服务器的编号：

matser-server 编辑myid文件 写入数子0;

segment1-server 编辑myid文件 写入数子1;

segment2-server 编辑myid文件 写入数子2;

segment3-server 编辑myid文件 写入数子3;

4.配置环境变量

# 在/etc/profile.d/env.sh中添加

export ZOOKEEPER_HOME=/app/apache-zookeeper-3.6.3

5.编写集群启动管理脚本 vim execZookeeper.sh

#!/bin/bash

if [ $# -lt 1 ]

then

    echo "No Args Input..."

    exit ;

fi

case $1 in

"start")

    echo " =================== 启动 zookeeper 集群 ======================"

    for host in hadoop101 hadoop102 hadoop103

    do

      echo "----------------- 启动 $host -------------------"

      ssh $host "/app/apache-zookeeper-3.6.3/bin/zkServer.sh start"

    done

;;

"stop")

    echo " =================== 关闭 zookeeper 集群 ======================"

    for host in hadoop101 hadoop102 hadoop103

    do

      echo "----------------- 关闭 $host -------------------"

      ssh $host "/app/apache-zookeeper-3.6.3/bin/zkServer.sh stop"

    done

;;

"status")

    echo " =================== 检查 zookeeper 集群 ======================"

    for host in hadoop101 hadoop102 hadoop103

    do

      echo "----------------- 检查 $host -------------------"

      ssh $host "/app/apache-zookeeper-3.6.3/bin/zkServer.sh status"

    done

esac

6.启动：sh execZookeeper.sh start;停止：sh execZookeeper.sh stop;状态：sh execZookeeper.sh status

7.启动zookeeper之后需要格式化一下：

hdfs zkfc -formatZK

十、hadoop部署

1.下载、解压

https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1

tar -zxvf hadoop-3.3.1.tar.gz 

2.配置环境变量

在/etc/profie.d/ 的文件夹下面新建一个文件 my_env.sh存储系统环境变量

#HADOOP_HOME

export HADOOP_HOME=/usr/local/src/hadoop-3.3.1

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

3.生效环境变量、并验证hadoop

source /etc/profile

hadoop

4.配置.sh文件

4.1.hadoop-3.3.1/etc/hadoop/hadoop_env.sh

配置用户

export HDFS_NAMENODE_USER="bdadmin"

export HDFS_DATANODE_USER="bdadmin"

export HDFS_SECONDARYNAMENODE_USER="bdadmin"

export YARN_RESOURCEMANAGER_USER="bdadmin"

export YARN_NODEMANAGER_USER="bdadmin"

配置路径

export JAVA_HOME=/usr/local/src/jdk1.8.0_321

export HADOOP_HOME=/usr/local/src/hadoop-3.3.1

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

配置网络接口：优先先择IPv4

export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"

配置操作系统类型

export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}

4.2.hadoop-3.3.1/etc/hadoop/mapred_env.sh

export JAVA_HOME=/usr/local/src/jdk1.8.0_321

4.3.hadoop-3.3.1/etc/hadoop/yarn_env.sh

export JAVA_HOME=/usr/local/src/jdk1.8.0_321

5.配置自定以site.xml文件

5.1hadoop-3.3.1/etc/hadoop/core-site.xml

       

   

        fs.defaultFS

        hdfs://master-server:8020

        Where HDFS NameNode can be found on the network

   

       

   

        hadoop.tmp.dir

        /home/bdadmin/hadoop/core-site/data/hdfs/tmp

        Where Hadoop will place all of its working files

   

   

   

        hadoop.http.staticuser.user

        hadoop

   

   

   

        hadoop.proxyuser.hadoop.hosts

        *

   

   

        hadoop.proxyuser.hadoop.groups

        *

   

5.2hadoop-3.3.1/etc/hadoop/hdfs-site.xml

   

   

      dfs.replication

      2

   

   

   

      dfs.namenode.name.dir

      /home/bdadmin/hadoop/hdfs-site/dfs/name

   

   

   

      dfs.datanode.data.dir

      /home/bdadmin/hadoop/hdfs-site/dfs/data

   

   

   

      dfs.nameservices

      mycluster

   

   

   

      dfs.ha.namenodes.mycluster

      nn1,nn2

   

   

   

   

      dfs.namenode.rpc-address.mycluster.nn1

      master-server:8020

   

   

      dfs.namenode.rpc-address.mycluster.nn2

      segment1-server:8020

   

   

   

      dfs.namenode.http-address.mycluster.nn1

      master-server:9870

   

   

      dfs.namenode.http-address.mycluster.nn2

      segment1-server:9870

   

   

   

   

   

   

      dfs.namenode.shared.edits.dir

      qjournal://master-server:8485;segment1-server:8485;segment2-server:8485/mycluster

   

   

   

      dfs.journalnode.edits.dir

      /home/bdadmin/hadoop/log/journalnode-namenode

 

5.3hadoop-3.3.1/etc/hadoop/yarn-site.xml

   

   

yarn.nodemanager.aux-services

mapreduce_shuffle

   

   

   

yarn.resourcemanager.hostname

master-server

   

5.4hadoop-3.3.1/etc/hadoop/mapred-site.xml

   

   

      mapreduce.framework.name

      yarn

   

6.初始化namenode

hdfs namenode -format

7.当无法启动namenode的standby节点时，把namenode的active节点的current文件下的所有文件copy到standby下的current文件夹下面

出现如下情况时：

2022-03-24 14:49:52,315 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: segment1-server/192.168.197.124:8020. Already tried 42 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=100, sleepTime=10000 MILLISECONDS)

执行

scp -r /home/bdadmin/hadoop/hdfs-site/dfs/name/current/* bdadmin@segment1-server:/home/bdadmin/hadoop/hdfs-site/dfs/name/current/

【注意】

1.第一次执行hadoop启动，配置了resourcemanager后，初始化时考虑权限问题，如果resourcemanager不能够正常启动起来，停止服务resourcemanager，删除服务器中零时的文件内容：

rm -rf /tmp/*

再重新启动即可，若还有问题，则查看日志。

2.再重新格式化namenode后，需要操作清空如下路劲中的文件

①、再hadoop中的core-site文件找到配置的零时文件路劲，删除此路径下所有的内容

   

        hadoop.tmp.dir

        /home/bdadmin/hadoop/core-site/data/hdfs/tmp

        Where Hadoop will place all of its working files

   

$ rm -rf /home/bdadmin/hadoop/core-site/data/hdfs/tmp/*

②、再hadoop中的hdfs-site文件找到配置的数据节点存放位置，删除此路径下的所有内容

   

    dfs.namenode.name.dir

      file:///home/bdadmin/hadoop/hdfs-site/dfs/name

   

   

   

      dfs.datanode.data.dir

      file:///home/bdadmin/hadoop/hdfs-site/dfs/data

   

$ rm -rf /home/bdadmin/hadoop/hdfs-site/dfs/name

$ rm -rf /home/bdadmin/hadoop/hdfs-site/dfs/data

③、再hadoop中的hdfs-site文件找到配置的journalnode存放位置，删除此路径下的所有内容

   

      dfs.journalnode.edits.dir

      /home/bdadmin/hadoop/log/journalnode-namenode

   

$ rm -rf /home/bdadmin/hadoop/log/journalnode-namenode

④、重启hadoop服务，并对namenode初始化(格式化)

这时候会报错/home/bdadmin/hadoop/log/journalnode-namenode下存在文件，不能初始化。

再次直接重复①-②-③步骤

然后停掉hadoop服务，在重启就可以了。

⑤、最后把active中的namenode下的current下的所有文件复制拷贝一份到standby中的namenode

scp -r /home/bdadmin/hadoop/hdfs-site/dfs/name/current/* bdadmin@segment1-server:/home/bdadmin/hadoop/hdfs-site/dfs/name/current/

⑥、再次停掉hadoop服务，重新启动。