大数据平台搭建(1)

注意:因为博客中美元符号有特殊含义,所以将美元符号替换为&
涉及技术:
JDk1.8+zookeeper-3.4.10+hadoop-2.7.3.tar.gz+
hbase-1.3.1+scala-2.11.8.tgz+spark-2.1.0-bin-hadoop2.7+
apache-hive-2.3.2-bin.tar.gz+
mysql-5.7.20-linux-glibc2.12-x86_64.tar.gz

1.机器准备(一共准备5台机器,其中2台作为namenode,3台作为datanode)
D:\Virtual Machines\namenode01 namenode01 192.168.242.167 zookeeper
D:\Virtual Machines\namenode02 namenode02 192.168.242.168 zookeeper
D:\Virtual Machines\datanode01 datanode01 192.168.242.169 zookeeper
D:\Virtual Machines\datanode02 datanode02 192.168.242.170 zookeeper
D:\Virtual Machines\datanode03 datanode03 192.168.242.171 zookeeper

2.关于网络配置
a.主机名配置(vim /etc/sysconfig/network)
例如 HOSTNAME=namenode01
b.网络映射,即将IP与主机名绑定(vim /etc/hosts)
192.168.242.167 namenode01
192.168.242.168 namenode02
192.168.242.169 datanode01
192.168.242.170 datanode02
192.168.242.171 datanode03
c.禁用IPV6
echo ” “>>/etc/modprobe.d/dist.conf
echo “alias net-pf-10 off”>>/etc/modprobe.d/dist.conf
echo “alias ipv6 off”>>/etc/modprobe.d/dist.conf

3.重启所有的机器
shutdown -r now

4.关闭防火墙
关闭防火墙 service iptables stop
防火墙开机不自动启动 chkconfig iptables off
查看防火墙状况 chkconfig –list | grep iptables

5.关闭SELINUX
将SELINUX的值修改为disabled(vim /etc/syconfig/selinux)

6.ssh免密码登陆
a.回车四次,/root/.ssh文件夹下生成id_rsa私钥和id_rsa.pub公钥(ssh-keygen)
b.拷贝公钥到本机的公钥文件authorized_key(ssh-copy-id -i id_rsa.pub localhost)
c.拷贝公钥文件到其他机器的authorized_key中
(scp root@datanode03:/root/.ssh/authorized_keys /root/.ssh/)

7.系统时钟同步(暂时略过)
a.查看当前系统时间(date)
b.查看当前系统时间服务器的状态(service ntpd status)
c.开启当前系统的时间服务器(service ntpd start)

8.安装JDK
a.在指定目录解压JDK
b.配置JDK的环境变量(vim /etc/profile文件)
c.使profile文件生效(source /etc/profile)
9.安装zookeeper
a.上传:将zookeeper-3.4.10.tar.gz压缩包上传到/user/local路径下
b.解压:将上传的zookeeper包解压缩到当前的目录下(tar -zxvf zookeeper-3.4.10.tar.gz)
c.配置环境变量:在profile文件末尾添加配置(vim /etc/profile)
export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.10
export PATH=&ZOOKEEPER_HOME/bin:&ZOOKEEPER_HOME/conf:&PATH
d.生效:是配置的profile文件立刻生效(source /etc/profile)
e.配置zookeeper-3.4.10/conf/zoo.cfg文件,这个文件本身是没有的,有个zoo_sample.cfg模板
进入conf目录(cd zookeeper-3.4.5/conf)
拷贝模板(cp zoo_sample.cfg zoo.cfg)
修改zoo.cfg文件,红色是修改部分(vim zoo.cfg)
配置zoo.cfg内容如下:

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/usr/local/zookeeper-3.4.10/data

# the port at which the clients will connect

clientPort=2181

#

# Be sure to read the maintenance section of the # administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

server.1=namenode01:2888:3888
server.2=namenode02:2888:3888
server.3=datanode01:2888:3888
server.4=datanode02:2888:3888
server.5=datanode03:2888:3888

f.创建/user/local/zookeeper-3.4.10/data目录,并在data中创建myid文件
进入指定目录(cd /user/local/zookeeper-3.4.10)
创建目录data(mkdir data)
进入data目录(cd /user/local/zookeeper-3.4.10/data) 进入data文件夹下
创建myid文件,并在这个文本内写入1(touch myid)
g.测试zookeeper集群是否搭建好
启动zookeeper集群:(zkServer.sh start)
查看当前zookeeper集群的状态,一台leader其他的为follower:(zkServer.sh status)
启动客户端:zkCli.sh -server namenode02:2181