Kudu部署

一 Kudu的简介和部署

1.Kudu的特点

• OLAP工作负载的快速处理。
• 与MapReduce、Spark等Hadoop生态系统组件集成。
• 与Apache Impala的紧密集成，使其成为使用HDFS与Apache Parquet的一个很好的、可变的替代选择。

2.Kudu的rpm下载地址
http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.15.1/RPMS/x86_64/
3.下载的rpm文件

[hadoop@hadoop002 kudu]$ ll -h
total 348M
-rw-r--r-- 1 hadoop hadoop 117M Feb 15 11:51 kudu-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop 3.9M Feb 15 11:46 kudu-client0-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop  56K Feb 15 11:50 kudu-client-devel-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-rw-r-- 1 hadoop hadoop 228M Feb 15 11:56 kudu-debuginfo-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop 5.7K Feb 15 11:46 kudu-master-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop 5.7K Feb 15 11:46 kudu-tserver-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm

4.安装
4.1 首先要确认服务器的ntp服务已经安装好并启动

[hadoop@hadoop002 ~]$ service ntpd status
Redirecting to /bin/systemctl status  ntpd.service
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-11-10 13:53:13 CST; 4 months 28 days ago
 Main PID: 423 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─423 /usr/sbin/ntpd -u ntp:ntp -g
[hadoop@hadoop002 ~]$ 

4.2 RHEL或CentOS 6.6或更高版本需要从源头构建Kudu。要在7.0以上的版本上构建，必须安装Red Hat开发工具集(以便能够访问c++ 11的编译器)。
安装必备的库(如果没有安装的话)。

$ sudo yum install autoconf automake cyrus-sasl-devel cyrus-sasl-gssapi \
  cyrus-sasl-plain flex gcc gcc-c++ gdb git java-1.8.0-openjdk-devel \
  krb5-server krb5-workstation libtool make openssl-devel patch \
  pkgconfig redhat-lsb-core rsync unzip vim-common which

如果在RHEL或CentOS上构建的版本大于7.0，请安装Red Hat Developer Toolset。

$ DTLS_RPM=rhscl-devtoolset-3-epel-6-x86_64-1-2.noarch.rpm
$ DTLS_RPM_URL=https://www.softwarecollections.org/repos/rhscl/devtoolset-3/epel-6-x86_64/noarch/${DTLS_RPM}
$ wget ${DTLS_RPM_URL} -O ${DTLS_RPM}
$ sudo yum install -y scl-utils ${DTLS_RPM}
$ sudo yum install -y devtoolset-3-toolchain

4.3 本文通过rpm安装

[hadoop@hadoop002 kudu]$ sudo rpm -ivh --nodeps *
warning: kudu-1.7.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kudu-1.7.0+cdh5.15.1+0-1.cdh5.15.################################# [ 17%]
   2:kudu-client0-1.7.0+cdh5.15.1+0-1.################################# [ 33%]
   3:kudu-client-devel-1.7.0+cdh5.15.1################################# [ 50%]
   4:kudu-master-1.7.0+cdh5.15.1+0-1.c################################# [ 67%]
   5:kudu-tserver-1.7.0+cdh5.15.1+0-1.################################# [ 83%]
   6:kudu-debuginfo-1.7.0+cdh5.15.1+0-################################# [100%]
[hadoop@hadoop002 kudu]$ 

5.创建目录

[hadoop@hadoop002 data]$ cd kudu
[hadoop@hadoop002 ~]$ mkdir -p /data/kudu/kudu_master_data
[hadoop@hadoop002 ~]$ mkdir -p /data/kudu/kudu_tserver_data
[hadoop@hadoop002 kudu]$ ll
total 0
## master元数据目录
drwxrwxr-x 2 hadoop hadoop 6 Feb 15 13:08 kudu_master_data
##
drwxrwxr-x 2 hadoop hadoop 6 Feb 15 13:08 kudu_tserver_data
[hadoop@hadoop002 data]$ sudo chown -R kudu:kudu kudu
##创建日志目录
[hadoop@hadoop002 data]$ sudo mkdir -p /data/log/kudu
[hadoop@hadoop002 data]$ sudo chown -R kudu:kudu /data/log/kudu

6.修改配置文件
[hadoop@hadoop002 default]$ cd /etc/kudu/conf
[hadoop@hadoop002 conf]$ sudo vim master.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-master.
--fromenv=rpc_bind_addresses
--fromenv=log_dir

--fs_wal_dir=/home/hadoop/data/kudu/kudu_master_data
--fs_data_dirs=/home/hadoop/data/kudu/kudu_master_data

[hadoop@hadoop002 conf]$ sudo vim tserver.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-tserver.
--fromenv=rpc_bind_addresses
--fromenv=log_dir

--fs_wal_dir=/data/kudu/kudu_tserver_data
--fs_data_dirs=/data/kudu/kudu_tserver_data
--tserver_master_addrs=hadoop002:7051

6.启动

[hadoop@hadoop002 conf]$ cd /etc/init.d/
[hadoop@hadoop002 init.d]$ ll
total 72
-rwxr-xr-x 1 root root   250 Mar 30  2018 agentboot
lrwxrwxrwx 1 root root    35 Aug  4  2017 denyhost -> /usr/share/denyhosts/daemon-control
-rw-r--r-- 1 root root 15131 Sep 12  2016 functions
lrwxrwxrwx 1 root root    41 Nov 29 12:54 ifritd -> /usr/local/share/jcloud/ifrit/sbin/ifritd
-rwxrwxr-x 1 root root  6258 Oct 28 12:59 jdog_service
-rwxr-xr-x 1 root root  5638 Aug 10  2018 kudu-master
-rwxr-xr-x 1 root root  5642 Aug 10  2018 kudu-tserver
-rwxr-xr-x 1 root root 10975 Oct 28 18:38 mysql
-rwxr-xr-x 1 root root  2989 Sep 12  2016 netconsole
-rwxr-xr-x 1 root root  6643 Sep 12  2016 network
-rw-r--r-- 1 root root  1160 Apr 11  2018 README
[hadoop@hadoop002 init.d]$ sudo ./kudu-master start
[hadoop@hadoop002 init.d]$ sudo ./kudu-tserver start
[hadoop@hadoop002 init.d]$ ps aux | grep kudu
kudu     29385  0.2  0.4 667692 37220 ?        Sl   21:54   0:01 /usr/lib/kudu/sbin/kudu-master --server_dump_info_path=/var/run/kudu/kudu-master-kudu.json --flagfile=/etc/kudu/conf/master.gflagfile
kudu     30995  0.4  0.2 1117024 17592 ?       Sl   22:02   0:00 /usr/lib/kudu/sbin/kudu-tserver --server_dump_info_path=/var/run/kudu/kudu-tserver-kudu.json --flagfile=/etc/kudu/conf/tserver.gflagfile
hadoop   31250  0.0  0.0 112716   960 pts/0    S+   22:02   0:00 grep --color=auto kudu

通过UI界面查看：access the Master or Tablet Server web UI by opening http://<_host_name_>:8051/ for masters or http://<_host_name_>:8050/ for tablet servers.

部署启动报错解决办法
1../kudu-master: line 47: /lib/lsb/init-functions: No such file or directory

[hadoop@hadoop002 init.d]$ yum install redhat-lsb -y
[hadoop@hadoop002 init.d]$ sudo ./kudu-master start
Kudu Master Server is running                              [  OK  ]
[hadoop@hadoop002 init.d]$ sudo ./kudu-tserver start
Started Kudu Tablet Server (kudu-tserver):                 [  OK  ]
[hadoop@hadoop002 init.d]$ 

2.在kudu webUI上的log中，如果遇到如下错误

W0407 23:38:17.507225 19312 heartbeater.cc:587] Failed to heartbeat to hadoop002:7051: Invalid argument: Failed to ping master at hadoop002:7051: Client connection negotiation failed: client connection to 192.168.0.3:7051: unable to find SASL plugin: PLAIN
W0407 23:38:18.507685 13567 negotiation.cc:313] Failed RPC negotiation. Trace:
0407 23:38:18.507519 (+     0us) reactor.cc:577] Submitting negotiation task for client connection to 192.168.0.3:7051
0407 23:38:18.507646 (+   127us) negotiation.cc:304] Negotiation complete: Invalid argument: Client connection negotiation failed: client connection to 192.168.0.3:7051: unable to find SASL plugin: PLAIN
Metrics: {"client-negotiator.queue_time_us":57,"thread_start_us":22,"threads_started":1}

或者如下的报错

Exceptions received: org.apache.kudu.client.RecoverableException: connection disconnected, [28844ms] querying master, [28844ms] Sub rpc: ConnectToMaster sending RPC to server master-hadoop002:7051, [29108ms] Sub rpc: ConnectToMaster received from server master-hadoop002:7051 response Network error: connection disconnected, [29108ms] delaying RPC due to Service unavailable: Master config (hadoop002:7051) has no leader. 

解决方法如下：

yum install gcc python-devel
yum install cyrus-sasl*

3.如果本地IDEA开发工具遇到如下报错
 

W0407 23:47:33.307684 16730 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from IP:4581: unauthenticated connections from publicly routable IPs are prohibited. See --trusted_subnets flag for more information.: IP:4581

需要在master.gflagfile添加如下配置

#添加权限设置：
#方式一： 
-rpc_encryption=disabled
-rpc_authentication=disabled
#方式二： 
--trusted_subnets=0.0.0.0/0

然后重启kudu服务。
4.如果本地IDEA开发工具遇到如下报错
 

INFO AsyncKuduClient: Invalidating location 8cd8c704667747d98afb3ef342f6f1b5(hadoop002:7050) for tablet 01645c12abce4ab38e2f7a97d6991892: connection disconnected

需要在tserver.gflagfile添加如下配置

#添加权限设置：
#方式一： 
-rpc_encryption=disabled
-rpc_authentication=disabled
#方式二： 
--trusted_subnets=0.0.0.0/0

然后重启kudu服务。