Apache Doris 0.14.0版本 MPP数据库编译和安装
目录
- 1. 编译
- 2. PE的高可用搭配
- 3. 安装
-
- 3.1 安装第一个FE
- 3.2 安装第一个BE
- 3.3 新增一个PE(follower/observer)
- 3.4 新增一个BE
1. 编译
官方提供Docker环境编译、Centos\Ubuntu环境编译、ARM64平台编译，我们这里采样Docker环境编译
下载Docker镜像
[root@bigdata001 ~]#
[root@bigdata001 ~]# docker pull apache/incubator-doris:build-env-1.2
build-env-1.2: Pulling from apache/incubator-doris
7a376117f479: Pull complete
Digest: sha256:a0b8f7e3da126b2692600174cff762569fd38d722f54512a145d9b40a764747c
Status: Downloaded newer image for apache/incubator-doris:build-env-1.2
docker.io/apache/incubator-doris:build-env-1.2
[root@bigdata001 ~]#
运行镜像
[root@bigdata001 ~]#
[root@bigdata001 ~]# docker run -it apache/incubator-doris:build-env-1.2
[root@efff319b8bb7 ~]#
下载解压Apache Doris源码
[root@efff319b8bb7 ~]#
[root@efff319b8bb7 ~]# wget --no-check-certificate https://dlcdn.apache.org/incubator/doris/0.14.0-incubating/apache-doris-0.14.0-incubating-src.tar.gz
[root@efff319b8bb7 ~]#
[root@efff319b8bb7 ~]# tar -zxvf apache-doris-0.14.0-incubating-src.tar.gz
[root@efff319b8bb7 ~]#
编译源码
[root@efff319b8bb7 ~]#
[root@efff319b8bb7 ~]# cd apache-doris-0.14.0-incubating-src
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]#
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]# sh build.sh
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]#
编译后的结果文件在output下
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]#
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]# ls output
be fe udf
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]#
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]# zip -r output.zip output
[root@efff319b8bb7 apache-doris-0.14.0-incubating-src]#
将output.zip导出到宿主机
[root@bigdata001 ~]#
[root@bigdata001 ~]# docker cp efff319b8bb7:/root/apache-doris-0.14.0-incubating-src/output.zip /root
[root@bigdata001 ~]#
[root@bigdata001 ~]# ls output.zip
output.zip
[root@bigdata001 ~]#
2. PE的高可用搭配
- 一个follower和一个observer可以实现读高可用
- 3个follower可以实现读写高可用，follower数量必须为奇数
- observer只负责同步follower的元数据，不参加Leader选举
3. 安装
服务器之间的时间偏差要求在5秒内，时间同步可以参考Centos7服务器通过NTP设置时间同步
Doris的FE和BE新增扩容，不影响已经运行的FE和BE，所以我们先部署一个FE和一个BE
3.1 安装第一个FE
1. 将output.zip拷贝到/opt目录下，解压，并重命名
[root@bigdata001 ~]#
[root@bigdata001 ~]# cp output.zip /opt
[root@bigdata001 ~]# cd /opt
[root@bigdata001 opt]#
[root@bigdata001 opt]# unzip output.zip
[root@bigdata001 opt]#
[root@bigdata001 opt]# mv output apache-doris-0.14.0
[root@bigdata001 opt]#
[root@bigdata001 opt]# cd apache-doris-0.14.0/
[root@bigdata001 apache-doris-0.14.0]#
[root@bigdata001 apache-doris-0.14.0]# ls
be fe udf
[root@bigdata001 apache-doris-0.14.0]#
如果条件好的，FE和BE应该部署在不同的服务器上，即使在同一服务器，也应该位于不同的磁盘上
2. 新增doris-meta目录
[root@bigdata001 fe]#
[root@bigdata001 fe]# mkdir doris-meta
[root@bigdata001 fe]#
3. 修改fe/conf/fe.conf：
修改内容如下：
JAVA_OPTS="-Xmx8192m ......省略内容......"
meta_dir = /opt/apache-doris-0.14.0/fe/doris-meta
priority_networks = 192.168.8.111/24
http_port = 7030
新增内容如下：
JAVA_HOME=/opt/jdk1.8.0_201
4. 启动FE
[root@bigdata001 fe]#
[root@bigdata001 fe]# sh bin/start_fe.sh --daemon
[root@bigdata001 fe]#
3.2 安装第一个BE
新增doris-data.HDD目录
[root@bigdata001 be]#
[root@bigdata001 be]# mkdir doris-data.HDD
[root@bigdata001 be]#
修改be/conf/be.conf
修改内容如下：
priority_networks = 192.168.8.111/24
storage_root_path = /opt/apache-doris-0.14.0/be/doris-data.HDD,200
webserver_port = 7020
brpc_port = 8061
参数说明：
- HDD表示目录所在的磁盘是HDD类型，可以通过命令
lsblk -d -o name,rota进行查看，rota返回的是1，表示可以旋转，是机械硬盘HDD - 200表示目录的最大储存容量
修改open files的数量
[root@bigdata001 be]#
[root@bigdata001 be]# ulimit -n 65536
[root@bigdata001 be]#
- 可以通过
ulimit -a进行查看
FE添加BE
[root@bigdata001 opt]#
[root@bigdata001 opt]# mysql -h bigdata001 -P 9030 -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.0
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> ALTER SYSTEM ADD BACKEND "192.168.8.111:9050";
Query OK, 0 rows affected (0.03 sec)
mysql>
- 连接PE的端口为PE的query_port参数值
- 添加BACKEND的端口为BE的heartbeat_service_port参数值
启动BE
[root@bigdata001 be]#
[root@bigdata001 be]# sh bin/start_be.sh --daemon
[root@bigdata001 be]#
查看BE启动情况
再次通过mysql客户端连接PE，进行查看, Alive列为true表示BE启动成功
mysql>
mysql> SHOW PROC '/backends';
+-----------+-----------------+---------------+------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+------------------------+----------------------------------------+
| BackendId | Cluster | IP | HostName | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version | Status |
+-----------+-----------------+---------------+------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+------------------------+----------------------------------------+
| 11001 | default_cluster | 192.168.8.111 | bigdata001 | 9050 | 9060 | 7020 | 8061 | 2021-10-13 16:40:59 | 2021-10-13 16:41:18 | true | false | false | 0 | .000 | 1.000 B | .000 | 0.00 % | 0.00 % | | 0.14.0-release-Unknown | {"lastSuccessReportTabletsTime":"N/A"} |
+-----------+-----------------+---------------+------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+------------------------+----------------------------------------+
1 row in set (0.01 sec)
mysql>
3.3 新增一个PE(follower/observer)
前面的步骤可以参考【安装第一个FE】的前3步，但注意fe/conf/fe.conf中的priority_networks参数，如在bigdata002上安装该参数就是priority_networks = 192.168.8.112/24
向Leader PE添加follower
[root@bigdata001 opt]#
[root@bigdata001 opt]# mysql -h bigdata001 -P 9030 -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.0
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> ALTER SYSTEM ADD FOLLOWER "192.168.8.112:9010";
Query OK, 0 rows affected (0.01 sec)
mysql>
- 端口是fe/conf/fe.conf中的edit_log_port参数值
- 如果添加observer, 则将上面的FOLLOWER修改为OBSERVER即可
在bigdata002上启动FE
[root@bigdata002 fe]#
[root@bigdata002 fe]# sh bin/start_fe.sh --helper bigdata001:9010 --daemon
[root@bigdata002 fe]#
- –helper参数只在初始化的时候需要
- 端口是fe/conf/fe.conf中的edit_log_port参数值
再次通过mysql客户端连接PE，进行查看, Alive列为true表示PE启动成功
mysql>
mysql> SHOW PROC '/frontends';
+----------------------------------+---------------+------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+------------------------+
| Name | IP | HostName | EditLogPort | HttpPort | QueryPort | RpcPort | Role | IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat | IsHelper | ErrMsg | Version |
+----------------------------------+---------------+------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+------------------------+
| 192.168.8.111_9010_1634120506159 | 192.168.8.111 | bigdata001 | 9010 | 7030 | 9030 | 9020 | FOLLOWER | true | 1895658427 | true | true | 4302 | 2021-10-13 22:20:58 | true | | 0.14.0-release-Unknown |
| 192.168.8.112_9010_1634134770528 | 192.168.8.112 | bigdata002 | 9010 | 7030 | 9030 | 9020 | FOLLOWER | false | 1895658427 | true | true | 4300 | 2021-10-13 22:20:58 | true | | 0.14.0-release-Unknown |
+----------------------------------+---------------+------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+------------------------+
2 rows in set (0.05 sec)
mysql>
3.4 新增一个BE
-
前面的步骤可以参考【安装第一个BE】，但注意be/conf/be.conf中的priority_networks参数，如在bigdata002上安装该参数就是
priority_networks = 192.168.8.112/24 -
数据会在几个小时到一天时间内，自动均衡到新增的BE上