手把手教你快速在生产环境搭建Doris集群附集群启停管理脚本

手把手教你快速在生产环境搭建Doris集群

一、部署规划

组件分布规划

节点	node4	node5	node6	node7	node8	Node9	Node10	Node11
服务	FE（follower） BE	FE（follower） BE	FE（follower） BE	FE（observer） BE	BE Broker	BE Broker	BE	BE
Ip	192.168.22.34	192.168.22.35	192.168.22.36	192.168.22.37	192.168.22.20	192.168.22.16	192.168.22.17	192.168.22.18

二、部署准备

安装包准备

根据自己集群的软硬件配置，选择合适的版本进行编译，这里我下载编译doris1.2.2版本源码，推荐使用Docker

https://doris.apache.org/zh-CN/docs/install/source-install/compilation

安装目录准备

1.部署目录规划

为了方便后期升级维护，数据存储，元数据和日志尽量不要放在安装包内。

FE	部署目录：/opt/module/doris/fe 日志目录：/var/log/doris/fe 元数据目录：/mnt/disk4/doris/meta
BE	部署目录：/opt/module/doris/be 日志目录：/var/log/doris/be 数据存储目录：/mnt/disk4/doris/storage，/mnt/disk5/doris/storage，/mnt/disk6/doris/storage
Broker	部署目录：/opt/module/doris/apache_hdfs_broker

2.数据存储目录创建

在BE上创建数据存储目录，根据规划node4到node11

[root@node4 hadoop]# mkdir -p /mnt/disk4/doris/storage
[root@node4 hadoop]# mkdir -p /mnt/disk5/doris/storage
[root@node4 hadoop]# mkdir -p /mnt/disk6/doris/storage

3.元数据存储目录创建

FE上需要创建元数据存储目录，node4到node7

[root@node4 hadoop]# mkdir -p /mnt/disk4/doris/meta

4.修改数据和元数据文件目录归属

管理Doris时，我们不能使用root用户直接管理，我们使用具有管理权限的hadoop用户进行管理

node4到node11，元数据和数据权限一起修改

[root@node4 hadoop]# chown -R hadoop:hadoop /mnt/disk4/doris
[root@node4 hadoop]# chown -R hadoop:hadoop /mnt/disk5/doris
[root@node4 hadoop]# chown -R hadoop:hadoop /mnt/disk6/doris

5.日志目录创建

BE的日志目录，node4-node11

[root@node4 hadoop]# mkdir -p /var/log/doris/be

FE的日志目录，node4-node7

[root@node4 hadoop]# mkdir -p /var/log/doris/fe

6.修改日志目录权限归属

[root@node4 hadoop]# chown -R hadoop:hadoop /var/log/doris

三、部署安装

配置文件

1.修改FE配置文件

vim /opt/module/doris-1.2.2/fe/conf/fe.conf

日志存储目录

LOG_DIR = /var/log/doris/fe

修改jvm参数，-Xmx调整为16G

JAVA_OPTS="-Xmx16384m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

元数据存储目录

meta_dir = /mnt/disk4/doris/meta

绑定ip

priority_networks = 192.168.22.34/24

2.修改be配置

配置java环境变量

由于从 1.2 版本开始支持 Java UDF 函数，BE 依赖于 Java 环境。所以要预先配置 JAVA_HOME 环境变量，也可以在 start_be.sh 启动脚本第一行添加 export JAVA_HOME=your_java_home_path 来添加环境变量。

vim /opt/module/doris-1.2.2/be/bin/start_be.sh

第一行添加

export JAVA_HOME=/usr/java/jdk

配置be，修改配置文件

vim /opt/module/doris-1.2.2/be/conf/be.conf

日志存储目录

 sys_log_dir = /var/log/doris/be

默认的8040端口和hadoop端口冲突了修改为18040

# ports for admin, web, heartbeat service 
be_port = 9060
webserver_port = 18040
heartbeat_service_port = 9050
brpc_port = 8060

修改绑定ip

priority_networks = 192.168.22.34/24

修改存储目录

路径和硬盘类型间用“，”隔开，多路径用“；”隔开，我这里时机械硬盘medium值为HDD，固态硬盘使用SSD

storage_root_path = /mnt/disk4/doris/storage,medium:HDD;/mnt/disk5/doris/storage,medium:HDD;/mnt/disk6/doris/storage,medium:HDD

3.Broker配置

把hdfs配置文件复制到Broker目录下

 mv hdfs-site.xml /opt/module/doris-1.2.2/apache_hdfs_broker/conf/

4.分发与修改配置

4.1.分发

到node4-node11

xsync /opt/module/doris-1.2.2

4.2.修改绑定ip

修改其余节点的绑定ip

node4-node7

vim /opt/module/doris-1.2.2/fe/conf/fe.conf

fe改为对应的绑定ip

...

be修改为对应绑定ip

node8-node11

 vim /opt/module/doris-1.2.2/be/conf/be.conf 

be修改为对应绑定ip

...

5.添加组件和启动集群

5.1.启动第一个FE

[hadoop@node4 doris]$ fe/bin/start_fe.sh --daemon

查看一下进出是否存在

[hadoop@node4 ~]$ jps | grep PaloFe
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
221813 dorisFE

5.2安装mysql客户端

集群安装mysql客户端

使用客户端连接Fe

mysql -h node4 -P 9030 -uroot

查看fe节点

mysql> show frontends;
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
| Name                              | IP             | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | IsMaster | ClusterId  | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | Version                  | CurrentConnected |
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
| 192.168.22.34_9010_1679021376875 | 192.168.22.34 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | true     | 1095009859 | true | true  | 144               | 2023-03-17 10:57:15 | true     |        | doris-1.2.2-rc01-Unknown | Yes              |
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+

第一次连接，没有密码，这里我们修改root密码

SET PASSWORD FOR 'root' = PASSWORD('000000');

5.3.添加其余的FE节点

ALTER SYSTEM ADD FOLLOWER "192.168.22.35:9010";
ALTER SYSTEM ADD FOLLOWER "192.168.22.36:9010";
ALTER SYSTEM ADD OBSERVER "192.168.22.37:9010";

再查看

show frontends

可以看到其他节点FE已经添加上了，只是连不上，因为我们还没有启动

mysql> show frontends;
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+-------+-------+-------------------+---------------------+----------+--------------------------------------------------------------+--------------------------+------------------+
| Name                              | IP             | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | IsMaster | ClusterId  | Join  | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg                                                       | Version                  | CurrentConnected |
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+-------+-------+-------------------+---------------------+----------+--------------------------------------------------------------+--------------------------+------------------+
| 192.168.22.36_9010_1679022090994 | 192.168.22.36 | 9010        | 8030     | 0         | 0       | FOLLOWER | false    | 1095009859 | false | false | 0                 | NULL                | true     | java.net.ConnectException: 拒绝连接 (Connection refused)     | NULL                     | No               |
| 192.168.22.34_9010_1679021376875 | 192.168.22.34 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | true     | 1095009859 | true  | true  | 232               | 2023-03-17 11:01:57 | true     |                                                              | doris-1.2.2-rc01-Unknown | Yes              |
| 192.168.22.35_9010_1679022090967 | 192.168.22.35 | 9010        | 8030     | 0         | 0       | FOLLOWER | false    | 1095009859 | false | false | 0                 | NULL                | true     | java.net.ConnectException: 拒绝连接 (Connection refused)     | NULL                     | No               |
| 192.168.22.37_9010_1679022093117 | 192.168.22.37 | 9010        | 8030     | 0         | 0       | OBSERVER | false    | 1095009859 | false | false | 0                 | NULL                | false    | java.net.ConnectException: 拒绝连接 (Connection refused)     | NULL                     | No               |
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+-------+-------+-------------------+---------------------+----------+--------------------------------------------------------------+--------------------------+------------------+
4 rows in set (0.02 sec)

5.4.启动其他FE

注意：第一次启动需指定–helper参数，后续再启动无需指定此参数

node5-node7

分别进入对应节点的fe目录

cd /opt/module/doris-1.2.2/fe

分别执行如下命令

bin/start_fe.sh --helper 192.168.22.34:9010 --daemon

分别查看FE进程

[hadoop@node7 fe]$ jps | grep  PaloFe
12644 PaloFe

第三次使用mysql客户端查看

show frontends

结果如下

mysql> show frontends;
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
| Name                              | IP             | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | IsMaster | ClusterId  | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | Version                  | CurrentConnected |
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
| 192.168.22.36_9010_1679022090994 | 192.168.22.36 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | false    | 1095009859 | true | true  | 357               | 2023-03-17 11:09:03 | true     |        | doris-1.2.2-rc01-Unknown | No               |
| 192.168.22.34_9010_1679021376875 | 192.168.22.34 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | true     | 1095009859 | true | true  | 359               | 2023-03-17 11:09:03 | true     |        | doris-1.2.2-rc01-Unknown | Yes              |
| 192.168.22.35_9010_1679022090967 | 192.168.22.35 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | false    | 1095009859 | true | true  | 357               | 2023-03-17 11:09:03 | true     |        | doris-1.2.2-rc01-Unknown | No               |
| 192.168.22.37_9010_1679022093117 | 192.168.22.37 | 9010        | 8030     | 9030      | 9020    | OBSERVER | false    | 1095009859 | true | true  | 357               | 2023-03-17 11:09:03 | false    |        | doris-1.2.2-rc01-Unknown | No               |
+-----------------------------------+----------------+-------------+----------+-----------+---------+----------+----------+------------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+

注意看4个FE全部处于激活状态

5.5.添加BE

mysql客户端执行对应的如下语句

ALTER SYSTEM ADD BACKEND "192.168.22.34:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.35:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.36:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.37:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.20:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.16:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.17:9050";
ALTER SYSTEM ADD BACKEND "192.168.22.18:9050";

6.6.启动其他BE

node4-node11

进入BE目录

cd /opt/module/doris/be

启动

/be/bin/start_be.sh --daemon

提示(若没有如下提示，跳过)

Please set vm.max_map_count to be 2000000 under root using 'sysctl -w vm.max_map_count=2000000'.

根据提示，分别在be节点执行

sysctl -w vm.max_map_count=2000000

再在各个be节点，启动be。

查看状况

mysql客户端

SHOW  backends;

mysql> SHOW  backends;
+-----------+-----------------+----------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+--------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| BackendId | Cluster         | IP             | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | RemoteUsedCapacity | Tag                      | ErrMsg | Version                  | Status                                                                                                                        | HeartbeatFailureCounter | NodeRole |
+-----------+-----------------+----------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+--------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| 11006     | default_cluster | 192.168.22.16 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:14:29 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 10.567 TB     | 10.911 TB     | 3.15 %  | 3.24 %         | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:37","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11007     | default_cluster | 192.168.22.17 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:14:29 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 10.575 TB     | 10.911 TB     | 3.08 %  | 3.11 %         | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11008     | default_cluster | 192.168.22.18 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:14:30 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 10.562 TB     | 10.911 TB     | 3.20 %  | 3.24 %         | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11001     | default_cluster | 192.168.22.34 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:09:12 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 7.731 TB      | 10.744 TB     | 28.05 % | 28.51 %        | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:25","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11002     | default_cluster | 192.168.22.35 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:09:13 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 7.832 TB      | 10.744 TB     | 27.11 % | 27.34 %        | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11003     | default_cluster | 192.168.22.36 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:09:12 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 8.204 TB      | 10.744 TB     | 23.64 % | 24.17 %        | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:41","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11004     | default_cluster | 192.168.22.37 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:09:14 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 8.424 TB      | 10.744 TB     | 21.59 % | 23.50 %        | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:33","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 11005     | default_cluster | 192.168.22.20 | 9050          | 9060   | 18040    | 8060     | 2023-03-17 13:09:16 | 2023-03-17 13:14:52 | true  | false                | false                 | 0         | 0.000            | 7.340 TB      | 10.744 TB     | 31.68 % | 31.77 %        | 0.000              | {"location" : "default"} |        | doris-1.2.2-rc01-Unknown | {"lastSuccessReportTabletsTime":"2023-03-17 13:14:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
+-----------+-----------------+----------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+--------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+

查看进程

[hadoop@node4 be]$ ps -ef | grep doris_be
hadoop    52260      1  0 17:23 pts/2    00:00:08 /opt/module/doris/be/lib/doris_be

6.7.添加Broker

mySql

ALTER SYSTEM ADD BROKER broker8 "192.168.22.38:8000";
ALTER SYSTEM ADD BROKER broker9 "192.168.22.39:8000";

6.8.启动broker

node8和node9上启动

[hadoop@node8 doris]$ cd apache_hdfs_broker/
[hadoop@node8 apache_hdfs_broker]$ bin/start_broker.sh --daemon

查看进程

[hadoop@node8 apache_hdfs_broker]$ jps | grep BrokerBootstrap
37003 BrokerBootstrap

6.9.编写集群启停脚本

群起脚本

#!/bin/bash

# use-method: starrocks.sh start|stop|restart

fe_hosts=(node4 node5 node6  node7)
be_hosts=(node4 node5 node6  node7 node8 node9 node10 node11)
broker_hosts=(node8 kafka1)

case $1 in

"start"){

    for i in ${fe_hosts[*]}
    do
        echo "=================== start $i's fe service ================"
        ssh $i "source /etc/profile.d/my_env.sh ;cd /opt/module/starRocks;./fe/bin/start_fe.sh --daemon"
    done


    for i in ${be_hosts[*]}
    do
        echo "=================== start $i's be  service ================"
        ssh $i "source /etc/profile.d/my_env.sh ;cd /opt/module/starRocks;./be/bin/start_be.sh --daemon"   
    done
	
	
	    for i in ${broker_hosts[*]}
    do
        echo "=================== start $i's  broker service ================" 
#        ssh $i "source /etc/profile.d/my_env.sh ;cd /opt/module/starRocks;./apache_hdfs_broker/bin/start_broker.sh --daemon"
    done

};;

"stop"){

    for i in ${broker_hosts[*]}
    do
        echo "=================== stop $i's broker service ================"
        ssh $i "/opt/module/starRocks/apache_hdfs_broker/bin/stop_broker.sh"
    done
	
	
	for i in ${be_hosts[*]}
    do
        echo "=================== stop $i's be service ================"
        ssh $i "/opt/module/starRocks/be/bin/stop_be.sh"
    done

    for i in ${fe_hosts[*]}
    do
        echo "=================== stop $i's fe service ================"
         ssh $i "/opt/module/starRocks/fe/bin/stop_fe.sh"
    done

};;

"status"){
    for i in ${fe_hosts[*]}
    do
        echo "=================== status $i's fe service ================"
         ssh $i "jps | grep StarRocksFE"
    done

    for i in ${be_hosts[*]}
    do
        echo "=================== status $i's be  service ================"
        ssh $i "ps -ef | grep starrocks_be | grep -v 'grep'"
    done

    for i in ${broker_hosts[*]}
    do
        echo "=================== status $i's broker service ================"
        ssh $i "jps | grep BrokerBootstrap | grep -v ‘JAVA_TOOL_OPTIONS’"
    done
};;

"restart")

 starrocks-cluster.sh stop

 sleep 2

 starrocks-cluster.sh start

;;

*)

 echo "Parameter ERROR!!! starrocks.sh start|stop|restart"

 ;;

esac

添加脚本执行权限

sudo +x doris-cluster.sh

6.10.web界面

be的web界面

http://node4.bigdata.59wanmei.com:18040/
http://node5.bigdata.59wanmei.com:18040/
http://node6.bigdata.59wanmei.com:18040/
http://node7.bigdata.59wanmei.com:18040/
。。。

fe的web界面

http://node4.bigdata.59wanmei.com:8030/home
http://node5.bigdata.59wanmei.com:8030/home
http://node6.bigdata.59wanmei.com:8030/home
http://node7.bigdata.59wanmei.com:8030/home

四、账号权限管理

1.添加一个账号

 mysql -h node4 -P 9030 -uroot -p'000000'

mysql客户端执行

set password=password('000000');

创建数据库

 create database test_db;

创建一个普通用户

CREATE USER test IDENTIFIED BY 'test000000';

给当前用户bus_log库的一个只读权限

GRANT SELECT_PRIV ON test_db.* TO test;

好了，集群搭建完成。