docker集群部署doris

1. 前言

这篇文章主要讲述如何在 docker 环境下搭建 doris 集群，以及一些注意事项。开始前请参考下必备技能表

技能槽	熟练度
docker	熟悉
dockerfile	熟悉
docker-compose	熟悉
linux	熟悉
ctrl + c , ctrl + v	精通

如果已经具备以上技能，并满足对应的熟练度要求，那么请继续，反之去熟悉一下技能

2. 简介

引用官网一句话

支持对海量大数据进行快速分析的 MPP 数据库

官网地址：https://doris.apache.org

3. 编译

由于官方没有直接给出编译后的程序，所以只能自己编译后使用

个人猜测应该是在 apache 还处于孵化阶段的原因。

本教程使用官方比较推荐的 docker方式编译

3.1. 源码下载

前往github下载 Apache Doris 0.14.0 (incubating) Release 源码

3.2. 环境搭建

将源码解压至当前目录的source/0.14.0下

如对 docker-compose 比较了解的话，目录放哪都无所谓，只要能挂在到容器的 /root/source 即可

创建 build.yml并键入以下内容

version: "3.7"

networks:
  doris-build:
    name: doris-build
    driver: bridge

services:
  doris-build:
    image: apachedoris/doris-dev:build-env-1.2
    container_name: dev
    hostname: dev
    privileged: true
    command: ["source/0.14.0/build.sh"]
    volumes:
      - ./data/.m2:/root/.m2
      - ./source:/root/source
    networks:
      - doris-build

3.3. 编译源码

执行 docker-compose -f build.yml up，等待执行完毕，整个过程大概需要十分钟左右，因此第一次需要到mavan下载第三方库。

如果期间遇到Could not transfer artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh from/to xxx问题，根据 github 上的 PR 修改下 fe/pom.xml 的 cloudera 仓库

此时你会在源码的目录下得到output文件夹，目录结构如下

source/0.14.0/output/
├── be
│   ├── bin
│   ├── conf
│   ├── lib
│   └── www
├── fe
│   ├── bin
│   ├── conf
│   ├── lib
│   ├── spark-dpp
│   └── webroot
└── udf
    ├── include
    └── lib

4. 运行

鉴于编译环境用 docker，那么我们运行的时候也使用 docker

4.1. 制作镜像

新建Dockerfile并键入以下内容

FROM centos:7.5.1804

RUN \
	yum clean metadata && \
	yum -y install epel-release && \
	yum clean metadata && \
	yum -y install make which tar rpm-build yum-utils python-argparse python-yaml java-1.8.0-openjdk-devel gcc gcc-c|| createrepo jq glibc-static krb5-workstation openssh-clients && \
	yum clean all

ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk

RUN mkdir -p /home/doris

COPY ./source/0.14.0/output/fe /home/doris/fe

COPY ./source/0.14.0/output/be /home/doris/be

COPY ./source/0.14.0/fs_brokers/apache_hdfs_broker/output/apache_hdfs_broker /home/doris/fs_broker

运行docker build --pull --rm -f Dockerfile -t doris:0.14.00 .编译镜像

4.2. 编排容器

新建 docker-compose.yml并键入以下内容

version: "3.7"
networks:
  doris:
    name: doris
    driver: bridge

services:
  fe-leader:
    image: doris:0.14.00
    restart: always
    container_name: fe-leader
    hostname: fe-leader
    environment:
      - TZ=Asia/Shanghai
    command: /home/doris/fe/bin/start_fe.sh
    volumes:
      - ./doris/fe/leader/log:/home/doris/fe/log
      - ./doris/fe/leader/doris-meta:/home/doris/fe/doris-meta
      - ./doris/fe/leader/conf:/home/doris/fe/conf
      - /etc/localtime:/etc/localtime:ro
    stdin_open: true
    tty: true
    ports:
      - 8030:8030
      - 8040:8040
      - 9030:9030
    networks:
      - doris

  fe-follower-1:
    image: doris:0.14.00
    restart: always
    container_name: fe-follower-1
    hostname: fe-follower-1
    environment:
      - TZ=Asia/Shanghai
    command: /home/doris/fe/bin/start_fe.sh --helper fe-leader:9010
    volumes:
      - ./doris/fe/fe-follower-1/log:/home/doris/fe/log
      - ./doris/fe/fe-follower-1/doris-meta:/home/doris/fe/doris-meta
      - ./doris/fe/fe-follower-1/conf:/home/doris/fe/conf
      - /etc/localtime:/etc/localtime:ro
    stdin_open: true
    tty: true
    security_opt:
      - seccomp:unconfined
    privileged: true
    depends_on:
      - fe-leader
    networks:
      - doris

  fe-observer-1:
    image: doris:0.14.00
    restart: always
    container_name: fe-observer-1
    hostname: fe-observer-1
    environment:
      - TZ=Asia/Shanghai
    command: /home/doris/fe/bin/start_fe.sh --helper fe-leader:9010
    volumes:
      - ./doris/fe/fe-observer-1/log:/home/doris/fe/log
      - ./doris/fe/fe-observer-1/doris-meta:/home/doris/fe/doris-meta
      - ./doris/fe/fe-observer-1/conf:/home/doris/fe/conf

      - /etc/localtime:/etc/localtime:ro
    stdin_open: true
    tty: true
    security_opt:
      - seccomp:unconfined
    privileged: true
    depends_on:
      - fe-leader
    networks:
      - doris

  be:
    image: doris:0.14.00
    restart: always
    container_name: be
    hostname: be
    environment:
      - TZ=Asia/Shanghai
    command: /home/doris/be/bin/start_be.sh
    volumes:
      - "./doris/be/log:/home/doris/be/log"
      - "./doris/be/storage:/home/doris/be/storage"
      - "./doris/be/conf:/home/doris/be/conf/"
      - "/etc/localtime:/etc/localtime:ro"
    stdin_open: true
    tty: true
    security_opt:
      - seccomp:unconfined
    depends_on:
      - fe-leader
    networks:
      - doris

  mysql:
    image: "mysql:5.7"
    container_name: mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: "123456"
      MYSQL_USER: "root"
      MYSQL_PASS: "123456"
      TZ: Asia/Shanghai
    volumes:
      - ./doris/mysql:/var/lib/mysql
      - ./doris/mysql:/docker-entrypoint-initdb.d/
      - ./doris/mysql/sources.list:/etc/apt/sources.list
    depends_on:
      - fe-leader
    networks:
      - doris

4.3. 运行容器

1. 新建doris/fe文件夹并保持如下结构

doris/fe/
├── fe-follower-1
│   ├── conf
│   ├── doris-meta
│   └── log
├── fe-observer-1
│   ├── conf
│   ├── doris-meta
│   └── log
└── leader
    ├── conf
    ├── doris-meta
    └── log

2. 新建doris/be/conf文件夹
3. 复制前面编译出来的 output/fe/conf/fe.conf 到对应的 fe 的 conf 文件夹里
4. 复制output/be/conf/be.conf 到对应的 conf 文件夹里
5. 执行docker-compose up -d

5. 配置

当容器运行后，各容器之间是独立运行的，需要将follower、oberver、be添加到leader中才能形成集群，如果只不需要集群可以只添加be至leader，添加前后顺序可以随便

• leader follower oberver 相关资料请查阅官方文档，这里就不阐述了
• 当且必须在be被添加到leader后，可使用mysql数据库连接工具如navcat、 dbeaver连接到leader的9010端口，该端口是mysql协议，默认账号为root，密码为空

5.1. 添加 follower

执行docker exec -it mysql /bin/bash进入mysql容器

执行脚本将follower添加到leader中

mysql -h fe-leader -u root -P 9030 <<EOF
    ALTER SYSTEM ADD FOLLOWER "fe-follower-1:9010";
EOF

5.2. 添加 observer

同样在mysql容器内执行脚本

mysql -h fe-leader -u root -P 9030 <<EOF
    ALTER SYSTEM ADD OBSERVER "fe-observer-1:9010";
EOF

5.3. 添加 be

依旧在mysql容器内执行脚本

mysql -h fe-leader -u root -P 9030 <<EOF
    ALTER SYSTEM ADD BACKEND "be:9050";
EOF

5.4. 验证各容器运行状态

在mysql容器内执行脚本查看所有 fe 节点信息

mysql -h fe-leader -u root -P 9030 <<EOF
    SHOW PROC '/frontends';
EOF

输出内容如下

Name	IP	HostName	EditLogPort	HttpPort	QueryPort	RpcPort	Role	IsMaster	ClusterId	Join	Alive	ReplayedJournalId	LastHeartbeat	IsHelper	ErrMsg	Version
172.26.0.6_9010_1624243783898	172.26.0.6	fe-observer-1.doris	9010	8030	9030	9020	OBSERVER	false	1667475297	true	true	557	2021-06-21 11:00:55	false		0.14.0-release-Unknown
172.26.0.3_9010_1624243326588	172.26.0.3	fe-follower-1.doris	9010	8030	9030	9020	FOLLOWER	false	1667475297	true	true	557	2021-06-21 11:00:55	true		0.14.0-release-Unknown
172.26.0.2_9010_1624242600423	172.26.0.2	fe-leader	9010	8030	9030	9020	FOLLOWER	true	1667475297	true	true	558	2021-06-21 11:00:55	true		0.14.0-release-Unknown

如果Alive列为true则标识已成功添加到leader中，反之则失败，此时需要查看对应日志fe.log文件,根据具体信息解决

doris/fe/
├── fe-follower-1
│   ├── conf
│   │   ├── fe.conf
│   │   └── log4j2-spring.xml
│   ├── doris-meta
│   │   ├── bdb
│   │   └── image
│   └── log
│       ├── fe.audit.log
│       ├── fe.gc.log.20210621-103001
│       ├── fe.log
│       ├── fe.out
│       └── fe.warn.log
├── fe-observer-1
│   ├── conf
│   │   ├── fe.conf
│   │   └── log4j2-spring.xml
│   ├── doris-meta
│   │   ├── bdb
│   │   └── image
│   └── log
│       ├── fe.audit.log
│       ├── fe.gc.log.20210621-104708
│       ├── fe.gc.log.20210621-105052
│       ├── fe.log
│       ├── fe.out
│       └── fe.warn.log
└── leader
    ├── conf
    │   ├── fe.conf
    │   ├── log4j2-spring.xml
    │   ├── table1_data
    │   ├── table2_data
    │   ├── tools
    │   └── up.sh
    ├── doris-meta
    │   ├── bdb
    │   └── image
    └── log
        ├── fe.audit.log
        ├── fe.gc.log.20210621-102958
        ├── fe.log
        ├── fe.out
        └── fe.warn.log

在mysql容器内执行脚本查看所有 be 节点信息

mysql -h fe-leader -u root -P 9030 <<EOF
    SHOW PROC '/backends';
EOF

输出内容如下

BackendId	Cluster	IP	HostName	HeartbeatPort	BePort	HttpPort	BrpcPort	LastStartTime	LastHeartbeat	Alive	SystemDecommissioned	ClusterDecommissioned	TabletNum	DataUsedCapacity	AvailCapacity	TotalCapacity	UsedPct	MaxDiskUsedPct	ErrMsg	Version	Status
10004	default_cluster	172.26.0.4	be.doris	9050	9060	8040	8060	2021-06-21 10:40:42	2021-06-21 16:09:01	true	false	false	0	.000	140.327 GB	263.000 GB	46.64 %	46.64 %		0.14.0-release-Unknown	{“lastSuccessReportTabletsTime”:“2021-06-21 16:08:22”}

结语

不出意外，到这里你已经完成了 doris 的集群搭建了，至此本教程的使命也完成了。关于如何使用还需要查阅官方文档。

常见问题

1. wait catalog to be ready

fe启动后日志不断输出如下内容

2021-06-24 14:14:56,907 INFO (UNKNOWN 172.26.0.6_9010_1624243783898(-1)|1) [Catalog.waitForReady():785] wait catalog to be ready. FE type: UNKNOWN. is ready: false
2021-06-24 14:14:58,908 INFO (UNKNOWN 172.26.0.6_9010_1624243783898(-1)|1) [Catalog.waitForReady():785] wait catalog to be ready. FE type: UNKNOWN. is ready: false
2021-06-24 14:15:00,909 INFO (UNKNOWN 172.26.0.6_9010_1624243783898(-1)|1) [Catalog.waitForReady():785] wait catalog to be ready. FE type: UNKNOWN. is ready: false
2021-06-24 14:15:02,910 INFO (UNKNOWN 172.26.0.6_9010_1624243783898(-1)|1) [Catalog.waitForReady():785] wait catalog to be ready. FE type: UNKNOWN. is ready: false
2021-06-24 14:15:04,911 INFO (UNKNOWN 172.26.0.6_9010_1624243783898(-1)|1) [Catalog.waitForReady():785] wait catalog to be ready. FE type: UNKNOWN. is ready: false

解决方法

或者参考 github#5292 的解决方法

1. 清除非Master节点的元数据
2. 在Master节点的FE的配置文件fe.conf文件添加如下内容并重启所有FE

metadata_failure_recovery = true

3. 等待各节点都正常后启动并通过SHOW PROC '/frontends';确保每个节点的Join和Alive都为true后，注释掉第二步的设置，避免出现不必要的问题

2. Key columns should be a ordered prefix of the schema

建表时输出如下异常

Key columns should be a ordered prefix of the schema

建表语句如下

CREATE TABLE Location
(
CREATE TABLE `Location` (
  `Id` varchar(36) NOT NULL COMMENT "主键",
  `Latitude` int(11) NOT NULL COMMENT "纬度信息，以度为单位的纬度值乘以 10 的 6 次方，精确到百万分之一度",
  `Longitude` int(11) NOT NULL COMMENT "经度信息，以度为单位的经度值乘以 10 的 6 次方，精确到百万分之一度",
  `Altitude` smallint(6) NOT NULL COMMENT "高程（海拔高度），海拔高度，单位为米（m）",
  `Speed` smallint(6) NOT NULL COMMENT "速度，1/10km/h",
  `Direction` smallint(6) NOT NULL COMMENT "方向，0-359，正北为 0，顺时针",
  `DateTime` datetime NOT NULL COMMENT "终端上报时间",
  `SimNumber` varchar(15) NULL COMMENT "sim卡号",
  `IsBatch` boolean NOT NULL COMMENT "是否为补传消息",
  `CreateDateTime` datetime NOT NULL COMMENT "创建时间"
) ENGINE=OLAP
UNIQUE KEY(`Id`, `SimNumber`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`SimNumber`) BUCKETS 10
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);

解决方法

只需要修改语句中列的顺序即可

将KEY中的属性移至到前面，如本例中将SimNumber` varchar(15) NULL COMMENT "sim卡号",`移至Idvarchar(36) NOT NULL COMMENT "主键",后面即可

CREATE TABLE `Location` (
  `Id` varchar(36) NOT NULL COMMENT "主键",
  `SimNumber` varchar(15) NULL COMMENT "sim卡号",
  `Latitude` int(11) NOT NULL COMMENT "纬度信息，以度为单位的纬度值乘以 10 的 6 次方，精确到百万分之一度",
  `Longitude` int(11) NOT NULL COMMENT "经度信息，以度为单位的经度值乘以 10 的 6 次方，精确到百万分之一度",
  `Altitude` smallint(6) NOT NULL COMMENT "高程（海拔高度），海拔高度，单位为米（m）",
  `Speed` smallint(6) NOT NULL COMMENT "速度，1/10km/h",
  `Direction` smallint(6) NOT NULL COMMENT "方向，0-359，正北为 0，顺时针",
  `DateTime` datetime NOT NULL COMMENT "终端上报时间",
  `IsBatch` boolean NOT NULL COMMENT "是否为补传消息",
  `CreateDateTime` datetime NOT NULL COMMENT "创建时间"
) ENGINE=OLAP
UNIQUE KEY(`Id`, `SimNumber`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`SimNumber`) BUCKETS 10
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);

注意事项

• 容器最好使用静态IP方式，因为这玩意比较依赖IP
• 使用Stream load方式导入数据时请确保数据中没有包含'，否则会报value is null while columns is not nullable错误