流计算开发平台StreamPark安装

一、StreamPark简介

StreamPark（原StreamX）是一个流处理应用程序开发管理框架。初衷是让流处理更简单，旨在轻松构建和管理流处理应用程序，提供使用 Apache Flink 和 Apache Spark 编写流处理应用程序的开发框架，未来将支持更多其他引擎。同时，StreamPark 提供了一个流处理应用管理平台，核心能力包括但不限于应用开发、调试、交互查询、部署、运维、实时数仓等。

官方网站：https://streampark.apache.org

二、快速安装使用

1. 使用docker-compose安装

1.1 安装docker

首先要安装docker，要求如下：

Docker 1.13.1+
Docker Compose 1.28.0+

1.2 安装docker-compose

docker-compose安装方式有很多，这里在尝试pip安装后，启动StreamPark后频繁报错，所以推荐二进制安装方式。

• 首先下载docker-compose文件，使用daocloud下载会快点，安装最新稳定版v2.16.0

  sudo curl -L https://get.daocloud.io/docker/compose/releases/download/2.16.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
  

• 接着将docker-compose添加可执行权限

  sudo chmod +x /usr/local/bin/docker-compose
  

• 查询版本

  docker-compose --version
  

###1.3 下载docker-compose.yaml和.env

官方给的指令：

wget https://github.com/apache/incubator-streampark/blob/dev/deploy/docker/docker-compose.yaml
wget https://github.com/apache/incubator-streampark/blob/dev/deploy/docker/.env

这里我下载到的文件里面是html代码，且下载麻烦，下面直接给文件内容，创建两个文件就行：

sudo mkdir -p streampark
sudo vim docker-compose.yaml

• docker-compose.yaml

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

version: '3.8'
services:
  streampark-console:
    image: apache/streampark:latest
    command: ${RUN_COMMAND}
    ports:
      - 10000:10000
    env_file: .env
    volumes:
      - flink:/streampark/flink/${FLINK}
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/hosts:/etc/hosts:ro
      - ~/.kube:/root/.kube:ro
    privileged: true
    restart: unless-stopped
    networks:
      - streampark

  flink-jobmanager:
    image: ${FLINK_IMAGE}
    ports:
      - "8081:8081"
    command: jobmanager
    volumes:
      - flink:/opt/flink
    env_file: .env
    restart: unless-stopped
    privileged: true
    networks:
      - streampark

  flink-taskmanager:
    image: ${FLINK_IMAGE}
    depends_on:
      - flink-jobmanager
    command: taskmanager
    deploy:
      replicas: 1
    env_file: .env
    restart: unless-stopped
    privileged: true
    networks:
      - streampark

networks:
  streampark:
    driver: bridge

volumes:
  flink:

这里要注意按照自己的需求调整配置，特别注意端口不能和现有服务端口冲突

• .env

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

TZ=Asia/Shanghai

SPRING_PROFILES_ACTIVE=h2 #mysql, pgsql
# If use mysql or pgsql, please set the following parameters
#SPRING_DATASOURCE_URL=jdbc:mysql://localhost:3306/streampark?useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8
#SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/streampark?stringtype=unspecified
#SPRING_DATASOURCE_USERNAME=root
#SPRING_DATASOURCE_PASSWORD=streampark

FLINK=flink1.14.5
FLINK_IMAGE=flink:1.14.5-scala_2.12

RUN_COMMAND='/bin/sh -c "wget -P lib https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.0.31/mysql-connector-j-8.0.31.jar && bash bin/startup.sh"'

JOB_MANAGER_RPC_ADDRESS=flink-jobmanager

1.4 启动容器

docker-compose up -d

1.5 查看服务

• 访问StreamPark http://yourip:10000，这里的yourip是指宿主机的ip
  • 默认账号和密码是admin 和 streampark
• 访问StreamPark http://yourip:8081，同上
• 这里的端口是和配置文件中的一致的，如果有改动，改成对应的端口即可

1.6 快速开始

请见官方文档：[StreamPark-快速开始](https://streampark.apache.org/zh-CN/docs/user-guide/quick-start)

2. 基于源码构建镜像进行StreamPark部署

未完待续。

三、常见报错

未完待续。。。

报错记录不在，后续有空更新