Flink Kafka Doris实战demo

Flink Kafka Doris实战demo

环境:

  • Flink 1.12
  • Doris 0.12
  • Kafka 1.0.1+kafka3.1.1

一:编译doris

参考官网Docker编译:https://github.com/apache/incubator-doris/wiki/Doris-Install

1.1 注意问题:

需要把fe/pom.xml中下载的Repository地址改下
cloudera-thirdparty
https://repository.cloudera.com/content/repositories/third-party/ 改为:

cloudera-public https://repository.cloudera.com/artifactory/public/

cloudera-plugins
https://repository.cloudera.com/content/groups/public/ 改为:

cloudera-public https://repository.cloudera.com/artifactory/public/

1.2 编译:

直接编译就行

1.3 如果不想编译,直接下载编译好的部署包(0.12,0.11版本)

https://download.csdn.net/download/leng91060404/16661347
https://download.csdn.net/download/leng91060404/16655427

二:部署doris(单机版测试)

参考:https://github.com/apache/incubator-doris/wiki/Doris-Install#3-%E9%83%A8%E7%BD%B2

2.1 注意

FE:

在fe目录下创建doris-meta目录;

根据需要修改priority_networks = ip 参数

BE:
修改参数storage_root_path = /home/disk1/doris;/home/disk2/doris;并创建目录;

根据需要修改priority_networks = ip 参数

2.2 启动

有问题直接查看log日志,进行分析解决

2.3 查看状态

BE节点需要先在FE中添加,才可加入集群。

可以使用 mysql-client 连接到 FE: ./mysql-client -h host-P port -uroot 其中 host 为 FE 所在节点 ip;port 为 fe/conf/fe.conf 中的 query_port;默认使用 root 账户,无密码登录。

FE状态查看:
用户可以通过 mysql 客户端登陆 Master FE。通过:
SHOW PROC '/frontends';
来查看当前 FE 的节点情况。
也可以通过前端页面连接:http://fe_hostname:fe_http_port/frontend 或者 http://fe_hostname:fe_http_port/system?path=//frontends 来查看 FE 节点的情况。
mysql> SHOW PROC '/frontends';
+--------------------------------+-------------+-----------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+
| Name                           | IP          | HostName  | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg |
+--------------------------------+-------------+-----------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+
| 172.16.3.76_9011_1618481783037 | xxxxxxxxx | xxxxxxxx | 9011        | 8030     | 9030      | 9020    | FOLLOWER | true     | 502881269 | true | true  | 24505             | 2021-04-16 16:03:57 | true     |        |
+--------------------------------+-------------+-----------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+
1 row in set (0.02 sec)
BE状态查看:
用户可以通过 mysql-client 登陆 Leader FE。通过:
SHOW PROC '/backends';
来查看当前 BE 的节点情况。
也可以通过前端页面连接:http://fe_hostname:fe_http_port/backend 或者 http://fe_hostname:fe_http_port/system?path=//backends 来查看 BE 节点的情况。
mysql> SHOW PROC '/backends';
+-----------+-----------------+-------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------------+
| BackendId | Cluster         | IP          | HostName  | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version             |
+-----------+-----------------+-------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------------+
| 10002     | default_cluster | xxxxxxxx | xxxxxx | 9050          | 9060   | 8040     | 8060     | 2021-04-15 18:17:38 | 2021-04-16 16:02:57 | true  | false                | false                 | 13        | 55.108 KB        | 64.740 GB     | 165.915 GB    | 60.98 % | 60.98 %        |        | 0.12.0-rc03-Unknown |
+-----------+-----------------+-------------+-----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------------+
1 row in set (0.00 sec)

三:flink程序(source:kafka、transfer:flink、sink:kafka)

3.1 数据实体类

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Item {

    private int id;
    private String name;
    private int counts;
    private Timestamp ts;

    @Override
    public String toString() {
        return "{" +
                "\"id\":" + "\"" + id + "\"" +
                ",\"name\":" + "\"" + name + "\"" +
                ",\"counts\":" + "\"" + counts + "\"" +
                ",\"ts\":" + "\"" + ts + "\"" +
                "}";

    }

3.2 模拟数据到source-kafka

{

        while (true) {
            Item item = generateItem();
            long startTime = System.currentTimeMillis();

            if (isAsync) {
                producer.send(new ProducerRecord(topic, item.toString()), new DemoCallBack(startTime, item.toString()));
                System.out.println("Sent message: (" + ", " + item.toString() + ")");
                try {
                    Thread.sleep(2000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            } else {
                try {
                    producer.send(new ProducerRecord(topic, item)).get();
                    System.out.println("Sent message: (" + ", " + item + ")");
                } catch (InterruptedException | ExecutionException e) {
                    e.printStackTrace();
                }
            }
        }
    }

数据:

{"id":"75392","name":"TIE4","counts":"77098","ts":"2021-04-16 11:40:48.423"}
{"id":"40904","name":"TIE3","counts":"26613","ts":"2021-04-16 11:40:52.436"}
{"id":"18363","name":"SHOE4","counts":"56329","ts":"2021-04-16 11:40:56.446"}
{"id":"17240","name":"TIE4","counts":"32312","ts":"2021-04-16 11:41:00.461"}

3.3 flink处理运行(source+transfer+sink)

sql:

public static final String KAFKA_SQL_EVEN_CREATE = "CREATE TABLE even (\n" +
            "  `id` INTEGER,\n" +
            "  `name` STRING,\n" +
            "  `counts` INTEGER,\n" +
            "  `ts` TIMESTAMP(3)," +
            " WATERMARK FOR ts as ts - INTERVAL '5' SECOND \n" +
            ") WITH (\n" +
            "  'connector' = 'kafka',\n" +
            "  'topic' = 'flink_sql_table_test_datas',\n" +
            "  'properties.bootstrap.servers' = 'xxxxxxxxxxxxx:9092',\n" +
            "  'properties.group.id' = 'testGroup',\n" +
            "  'scan.startup.mode' = 'earliest-offset',\n" +
            "  'format' = 'json'\n" +
            ")";

sql :

public static final String QUERY_EVEN_AGG_SQL =
            "SELECT\n"
                    + "  id as even_id,name as even_name,\n"
                    + "  COUNT(counts) even_cnt \n"
                    + "FROM even \n"
                    + "GROUP BY TUMBLE(ts, INTERVAL '20' SECOND),id,name";

deal :

{
        // set up execution environment
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().useBlinkPlanner().build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);

        //create table op
        tableEnv.executeSql(KAFKA_SQL_EVEN_CREATE);
        tableEnv.executeSql(KAFKA_SQL_ODD_CREATE);

        // windows aggregate count op
        Table resultEven = tableEnv.sqlQuery(QUERY_EVEN_AGG_SQL);
        resultEven.printSchema();

        Table resultOdd = tableEnv.sqlQuery(QUERY_ODD_AGG_SQL);
        resultOdd.printSchema();

        //sql join op
        Table joinTable = resultEven.join(resultOdd)
                .where(
                        $("even_name").isEqual($("odd_name")))
                .select(
                        $("even_id").as("id"),
                        $("even_name").as("name"),
                        $("even_cnt").as("counts"),
                        $("ts"));

        joinTable.printSchema();

        //sink to kafka 1
        DataStream<ItemSink> sinkResultStream = tableEnv.toAppendStream(joinTable, ItemSink.class);

        //kafka connector
        Properties props = new Properties();
        props.put("bootstrap.servers", KafkaProperties.KAFKA_SERVER_URL + ":" + KafkaProperties.KAFKA_SERVER_PORT);
        props.put("client.id", KafkaProperties.CLIENT_ID);

        FlinkKafkaProducer<ItemSink> myProducer = new FlinkKafkaProducer<>(
                KafkaProperties.TOPIC_SINK,
                new ProducerStringSerializationSchema(KafkaProperties.TOPIC_SINK),
                props,
                FlinkKafkaProducer.Semantic.EXACTLY_ONCE);

        sinkResultStream.addSink(myProducer);

        //run
        env.execute("Streaming Window SQL Job");

    }

sink-kafka数据

41219,HAT9,1,2021-04-16 11:32:08.524
41219,HAT9,1,2021-04-16 11:33:30.846
41219,HAT9,1,2021-04-16 11:40:36.387
41219,HAT9,1,2021-04-14 19:21:46.738

四:kafka load to doris(flink上面已经sink到kafka)

4.1 建库

CREATE DATABASE example_db;

4.2 建表

CREATE TABLE item
(
    id INTEGER,
    name VARCHAR(256) DEFAULT '',
    ts DATETIME,
    counts BIGINT SUM DEFAULT '0'
)
    AGGREGATE KEY(id, name, ts)
DISTRIBUTED BY HASH(id) BUCKETS 3
PROPERTIES("replication_num" = "1");

4.3 加载Kafka2Doris任务

CREATE ROUTINE LOAD example_db.task1 ON item
        COLUMNS TERMINATED BY ",",
        COLUMNS(id, name, counts, ts)
        PROPERTIES
        (
            "desired_concurrent_number"="1",
            "max_batch_interval" = "20",
            "max_batch_rows" = "300000",
            "max_batch_size" = "209715200",
            "strict_mode" = "false"
        )
        FROM KAFKA
        (
            "kafka_broker_list" = "xxxxxxx:9092",
            "kafka_topic" = "sinkTopic",
            "property.group.id" = "1234",
            "property.client.id" = "12345",
             "kafka_partitions" = "0,1,2",
            "kafka_offsets" = "0,0,0"
        );

4.4 查看任务

SHOW ROUTINE LOAD FOR example_db.task1;
mysql> SHOW ROUTINE LOAD FOR example_db.task1;
+-------+-------+---------------------+-----------+---------+--------------------------------+-----------+---------+----------------+----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+----------------------+--------------+----------+
| Id    | Name  | CreateTime          | PauseTime | EndTime | DbName                         | TableName | State   | DataSourceType | CurrentTaskNum | JobProperties                                                                                                                                                                                                                         | DataSourceProperties                                                                                    | CustomProperties                        | Statistic                                                                                                                                                                                                     | Progress                           | ReasonOfStateChanged | ErrorLogUrls | OtherMsg |
+-------+-------+---------------------+-----------+---------+--------------------------------+-----------+---------+----------------+----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+----------------------+--------------+----------+
| 10037 | task1 | 2021-04-16 11:27:39 | N/A       | N/A     | default_cluster:flink_kafka_db | item      | RUNNING | KAFKA          | 1              | {"partitions":"*","columnToColumnExpr":"id,name,counts,ts","maxBatchIntervalS":"20","whereExpr":"*","maxBatchSizeBytes":"209715200","columnSeparator":"','","maxErrorNum":"0","currentTaskConcurrentNum":"1","maxBatchRows":"300000"} | {"topic":"flink_sql_table_test_sink","currentKafkaPartitions":"0,1,2","brokerList":"172.16.2.148:9092"} | {"group.id":"1234","client.id":"12345"} | {"receivedBytes":726452,"errorRows":0,"committedTaskNum":159,"loadedRows":20118,"loadRowsRate":0,"abortedTaskNum":629,"totalRows":20118,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":3247070} | {"0":"6059","1":"6915","2":"7141"} |                      |              |          |
+-------+-------+---------------------+-----------+---------+--------------------------------+-----------+---------+----------------+----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+----------------------+--------------+----------+
1 row in set (0.00 sec)
SHOW ROUTINE LOAD TASK WHERE JobName = "task1";
mysql> SHOW ROUTINE LOAD TASK WHERE JobName = "task1";
+-----------------------------------+-------+-----------+-------+---------------------+---------------------+---------+-------+------------------------------+
| TaskId                            | TxnId | TxnStatus | JobId | CreateTime          | ExecuteStartTime    | Timeout | BeId  | DataSourceProperties         |
+-----------------------------------+-------+-----------+-------+---------------------+---------------------+---------+-------+------------------------------+
| 56543bac022d4fdb-bb5944617d6480ac | 788   | UNKNOWN   | 10037 | 2021-04-16 15:56:50 | 2021-04-16 15:56:50 | 40      | 10002 | {"0":6060,"1":6916,"2":7142} |
+-----------------------------------+-------+-----------+-------+---------------------+---------------------+---------+-------+------------------------------+

4.5 通过mysql查看数据

./mysql-client -h host -P port -uroot
mysql> select * from item order by id desc limit 10;
+-------+-------+---------------------+--------+
| id    | name  | ts                  | counts |
+-------+-------+---------------------+--------+
| 99995 | SHOE2 | 2021-04-16 11:34:45 |      3 |
| 99995 | SHOE2 | 2021-04-15 19:05:59 |      4 |
| 99995 | SHOE2 | 2021-04-16 11:35:59 |      3 |
| 99995 | SHOE2 | 2021-04-16 11:35:05 |      3 |
| 99995 | SHOE2 | 2021-04-14 19:22:06 |      4 |
| 99995 | SHOE2 | 2021-04-16 11:35:57 |      3 |
| 99995 | SHOE2 | 2021-04-16 11:38:45 |      3 |
| 99995 | SHOE2 | 2021-04-16 11:38:11 |      3 |
| 99995 | SHOE2 | 2021-04-14 19:22:24 |      4 |
| 99995 | SHOE2 | 2021-04-14 19:21:12 |      4 |
+-------+-------+---------------------+--------+
10 rows in set (0.01 sec)

五:demo代码参考

https://gitee.com/suntyu_admin/flink-kafka-doris

你可能感兴趣的:(Fink,大数据,Doris,flink,kafka)