Pg-replicate-pipelineDB: Postgresql 数据流实时计算


Postgresql版本:10

PipelineDB安装参考:https://blog.csdn.net/u012551524/article/details/86705632

Pg-replicate-pipelineDB下载安装参考:https://github.com/TracyGao01/pg-replicate-pipelineDB


目的:

实时计算写入Postgresql的数据,做统计展现

解决:

通过Pg-replicate-pipelineDB插件将数据实时写入pipelineDB,由pipelineDB实时计算并输出

安装:

1、安装 Wal2json:

git clone https://github.com/eulerto/wal2json make && make install

2、修改 postgresql postgresql.conf,添加如下配置

shared_preload_libraries = 'wal2json'

wal_level = logical

max_replication_slots = 4

3、重启Postgresql

4、安装Pg-replicate-pipelineDB,启动

  • 下载Pg-replicate-pipelineDB zip包,并解压(路径:https://github.com/TracyGao01/pg-replicate-pipelineDB)

  • 安装:python setup.py install

  • 启动: pg_replicate_pipelinedb --config=/opt/software/pg-replicate-pipelineDB/1.json

    • 参数 ‘--config=/opt/software/pg-replicate-pipelineDB/1.json‘ 内容如下

       {#Config postgresql logic replication slot

      "replication_slot": {

      "name": "pipelinedb_slot_1",  #name

      "is_temp": false    # is temp

      },

      #Config 需要同步的表

      "tables": [  

      {

                      "name": "test2",

                      "primary_key": "id",

                      "exclude_columns": ""  # 过滤掉不需要的字段

              }],

      #Config pipelineDB connector

      "pipelinedb": {

      "port": 5432,

      "host": "10.0.10.68",

      "database": "df",

      "username": "postgres",

      "password": "postgres"

      },

      #Config postgresql connector

      "postgres": {

      "port": 5432,

      "host": "10.0.10.68",

      "database": "postgres",

      "username": "postgres",

      "password": "postgres"

      },

      "inital_sync": false }

  • 启动成功后如下图,可看到数据成功插入pipelineDB

你可能感兴趣的:(PipelineDB,python,Postgresql)