时序数据库m3db集群二进制部署

一、m3db集群二进制部署

1、环境说明

1）存储节点（包含种子节点）

172.24.2.63~65

2）协调器节点

172.24.2.62

2、安装前的准备（依次在所有节点操作）

1）关闭防火墙

$ systemctl status firewalld
$ systemctl stop firewalld
$ systemctl disable firewalld
$ systemctl status firewalld

2）关闭SELinux

$ vi /etc/selinux/config
SELINUX=disabled

3）配置时间同步

$ date -s "20200222 21:53:00"(由于云桌面环境没有时钟服务器，当时用的是手动同步，即手动统一设置为当前时间）

4）内核内置

$ sysctl -w vm.max_map_count=3000000
$ sysctl -w vm.swappiness=1
$ sysctl -w fs.file-max=3000000
$ sysctl -w fs.nr_open=3000000

5）建立缓存目录

$ mkdir -p /var/lib/m3kv

6）安装jq

$ yum install jq

3、存储节点安装（可选存储节点任意一台操作，这里选172.24.2.63）

1）二进制包安装

a）下载解压

下载地址

https://github.com/m3db/m3/releases

解压

$ tar -xvf m3_0.14.2_linux_amd64.tar.gz

b）执行文件拷贝

$ cp m3_0.14.2_linux_amd64/m3dbnode /usr/local/bin

$ scp m3_0.14.2_linux_amd64/m3dbnode [email protected]:/usr/local/bin
$ scp m3_0.14.2_linux_amd64/m3dbnode [email protected]:/usr/local/bin

2）配置文件

a）准备

$ mkdir -p m3db/config

$ vi m3db/config/m3dbnode.yml
coordinator:
  listenAddress:
    value: "0.0.0.0:7201"

  local:
    namespaces:
      - namespace: default
        type: unaggregated
        retention: 48h

  logging:
    level: info

  metrics:
    scope:
      prefix: "coordinator"
    prometheus:
      handlerPath: /metrics
      listenAddress: 0.0.0.0:7203 # until https://github.com/m3db/m3/issues/682 is resolved
    sanitization: prometheus
    samplingRate: 1.0
    extended: none

  tagOptions:
    # Configuration setting for generating metric IDs from tags.
    idScheme: quoted

db:
  logging:
    level: info

  metrics:
    prometheus:
      handlerPath: /metrics
    sanitization: prometheus
    samplingRate: 1.0
    extended: detailed

  hostID:
    resolver: hostname

# Fill-out the following and un-comment before using.
  config:
    service:
      env: default_env
      zone: embedded
      service: m3db
      cacheDir: /var/lib/m3kv
      etcdClusters:
        - zone: embedded
          endpoints:
            - 172.24.2.63:2379
            - 172.24.2.64:2379
            - 172.24.2.65:2379
    seedNodes:
      initialCluster:
        - hostID: chl-servmesh-test1
          endpoint: http://172.24.2.63:2380
        - hostID: chl-servmesh-test2
          endpoint: http://172.24.2.64:2380
        - hostID: chl-servmesh-test3
          endpoint: http://172.24.2.65:2380

  listenAddress: 0.0.0.0:9000
  clusterListenAddress: 0.0.0.0:9001
  httpNodeListenAddress: 0.0.0.0:9002
  httpClusterListenAddress: 0.0.0.0:9003
  debugListenAddress: 0.0.0.0:9004

  client:
    writeConsistencyLevel: majority
    readConsistencyLevel: unstrict_majority

  gcPercentage: 100

  writeNewSeriesAsync: true
  writeNewSeriesLimitPerSecond: 1048576
  writeNewSeriesBackoffDuration: 2ms

  bootstrap:
    bootstrappers:
        - filesystem
        - commitlog
        - peers
        - uninitialized_topology
    commitlog:
      returnUnfulfilledForCorruptCommitLogFiles: false

  cache:
    series:
      policy: lru
    postingsList:
      size: 262144

  commitlog:
    flushMaxBytes: 524288
    flushEvery: 1s
    queue:
      calculationType: fixed
      size: 2097152

  fs:
    filePathPrefix: /var/lib/m3db

b）拷贝

$ cp m3db/config/m3dbnode.yml /etc/m3db

$ scp m3db/config/m3dbnode.yml [email protected]:/etc/m3db
$ scp m3db/config/m3dbnode.yml [email protected]:/etc/m3db

c）建立存储数据目录（依次在存储节点操作）

$ mkdir -p /var/lib/m3db

3）自启动服务

a）准备

$ mkdir -p m3db/systemd

$ vi m3db/systemd/m3dbnode.service
[Unit]
Description="M3DB Timeseries Database"
Documentation=http://m3db.github.io/m3/
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/m3dbnode -f /etc/m3db/m3dbnode.yml
Restart=on-failure
RestartSec=10s
SuccessExitStatus=0

# May not be honored if higher than kernel limit (sysctl fs.file-max) or process
# limit (sysctl fs.nr_open). Also may not be honored if lower than systemd limit
# (system.conf) or systemd user limit (user.conf).
LimitNOFILE=3000000

[Install]
WantedBy=multi-user.target

b）拷贝

$ cp m3db/systemd/m3dbnode.service /usr/lib/systemd/system

$ scp m3db/systemd/m3dbnode.service [email protected]:/usr/lib/systemd/system
$ scp m3db/systemd/m3dbnode.service [email protected]:/usr/lib/systemd/system

4）启动存储节点（依次在存储节点操作）

$ systemctl daemon-reload

$ systemctl enable m3dbnode
$ systemctl start m3dbnode
$ systemctl status m3dbnode

5）创建命名空间和初始化拓扑

$ curl -X POST http://localhost:7201/api/v1/database/create -d '{
  "type": "cluster",
  "namespaceName": "prom_metics",
  "retentionTime": "48h",
  "numShards": "1024",
  "replicationFactor": "3",
  "hosts": [
        {
            "id": "chl-servmesh-test1",
            "isolationGroup": "us-east1-a",
            "zone": "embedded",
            "weight": 100,
            "address": "172.24.2.63",
            "port": 9000
        },
        {
            "id": "chl-servmesh-test2",
            "isolationGroup": "us-east1-b",
            "zone": "embedded",
            "weight": 100,
            "address": "172.24.2.64",
            "port": 9000
        },
        {
            "id": "chl-servmesh-test3",
            "isolationGroup": "us-east1-c",
            "zone": "embedded",
            "weight": 100,
            "address": "172.24.2.65",
            "port": 9000
        }
    ]
}'

$ curl http://localhost:7201/api/v1/placement | jq .

6）测试验证

$ curl -sS -X POST localhost:9003/writetagged -d '{
  "namespace": "prom_metics",
  "id": "foo",
  "tags": [
    {
      "name": "city",
      "value": "new_york"
    },
    {
      "name": "endpoint",
      "value": "/request"
    }
  ],
  "datapoint": {
    "timestamp": '"$(date "+%s")"',
    "value": 42.123456789
  }
}'

$ curl -sS -X POST http://localhost:9003/query -d '{
  "namespace": "prom_metics",
  "query": {
    "regexp": {
      "field": "city",
      "regexp": ".*"
    }
  },
  "rangeStart": 0,
  "rangeEnd": '"$(date "+%s")"'
}' | jq .

4、协调器节点安装（在协调器节点操作，这里是172.24.2.62，一般以Sidecar形式和Prometheus部在一起）

1）二进制包安装

a）解压

$ tar -xvf m3_0.14.2_linux_amd64.tar.gz

b）执行文件拷贝

$ cp m3_0.14.2_linux_amd64/m3coordinator /usr/local/bin

2）配置文件

a）准备

$ mkdir -p m3db/config

$ vi m3db/config/m3coordinator.yml
listenAddress:
  value: "0.0.0.0:7201"

logging:
  level: info

metrics:
  scope:
    prefix: "coordinator"
  prometheus:
    handlerPath: /metrics
    listenAddress: 0.0.0.0:7203 # until https://github.com/m3db/m3/issues/682 is resolved
  sanitization: prometheus
  samplingRate: 1.0
  extended: none

tagOptions:
  idScheme: quoted

clusters:
## Fill-out the following and un-comment before using, and
## make sure indent by two spaces is applied.
   - namespaces:
       - namespace: prom_metics
         retention: 48h
         type: unaggregated
     client:
       config:
         service:
           env: default_env
           zone: embedded
           service: m3db
           cacheDir: /var/lib/m3kv
           etcdClusters:
             - zone: embedded
               endpoints:
                 - 172.24.2.63:2379
                 - 172.24.2.64:2379
                 - 172.24.2.65:2379
##                ... etc, list only M3DB seed nodes
       writeConsistencyLevel: majority
       readConsistencyLevel: unstrict_majority

b）拷贝

$ mkdir -p /etc/m3coordinator

$ cp m3db/config/m3coordinator.yml /etc/m3coordinator

3）自启动服务

a）准备

$ mkdir -p m3db/systemd

$ vi m3db/systemd/m3coordinator.service
[Unit]
Description="M3 Coordinator"
Documentation=http://m3db.github.io/m3/
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/m3coordinator -f /etc/m3coordinator/m3coordinator.yml
Restart=on-failure
RestartSec=10s
SuccessExitStatus=0

# May not be honored if higher than kernel limit (sysctl fs.file-max) or process
# limit (sysctl fs.nr_open). Also may not be honored if lower than systemd limit
# (system.conf) or systemd user limit (user.conf).
LimitNOFILE=3000000

[Install]
WantedBy=multi-user.target

b）拷贝

$ cp m3db/systemd/m3coordinator.service /usr/lib/systemd/system

4）启动协调器节点

$ systemctl daemon-reload

$ systemctl enable m3coordinator
$ systemctl start m3coordinator
$ systemctl status m3coordinator

三、与Prometheus集成

在prometheus.yml文件中加入如下配置

remote_read:
  - url: "http://172.24.2.62:7201/api/v1/prom/remote/read"
    # To test reading even when local Prometheus has the data
    read_recent: true
remote_write:
  - url: "http://172.24.2.62:7201/api/v1/prom/remote/write"