Etcd - 分布式配置中心
Etcd 简介
Etcd 是一种分布式 kv 存储设施, 他具有一定的一致性,高性能,高可用的方案. 类似的 zookeeper, 但没有 zookeeper 那么重型,功能也没有覆盖那么多. 简单直接的应用就是配置中心
架构设计总览
clients 为多个需要 配置的服务, 中间层为 多个 grpc-proxy 做均衡负载, 以免一个 proxy 挂了之后 导致单点问题. grpc-proxy 可以 同时访问多个 etcd 服务器,进行 kv 的操作. 如果某一个 server 挂了,会自动访问别的 集群中的其他 server 以保证高可用.
etcd cluster 至少为 3 台, 如果小于 3 台则无法进行选举,造成集群 不可用. (这里需要用 promethus 进行监控预警)
etcd cluster 本地搭建
- 本地安装 etcd,或者直接下载预编译好的执行文件
https://github.com/etcd-io/etcd/releases - 获取 etcd 源码 (其实只是要一个配置文件 Procfile)
- 安装 goreman -> go get github.com/mattn/goreman
- 修改源码根目录的 Procfile
# Use goreman to run `go get github.com/mattn/goreman` etcd1: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd2: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd3: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd4: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra4 --listen-client-urls http://127.0.0.1:42379 --advertise-client-urls http://127.0.0.1:42379 --listen-peer-urls http://127.0.0.1:42380 --initial-advertise-peer-urls http://127.0.0.1:42380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr etcd5: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra5 --listen-client-urls http://127.0.0.1:52379 --advertise-client-urls http://127.0.0.1:52379 --listen-peer-urls http://127.0.0.1:52380 --initial-advertise-peer-urls http://127.0.0.1:52380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr proxy1: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd grpc-proxy start --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 --listen-addr=127.0.0.1:23790 --advertise-client-url=127.0.0.1:23790 --enable-pprof proxy2: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd grpc-proxy start --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 --listen-addr=127.0.0.1:23791 --advertise-client-url=127.0.0.1:23791 --enable-pprof |
- 启动集群 goreman -f Procfile start
配置解读
配置了 5 个 etcd 服务模拟不同的 server, 配置了 2 个 proxy, 模拟两个 grpc-proxy 当做均衡负载
–listen-client-urls etcd 监听的地址
–listen-peer-urls 集群内部 member 互相监听的地址
–name 集群成员的名字
–advertise-client-urls 集群内成员对外发布信息的地址
–initial-advertise-peer-urls 对集群内发布信息的地址
–initial-cluster-state Set to new for all members present during initial static or DNS bootstrapping. If this option is set to existing, etcd will attempt to join the existing cluster. If the wrong value is set, etcd will attempt to start but fail safely.
–initial-cluster-token 集群的标示
5 个 etcd server 形成 etcd-cluster-1, 2 个 proxy 对这个集群进行代理
集群检查
ETCDCTL_API=3 ./etcdctl endpoint –endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 status -w table
λ ETCDCTL_API=3 ./etcdctl endpoint --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 status -w table +-----------------+------------------+---------+---------+-----------+-----------+------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | +-----------------+------------------+---------+---------+-----------+-----------+------------+ | 127.0.0.1:2379 | 8211f1d0f64f3269 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:22379 | 91bc3c398fb3c146 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:32379 | fd422379fda50e48 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:42379 | 45d559f8148de837 | 3.3.9 | 25 kB | false | 2 | 12 | | 127.0.0.1:52379 | c91263fe1e1dd3b5 | 3.3.9 | 25 kB | true | 2 | 12 | +-----------------+------------------+---------+---------+-----------+-----------+------------+ |
λ ./etcdctl cluster-health member 45d559f8148de837 is healthy: got healthy result from http://127.0.0.1:42379 member 8211f1d0f64f3269 is healthy: got healthy result from http://127.0.0.1:2379 member 91bc3c398fb3c146 is healthy: got healthy result from http://127.0.0.1:22379 member c91263fe1e1dd3b5 is healthy: got healthy result from http://127.0.0.1:52379 member fd422379fda50e48 is healthy: got healthy result from http://127.0.0.1:32379 cluster is healthy |
编写测试客户端进行 kv 操作
go get go.etcd.io/etcd
package main
import (
"context"
"fmt"
"log"
"strconv"
"time"
"go.etcd.io/etcd/clientv3"
"vincent.com/etcd/etcd/etcdserver/api/v3rpc/rpctypes"
)
func main() {
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://127.0.0.1:23790", "http://127.0.0.1:23791"},
DialTimeout: 5 * time.Second,
})
if err != nil {
// handle error!
log.Fatalln(err.Error())
}
defer cli.Close()
defer cli.Delete(context.Background(), "sample_key")
for index := 0; index < 10000; index++ {
insertKV(cli, "sample_key", "sample_value"+strconv.Itoa(index))
if err != nil {
break
}
}
resp, err := cli.Get(context.Background(), "sample_key")
if err != nil {
log.Fatalln("get key error", err)
}
fmt.Printf("get the sample_key: %v\n", resp.Kvs)
}
func insertKV(cli *clientv3.Client, key string, value string) (err error) {
time.Sleep(2 * time.Millisecond)
_, err = cli.Put(context.Background(), key, value)
// cancel()
if err != nil {
switch err {
case context.Canceled:
log.Fatalf("ctx is canceled by another routine: %v", err)
case context.DeadlineExceeded:
log.Fatalf("ctx is attached with a deadline is exceeded: %v", err)
case rpctypes.ErrEmptyKey:
log.Fatalf("client-side error: %v", err)
default:
log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err)
}
}
return
}
|
客户端解读
etcd 客户端支持集群, 所以直接可以连接两个 proxy
这里进行了 1w 此的写入,每次写入会有 x ms 的延迟(我怕我的 mbp 受不了)
客户端测试
- 运行客户端程序
- 模拟 etcd server 挂掉 goreman run stop etcd2
- 当集群内健康机器少于 3 台的时候,客户端报错,集群整体不可用
集群性能检测
λ ETCDCTL_API=3 ./etcdctl check perf load="l" --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s PASS: Throughput is 150 writes/s PASS: Slowest request took 0.388842s PASS: Stddev is 0.029037s PASS |
注意:
如果用命令行,记得加 ETCDCTL_API=3
#Golang #Etcd
golang-uuidv1
Etcd - 分布式配置中心2 - runtime切换配置