PMM初体验
一、监控服务端
1、安装docker
yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine
yum install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce
systemctl start docker
systemctl enable docker
2、配置docker存储目录、国内docker源
配置文件：/etc/docker/daemon.json
{
"data-root": "/data/docker",
"registry-mirrors":["https://registry.docker-cn.com"]
}data-root按需配置，默认在/var/lib/docker
systemctl daemon-reload
systemctl restart docker
3、拉取PMM Server镜像
docker pull percona/pmm-server:1.17.3
附：
查看已经拉取的镜像清单：
docker images
删除指定IMAGE ID的镜像：
docker rmi ada7dbc5a8bc
注：删除镜像前，先删除对应容器
docker ps -a
docker rm 容器ID
4、创建容器
创建一个容器用来保存PMM Server数据
docker create --name pmm-data percona/pmm-server:1.17.3 /bin/true5、运行
运行以下命令：
docker run -d \
-p 8080:80 \
--volumes-from pmm-data \
--name pmm-server \
-e ORCHESTRATOR_ENABLED=true \
-e DISABLE_TELEMETRY=true \
--storage-opt size=30G \
--restart always \
percona/pmm-server:1.17.3docker run的额外功能：
--storage-opt size=30G，默认10G，避免不够用再扩容
添加 “-e ORCHESTRATOR_ENABLED=true \”，启用Orchestrator
添加 “-e DISABLE_TELEMETRY=true \”，关闭自动上报部分用户数据到percona.com的功能
进入容器：docker exec -i -t pmm-server /bin/bash
修改时区：ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
退出容器，执行：docker restart pmm-server
二、客户端
1、安装
Download Percona Monitoring and Management
yum install pmm-client-1.17.3-1.el7.x86_64.rpm
2、配置
2.1、server信息
pmm-admin config --server 192.168.237.9:8080
OK, PMM server is alive.
server端信息将被记录进：/usr/local/percona/pmm-client/pmm.yml
2.2、按需开放如下端口
42000：基础系统监控
42002：MySQL监控
42003：MongoDB
42004：ProxySQL
3、配置监控项
3.1、添加监控账号：
GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON *.* TO 'pmm'@' 127.0.0.1' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;
GRANT SELECT, UPDATE, DELETE, DROP ON performance_schema.* TO 'pmm'@'127.0.0.1';
3.2、添加监控项：
[root@ceshi23710 tmp]# pmm-admin add mysql --host 127.0.0.1 --port 3306 --user pmm --password pass node1
[linux:metrics] OK, already monitoring this system.
[mysql:metrics] OK, now monitoring MySQL metrics using DSN pmm:***@tcp(127.0.0.1:3306)
[mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN pmm:***@tcp(127.0.0.1:3306)
为了避免密码泄露：
（1）可以将上述命令放到shell脚本中执行：
#!/bin/bash
pmm-admin add mysql XXXX
（2）pmm-admin之后，执行history -c清空历史信息
查看本机已有监控项：pmm-admin list
[root@ceshi23710 tmp]# pmm-admin list
pmm-admin 1.17.3
PMM Server | 192.168.237.9:8080
Client Name | ceshi23710
Client Address | 192.168.237.10
Service Manager | linux-systemd
-------------- ----------- ----------- -------- ---------------------------- --------------------------------------------------------------------------------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
-------------- ----------- ----------- -------- ---------------------------- --------------------------------------------------------------------------------------
mysql:queries node2 - YES pmm:***@tcp(127.0.0.1:3306) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics ceshi23710 42000 YES -
mysql:metrics node2 42002 YES pmm:***@tcp(127.0.0.1:3306)
三、QAN功能
待更新
四、容器的热扩容
1、查看容器卷信息
获取容器ID
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
776a56114f35 percona/pmm-server:1.17.3 "/opt/entrypoint.sh" 36 minutes ago Up 35 minutes 443/tcp, 0.0.0.0:8080->80/tcp pmm-server
f50034bbf093 percona/pmm-server:1.17.3 "/bin/true" 39 minutes ago Created pmm-data
根据容器ID提取卷id
# docker inspect -f '{{ .GraphDriver.Data.DeviceName }}' pmm-server
docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
根据卷id提取thin信息
# dmsetup table
docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596: 0 62914560 thin 253:3 59
centos-home: 0 401743872 linear 8:17 2048
docker-253:2-272456627-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing
centos-swap: 0 16515072 linear 8:2 2048
centos-root: 0 87310336 linear 8:2 16517120
centos-root: 87310336 17547264 linear 8:17 4017459202、原30G调整为40G
计算公式如下：
echo $((40*1024*1024*1024/512))
83886080
3、根据1和2拼接出操作命令
# echo 0 83886080 thin 253:3 59 | dmsetup load docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
# dmsetup resume docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
XFS文件系统格式：xfs_growfs，ext3、ext4文件系统格式：resize2fs
# xfs_growfs /dev/mapper/docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
meta-data=/dev/mapper/docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596 isize=256 agcount=49, agsize=163824 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=7864320, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 7864320 to 10485760
4、验证
进入容器，查看磁盘空间：
docker exec -i -t pmm-server /bin/bash
df -h
5、启动报错的处理
[root@test02 docker]# docker start pmm-server
Error response from daemon: Failed to mount; dmesg: <4>[32087502.063016] XFS (dm-4): last sector read failed
<6>[32087508.595214] attempt to access beyond end of device
<6>[32087508.595223] dm-4: rw=32, want=83886080, limit=62914560
<4>[32087508.595231] XFS (dm-4): last sector read failed
: mount /dev/mapper/docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596:/home/docker/devicemapper/mnt/e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596, data: nouuid: input/output error
Error: failed to start containers: pmm-server
[root@test02 docker]#
[root@test02 docker]# echo 0 83886080 thin 253:3 59 | dmsetup load docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
[root@test02 docker]# dmsetup resume docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
[root@test02 docker]#
[root@test02 docker]# docker start pmm-server
pmm-server
6、一键扩容脚本
#!/bin/bash
CID=$1
SIZE=$2
if [ "$CID" != "" ] && [ "$SIZE" != "" ]; then
DEV=$(basename $(echo /dev/mapper/docker-*-$CID));
dmsetup table $DEV | sed "s/0 [0-9]* thin/0 $(($SIZE*1024*1024*1024/512)) thin/" | dmsetup load $DEV;
dmsetup resume $DEV;
xfs_growfs /dev/mapper/$DEV;
echo "Resize $CID completed."
else
echo "Usage: sh resize_container 459fd505311ad364309940ac24dcdb2bdfc68e3c3b0f291c9153fb54fbd46771 100";
fi
使用方法：
step1、根据容器ID提取卷id
# docker inspect -f '{{ .GraphDriver.Data.DeviceName }}' pmm-server
docker-253:2-272456627-e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596
step2、扩容，45G为例
./resize_container e621a70d0729720b457978bded0819484fb0aa280405bb92e8f2e017d6063596 45
注：
1、重启后同样需要再次运行该脚本
2、每次调整的大小不能小于上次，否则报错：xfs_growfs: /dev/mapper/docker-XXX is not a mounted XFS filesystem
参考文档：
dockerhub: percona/pmm-server
两种方式迁移 Docker 的默认安装 (存储) 目录
Percona Monitoring and Management Documentation: Setting Up a Docker Container for PMM Server
Percona Monitoring and Management Documentation: Creating a MySQL User Account to Be Used with PMM
PMM监控MySQL
CentOS7修改时区的正确姿势
Understanding PMM QAN – Graphs & Metrics
Docker 环境 Storage Pool 用完解决方案：resize-device-mapper