超级无敌倒霉端口号重复事件记录

背景：MySQL环境为单机多实例，端口号非3306，随机端口号

正常db平台创建实例，发现创建失败，报错如下，实例端口被占用

2019-06-10 17:18:05 139906110465920 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
2019-06-10 17:18:05 139906110465920 [ERROR] Do you already have another mysqld server running on port: 2222 ?
2019-06-10 17:18:05 139906110465920 [ERROR] Aborting
2019-06-10 17:19:05 139945829066624 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
2019-06-10 17:19:05 139945829066624 [ERROR] Do you already have another mysqld server running on port: 2222 ?
2019-06-10 17:19:05 139945829066624 [ERROR] Aborting
2019-06-10 17:25:49 139819086829440 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
2019-06-10 17:25:49 139819086829440 [ERROR] Do you already have another mysqld server running on port: 2222 ?
2019-06-10 17:25:49 139819086829440 [ERROR] Aborting

麻溜看一眼是否真的有其它实例占用端口。结果是木有

算了，换一个端口，但

2019-06-10 16:49:46 140167381636992 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
2019-06-10 16:49:46 140167381636992 [ERROR] Do you already have another mysqld server running on port: 33333 ?
2019-06-10 16:49:46 140167381636992 [ERROR] Aborting
2019-06-10 16:51:13 140119442626432 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
2019-06-10 16:51:13 140119442626432 [ERROR] Do you already have another mysqld server running on port: 33333 ?
2019-06-10 16:51:13 140119442626432 [ERROR] Aborting

有点懵逼，还是没查看到有啥应用占用端口。

难道是这两个端口有毒。换成3306，果然没有问题，不是平台的问题。

使用lsof继续定位问题，发现

# lsof
COMMAND    PID  USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
mysqld   33850 mysql   86u  IPv4 141820285      0t0  TCP 1.1.1.1:33333->2.2.2.2:1111 (ESTABLISHED)
mysqld  179224 mysql   86u  IPv4 398823713      0t0  TCP 1.1.1.1:2222->2.2.2.2:4444 (ESTABLISHED)

哎，运气真不好，可以看到，进程号179224对应的是4444的mysql实例，复制的客户端正巧占用了2222的端口，如下：

mysqld  179224 mysql   86u  IPv4 398823713      0t0  TCP 1.1.1.1:2222->2.2.2.2:4444 (ESTABLISHED)

第二次的实例端口，也非常不巧的别另外一个实例的复制用户的客户端占用了

mysqld   33850 mysql   86u  IPv4 141820285      0t0  TCP 1.1.1.1:33333->2.2.2.2:1111 (ESTABLISHED)

解决办法：
登陆到对应的mysql，执行stop slave; start slave;也就解决了
看来，每次创建实例的时候还要判断一下端口号了，尴尬。。

遇到第一次是巧合，连续遇到两次就是真倒霉