ubuntu系统使用docker gitlab 磁盘空间满问题解决记录

问题

发现git上传不了，登录服务器，table补齐命令都报错，发现根目录磁盘使用率100%

解决问题

问题排查：
1.使用排查docker使用空间

docker system df 

未见异常

2.清理系统缓存

sudo apt autoremove
rm -rf ~/.cache/thumbnails/*
sudo rm -rf ~/.local/share/Trash/*
....

等等一顿操作下来，还是100%,reboot
发现界面都进不去了！
屮！

3.进入终端
Ctrl+Alt+F2
登录终端

4.查找大文件夹

sudo du -hsx /* | sort -rh | head -10

一层层找进去，在/var/lib/docker/containers目录发发现了一个日志文件*-json.log占了几百个G
我屮艸芔茻！！！！
删掉，重启，系统启动正常

but，docker gitlab还是起不来，一直starting

一直报错…

[+] Running 2/1
 ⠿ Network docker_gitlab_default  Created                                                                                                                                                                                              0.1s
 ⠿ Container gitlab               Created                                                                                                                                                                                              0.1s
Attaching to gitlab
gitlab  | Thank you for using GitLab Docker Image!
gitlab  | Current version: gitlab-ce=11.1.4-ce.0
gitlab  | 
gitlab  | Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab  | And restart this container to reload settings.
gitlab  | To do it use docker exec:
gitlab  | 
gitlab  |   docker exec -it gitlab vim /etc/gitlab/gitlab.rb
gitlab  |   docker restart gitlab
gitlab  | 
gitlab  | For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab  | https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab  | 
gitlab  | If this container fails to start due to permission problems try to fix it by executing:
gitlab  | 
gitlab  |   docker exec -it gitlab update-permissions
gitlab  |   docker restart gitlab
gitlab  | 

gitlab  | Preparing services...
gitlab  | Starting services...
gitlab  | Configuring GitLab package...
gitlab  | /opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
gitlab  | /opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
gitlab  | JSON file with existing configuration not found inside /opt/gitlab/embedded/nodes.
gitlab  | Skipping config check.
gitlab  | gitlab preinstall: 
gitlab  | gitlab preinstall: This node does not appear to be running a database
gitlab  | gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab  | gitlab preinstall: 
gitlab  | Configuring GitLab...

使用docker exec -it gitlab update-permissions等操作，然并nuan~

找到最后持续输出的打印如下：

gitlab  | ==> /var/log/gitlab/postgresql/current <==
gitlab  | 2023-02-16_07:24:22.32111 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:22.32112 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab  | 2023-02-16_07:24:23.32501 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:23.32502 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab  | 2023-02-16_07:24:24.32886 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:24.32887 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab  | 2023-02-16_07:24:25.33276 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:25.33278 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.

进入docker 容器，在根目录find这个文件postmaster.pid,删之…，重启容器，一切正常！

切记

磁盘空间满先不要不要乱删东西！！！

一劳永逸解决问题

更改docker日志全局设置

在主机修改docker配置问题，只对新建docker容器有效。

# 排查Docker容器日志输出情况  docker  inspect  --format {{.LogPath}}  容器名/容器ID   
vim /etc/docker/daemon.json
{
   "registry-mirrors": [
       "https://mirror.ccs.tencentyun.com",
        "https://10odyesl.mirror.aliyuncs.com"
  ],
  "log-driver":"json-file",
  "log-opts": {"max-size":"300m", "max-file":"3"}
}

systemctl restart docker
systemctl daemon-reload 

更改容器配置（docker-compose）

修改 docker-compose.yml，增加如下配置。yml分级按两个空格缩进，可以使用在线yml格式校验

    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "2

查看配置：docker inspect -f ‘{{.HostConfig.LogConfig}}’ 容器名xxx

参考

Docker日志策略调整