java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起

项目中遇到的一个问题:项目忽然蹦了,用我们的域名登陆不上去了。

根据之前的经验,一般比如我们项目登不上去了或者数据库不上数据了(数据不更新),直接在Xshell上远程reboot一下,再重启一下tomcat就行。要还解决不了就登阿里云官网直接重启一下实例就解决了,之前一直这么干。	

但是这一次,重启了之后在浏览器上登陆还是没反应

然后排查过程是这样的:

  • 首先我们在svn上拉下来了一个最新版本的项目,然后导入到eclipse中配置好之后在tomcat中跑起来项目是能正常运行的.
    • 这个时候因为我们里面不是有个jeesite.properties配置文件嘛(它里面就写的是那个连接数据库用的jdbc.url、username、password\以及一些数据库连接池参数以及一些集成到spring框架上的比如redis、shiro等用到的一些参数,都在这个配置文件中配置着呢)。
    • 此时我们配置文件中配置的是localhost,也就是当地的数据库,此时他是正常的。
  • 但是当我们把jdbc.url换成39.105.60.166也就是我们阿里云数据库时,项目此时就报错了。
  • 然后我们初步判断是数据库满了或者说连接不上了(这是后面改好了的,这之前是占用100%)
  • 然后我们打开了pgAdmin 4 v2这个软件,在里面想连一下云数据库看一下里面的数据,结果在密码输入正确的情况下连不上云数据库(通过ip地址,也就是刚才的39.105.60.166),死活连接不上数据库,还报一个下面的错。
  • 在网上查了查,都说让改一些配置文件:postgresql.conf(在这个配置文件中设置一下在里面改一下listen.address=“*”,port=5432),pg_hba.conf中设置一下信用IP,然后在IPV4中添加一个ip,把本地的ip添加进去,以便数据库能访问到还是不行.
  • 到了第二天师兄师姐说不行咱问一问阿里云客服吧,然后和人家聊了聊最后把问题解决了,过程如下:
    给人家说的是:在我们没有改动的情况下云上的数据库突然连接不上了,然后把上面在报的错也就是:could not connect to server:Connection refused (0x0000274D/10061) is the server running on host “39.105.60.166” and accepting TCP/IP connections on port 5432?,
    然后人家让我们执行下:
[root@xhz ~]# netstat -ntlp
[root@xhz ~]# iptables -nL
  • 完了后查出来数据库服务没有启动(让我们查下环境部署核实下启动方式),用systemctl start postgresql-10.service这条命令重新启动下数据库服务。不过我们数据库重启失败了,不知道是自己操作原因还是怎么的,然后重启后重启失败报错。
  • 然后用journalctl -xe >>mysql.logs这个命令重定向到mysql.log这个文件中find / -name mysql.logs,这个命令也可以辅助用来找到mysql.log)。
    mysql.logs在最后面。

,然后阿里云那边让我们提供一下阿里云上面的实例id,并让我们对数据盘做快照,并给他们授权了后,
然后他们那边帮我们修改了一下把数据库重新启动了一遍。他们修改的过程如下:
netstat -antlpigrep 5432

两个配置文件postgresql.conf和pg_hba.conf中最主要的就是修改了下面内容:
postgresql.conf:

......
#------------------------------------------------------------------------------
# - Connection Settings -
listen_addresses = '*'		# what IP address(es) to listen on;
					# comma-separated list of addresses;
					# defaults to 'localhost'; use '*' for all
					# (change requires restart)
``
#tcpip_socket = true//把这个放出来就行了
port = 5432				# (change requires restart)
max_connections = 100			# (change requires restart)
......
# TYPE  DATABASE        USER            ADDRESS                 METHOD
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
host    replication     all             127.0.0.1/32            md5
host    replication     all             ::1/128                 md5

mysql.logs

  • 前言:vi /etc/my.cnf中指定了数据库错误日志位置及名称.log-error=/var/log/mariadb/mariadb.log(名字不一定就是这,就是个名字嘛,叫个error.log也行)
  • 除了这个错误日志还有一个慢查询日志用来记录SQL语句执行时间超过long_query_time这个变量定义的时长的查询语句。通过慢查询日志,可以查出哪些查询语句的执行效率很低,以便进行优化。
    • 对于数据库的日常管理工作,性能管理肯定会是占比最大的一块,一个好的性能分析工具会极大的提高数据库性能管理的效率,而 pt-query-digest 就是专门针对 MySQL 数据库慢查询日志的一个强力分析工具,相比于 mysqldumpslow ,其分析结果更加具体和完善。pt-query-digest 属于 Percona Toolkit 工具集中最常用的一种,号称 MySQL DBA 必备工具之一,其能够分析MySQL数据库的 slow log 、 general log 、 binary log 文件,同时也可以使用 show processlist 或从tcpdump 抓取的 MySQL 协议数据来进行分析
      • 一个博主关于pt-query-digest的文章,有一个使用例子
      • 一篇关于pt-query-digest工具分析最近一周的mysql-slow.log以及使用tcpdump抓包一段时间对该表的select语句的文章
  • 除了错误日志以及慢查询日志,还有二进制日志、中继日志、事务日志等等

下面是日志的内容,拷贝下来以备日后出现类似问题,再深入研究。

postgresql.conf
# -----------------------------
# PostgreSQL configuration file
# -----------------------------
...
#------------------------------------------------------------------------------
# FILE LOCATIONS
#------------------------------------------------------------------------------
...
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------
# - Connection Settings -

listen_addresses = '*'		# what IP address(es) to listen on;
					# comma-separated list of addresses;
					# defaults to 'localhost'; use '*' for all
					# (change requires restart)
#tcpip_socket = true
port = 5432				# (change requires restart)
max_connections = 100			# (change requires restart)
#superuser_reserved_connections = 3	# (change requires restart)
#unix_socket_directories = ''	# comma-separated list of directories
					# (change requires restart)
#unix_socket_group = ''			# (change requires restart)
#unix_socket_permissions = 0777		# begin with 0 to use octal notation
					# (change requires restart)
#bonjour = off				# advertise server via Bonjour
					# (change requires restart)
#bonjour_name = ''			# defaults to the computer name
					# (change requires restart)
# - Security and Authentication -
#authentication_timeout = 1min		# 1s-600s
#ssl = off
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_dh_params_file = ''
#ssl_cert_file = 'server.crt'
#ssl_key_file = 'server.key'
#ssl_ca_file = ''
#ssl_crl_file = ''
#password_encryption = md5		# md5 or scram-sha-256
#db_user_namespace = off
#row_security = on
# GSSAPI using Kerberos
#krb_server_keyfile = ''
#krb_caseins_users = off
# - TCP Keepalives -
# see "man 7 tcp" for details
#tcp_keepalives_idle = 0		# TCP_KEEPIDLE, in seconds;
					# 0 selects the system default
#tcp_keepalives_interval = 0		# TCP_KEEPINTVL, in seconds;
					# 0 selects the system default
#tcp_keepalives_count = 0		# TCP_KEEPCNT;
					# 0 selects the system default
......
#------------------------------------------------------------------------------
# ERROR HANDLING
#------------------------------------------------------------------------------
#exit_on_error = off			# terminate session on any error?
#restart_after_crash = on		# reinitialize after backend crash?
#------------------------------------------------------------------------------
# CONFIG FILE INCLUDES
#------------------------------------------------------------------------------
...

pg_hba.conf
# PostgreSQL Client Authentication Configuration File
...
# configuration parameter, or via the -i or -h command line switches.
# TYPE  DATABASE        USER            ADDRESS                 METHOD
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
host    replication     all             127.0.0.1/32            md5
host    replication     all             ::1/128                 md5

mysql.logs

-- Logs begin at Fri 2021-05-14 02:27:20 CST, end at Sun 2021-11-21 16:41:01 CST. --
Nov 21 16:02:08 xhz crontab[2634]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2636]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2637]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2639]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2640]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2642]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2643]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2645]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2693]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2698]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2700]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2702]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2704]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2705]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2707]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2708]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2710]: (postgres) REPLACE (postgres)
Nov 21 16:02:08 xhz crontab[2711]: (postgres) LIST (postgres)
Nov 21 16:02:08 xhz crontab[2713]: (postgres) REPLACE (postgres)
Nov 21 16:03:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:03:01 xhz systemd[1]: Started Session 12 of user root.
-- Subject: Unit session-12.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-12.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:03:01 xhz CROND[3057]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:03:01 xhz postfix/pickup[1262]: EAE2F40098: uid=0 from=<root>
Nov 21 16:03:01 xhz postfix/cleanup[1752]: EAE2F40098: message-id=<20211121080301.EAE2F40098@xhz.localdomain>
Nov 21 16:03:01 xhz postfix/qmgr[1263]: EAE2F40098: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:03:01 xhz postfix/local[1784]: EAE2F40098: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:03:01 xhz postfix/cleanup[1753]: EF7DE400AB: message-id=<20211121080301.EF7DE400AB@xhz.localdomain>
Nov 21 16:03:01 xhz postfix/bounce[1785]: EAE2F40098: sender non-delivery notification: EF7DE400AB
Nov 21 16:03:01 xhz postfix/qmgr[1263]: EF7DE400AB: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:03:01 xhz postfix/qmgr[1263]: EAE2F40098: removed
Nov 21 16:03:02 xhz postfix/local[1781]: EF7DE400AB: to=<root@xhz.localdomain>, relay=local, delay=0.03, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:03:02 xhz postfix/qmgr[1263]: EF7DE400AB: removed
Nov 21 16:03:14 xhz crontab[3149]: (postgres) LIST (postgres)
Nov 21 16:03:14 xhz crontab[3154]: (postgres) LIST (postgres)
Nov 21 16:03:14 xhz crontab[3156]: (postgres) REPLACE (postgres)
Nov 21 16:03:14 xhz crontab[3157]: (postgres) LIST (postgres)
Nov 21 16:03:14 xhz crontab[3159]: (postgres) REPLACE (postgres)
Nov 21 16:03:14 xhz crontab[3160]: (postgres) LIST (postgres)
Nov 21 16:03:14 xhz crontab[3162]: (postgres) REPLACE (postgres)
Nov 21 16:03:14 xhz crontab[3163]: (postgres) LIST (postgres)
Nov 21 16:03:14 xhz crontab[3165]: (postgres) REPLACE (postgres)
Nov 21 16:03:14 xhz crontab[3166]: (postgres) LIST (postgres)
Nov 21 16:03:14 xhz crontab[3168]: (postgres) REPLACE (postgres)
Nov 21 16:03:21 xhz crontab[3279]: (postgres) LIST (postgres)
Nov 21 16:03:21 xhz crontab[3280]: (postgres) LIST (postgres)
Nov 21 16:03:21 xhz crontab[3281]: (postgres) LIST (postgres)
Nov 21 16:03:22 xhz sshd[3341]: Connection closed by 127.0.0.1 port 52146 [preauth]
Nov 21 16:03:22 xhz sshd[3344]: Connection closed by 127.0.0.1 port 52148 [preauth]
Nov 21 16:03:23 xhz crontab[3351]: (postgres) LIST (postgres)
Nov 21 16:04:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:04:01 xhz systemd[1]: Started Session 13 of user root.
-- Subject: Unit session-13.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-13.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:04:01 xhz CROND[3606]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:04:01 xhz postfix/pickup[1262]: F4038400AB: uid=0 from=<root>
Nov 21 16:04:01 xhz postfix/cleanup[1753]: F4038400AB: message-id=<20211121080401.F4038400AB@xhz.localdomain>
Nov 21 16:04:02 xhz postfix/qmgr[1263]: F4038400AB: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:04:02 xhz postfix/local[1784]: F4038400AB: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:04:02 xhz postfix/cleanup[1753]: 02ADD40074: message-id=<20211121080402.02ADD40074@xhz.localdomain>
Nov 21 16:04:02 xhz postfix/bounce[1785]: F4038400AB: sender non-delivery notification: 02ADD40074
Nov 21 16:04:02 xhz postfix/qmgr[1263]: 02ADD40074: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:04:02 xhz postfix/qmgr[1263]: F4038400AB: removed
Nov 21 16:04:02 xhz postfix/local[1781]: 02ADD40074: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:04:02 xhz postfix/qmgr[1263]: 02ADD40074: removed
Nov 21 16:04:08 xhz crontab[3660]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3665]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3667]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3668]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3670]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3671]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3673]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3674]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3676]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3677]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3679]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3722]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3727]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3729]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3730]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3732]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3733]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3735]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3736]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3738]: (postgres) REPLACE (postgres)
Nov 21 16:04:08 xhz crontab[3739]: (postgres) LIST (postgres)
Nov 21 16:04:08 xhz crontab[3741]: (postgres) REPLACE (postgres)
Nov 21 16:05:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:05:01 xhz systemd[1]: Started Session 14 of user root.
-- Subject: Unit session-14.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-14.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:05:01 xhz CROND[4084]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:05:01 xhz postfix/pickup[1262]: 0AC31400AB: uid=0 from=<root>
Nov 21 16:05:01 xhz postfix/cleanup[4088]: 0AC31400AB: message-id=<20211121080501.0AC31400AB@xhz.localdomain>
Nov 21 16:05:01 xhz postfix/qmgr[1263]: 0AC31400AB: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:05:01 xhz postfix/local[1784]: 0AC31400AB: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.05, delays=0.02/0/0/0.03, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:05:01 xhz postfix/cleanup[4088]: 13218400C3: message-id=<20211121080501.13218400C3@xhz.localdomain>
Nov 21 16:05:01 xhz postfix/bounce[4090]: 0AC31400AB: sender non-delivery notification: 13218400C3
Nov 21 16:05:01 xhz postfix/qmgr[1263]: 13218400C3: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:05:01 xhz postfix/qmgr[1263]: 0AC31400AB: removed
Nov 21 16:05:01 xhz postfix/local[1781]: 13218400C3: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:05:01 xhz postfix/qmgr[1263]: 13218400C3: removed
Nov 21 16:05:30 xhz crontab[4268]: (postgres) LIST (postgres)
Nov 21 16:06:01 xhz systemd[1]: Started Session 15 of user root.
-- Subject: Unit session-15.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-15.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:06:01 xhz CROND[4451]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:06:01 xhz postfix/pickup[1262]: 11F4C40098: uid=0 from=<root>
Nov 21 16:06:01 xhz postfix/cleanup[4088]: 11F4C40098: message-id=<20211121080601.11F4C40098@xhz.localdomain>
Nov 21 16:06:01 xhz postfix/qmgr[1263]: 11F4C40098: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:06:01 xhz postfix/local[1784]: 11F4C40098: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:06:01 xhz postfix/cleanup[4088]: 15055400C3: message-id=<20211121080601.15055400C3@xhz.localdomain>
Nov 21 16:06:01 xhz postfix/bounce[4090]: 11F4C40098: sender non-delivery notification: 15055400C3
Nov 21 16:06:01 xhz postfix/qmgr[1263]: 15055400C3: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:06:01 xhz postfix/qmgr[1263]: 11F4C40098: removed
Nov 21 16:06:01 xhz postfix/local[1781]: 15055400C3: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:06:01 xhz postfix/qmgr[1263]: 15055400C3: removed
Nov 21 16:06:08 xhz crontab[4507]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4512]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4514]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4515]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4517]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4518]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4520]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4521]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4523]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4524]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4526]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4569]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4574]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4576]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4577]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4579]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4580]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4582]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4583]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4585]: (postgres) REPLACE (postgres)
Nov 21 16:06:08 xhz crontab[4586]: (postgres) LIST (postgres)
Nov 21 16:06:08 xhz crontab[4588]: (postgres) REPLACE (postgres)
Nov 21 16:06:30 xhz crontab[4741]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4746]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4748]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4750]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4752]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4753]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4755]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4756]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4758]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4759]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4761]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4804]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4809]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4811]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4812]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4814]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4815]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4817]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4818]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4820]: (postgres) REPLACE (postgres)
Nov 21 16:06:31 xhz crontab[4821]: (postgres) LIST (postgres)
Nov 21 16:06:31 xhz crontab[4823]: (postgres) REPLACE (postgres)
Nov 21 16:07:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:07:01 xhz systemd[1]: Started Session 16 of user root.
-- Subject: Unit session-16.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-16.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:07:01 xhz CROND[5029]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:07:01 xhz postfix/pickup[1262]: 1C35D400AB: uid=0 from=<root>
Nov 21 16:07:01 xhz postfix/cleanup[4088]: 1C35D400AB: message-id=<20211121080701.1C35D400AB@xhz.localdomain>
Nov 21 16:07:01 xhz postfix/qmgr[1263]: 1C35D400AB: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:07:01 xhz postfix/local[1784]: 1C35D400AB: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:07:01 xhz postfix/cleanup[4088]: 1ED82400C3: message-id=<20211121080701.1ED82400C3@xhz.localdomain>
Nov 21 16:07:01 xhz postfix/bounce[4090]: 1C35D400AB: sender non-delivery notification: 1ED82400C3
Nov 21 16:07:01 xhz postfix/qmgr[1263]: 1ED82400C3: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:07:01 xhz postfix/qmgr[1263]: 1C35D400AB: removed
Nov 21 16:07:01 xhz postfix/local[1781]: 1ED82400C3: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:07:01 xhz postfix/qmgr[1263]: 1ED82400C3: removed
Nov 21 16:08:01 xhz systemd[1]: Started Session 17 of user root.
-- Subject: Unit session-17.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-17.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:08:01 xhz CROND[5512]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:08:01 xhz postfix/pickup[1262]: 25363400D0: uid=0 from=<root>
Nov 21 16:08:01 xhz postfix/cleanup[4088]: 25363400D0: message-id=<20211121080801.25363400D0@xhz.localdomain>
Nov 21 16:08:01 xhz postfix/qmgr[1263]: 25363400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:08:01 xhz postfix/local[1784]: 25363400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:08:01 xhz postfix/cleanup[4088]: 27CC1400F2: message-id=<20211121080801.27CC1400F2@xhz.localdomain>
Nov 21 16:08:01 xhz postfix/bounce[4090]: 25363400D0: sender non-delivery notification: 27CC1400F2
Nov 21 16:08:01 xhz postfix/qmgr[1263]: 27CC1400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:08:01 xhz postfix/qmgr[1263]: 25363400D0: removed
Nov 21 16:08:01 xhz postfix/local[1781]: 27CC1400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:08:01 xhz postfix/qmgr[1263]: 27CC1400F2: removed
Nov 21 16:08:08 xhz crontab[5577]: (postgres) LIST (postgres)
Nov 21 16:08:08 xhz crontab[5582]: (postgres) LIST (postgres)
Nov 21 16:08:08 xhz crontab[5584]: (postgres) REPLACE (postgres)
Nov 21 16:08:08 xhz crontab[5585]: (postgres) LIST (postgres)
Nov 21 16:08:08 xhz crontab[5587]: (postgres) REPLACE (postgres)
Nov 21 16:08:08 xhz crontab[5588]: (postgres) LIST (postgres)
Nov 21 16:08:08 xhz crontab[5590]: (postgres) REPLACE (postgres)
Nov 21 16:08:08 xhz crontab[5591]: (postgres) LIST (postgres)
Nov 21 16:08:08 xhz crontab[5593]: (postgres) REPLACE (postgres)
Nov 21 16:08:08 xhz crontab[5594]: (postgres) LIST (postgres)
Nov 21 16:08:08 xhz crontab[5596]: (postgres) REPLACE (postgres)
Nov 21 16:08:09 xhz crontab[5640]: (postgres) LIST (postgres)
Nov 21 16:08:09 xhz crontab[5645]: (postgres) LIST (postgres)
Nov 21 16:08:09 xhz crontab[5647]: (postgres) REPLACE (postgres)
Nov 21 16:08:09 xhz crontab[5648]: (postgres) LIST (postgres)
Nov 21 16:08:09 xhz crontab[5650]: (postgres) REPLACE (postgres)
Nov 21 16:08:09 xhz crontab[5651]: (postgres) LIST (postgres)
Nov 21 16:08:09 xhz crontab[5653]: (postgres) REPLACE (postgres)
Nov 21 16:08:09 xhz crontab[5654]: (postgres) LIST (postgres)
Nov 21 16:08:09 xhz crontab[5656]: (postgres) REPLACE (postgres)
Nov 21 16:08:09 xhz crontab[5657]: (postgres) LIST (postgres)
Nov 21 16:08:09 xhz crontab[5659]: (postgres) REPLACE (postgres)
Nov 21 16:09:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:09:01 xhz systemd[1]: Started Session 18 of user root.
-- Subject: Unit session-18.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-18.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:09:01 xhz CROND[6015]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:09:01 xhz postfix/pickup[1262]: 3657A400D0: uid=0 from=<root>
Nov 21 16:09:01 xhz postfix/cleanup[4088]: 3657A400D0: message-id=<20211121080901.3657A400D0@xhz.localdomain>
Nov 21 16:09:01 xhz postfix/qmgr[1263]: 3657A400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:09:01 xhz postfix/local[1784]: 3657A400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.04, delays=0.03/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:09:01 xhz postfix/cleanup[4088]: 3A2E6400F2: message-id=<20211121080901.3A2E6400F2@xhz.localdomain>
Nov 21 16:09:01 xhz postfix/bounce[4090]: 3657A400D0: sender non-delivery notification: 3A2E6400F2
Nov 21 16:09:01 xhz postfix/qmgr[1263]: 3A2E6400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:09:01 xhz postfix/qmgr[1263]: 3657A400D0: removed
Nov 21 16:09:01 xhz postfix/local[1781]: 3A2E6400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:09:01 xhz postfix/qmgr[1263]: 3A2E6400F2: removed
Nov 21 16:10:01 xhz systemd[1]: Started Session 20 of user root.
-- Subject: Unit session-20.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-20.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:10:01 xhz systemd[1]: Started Session 19 of user root.
-- Subject: Unit session-19.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-19.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:10:01 xhz CROND[6370]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:10:01 xhz CROND[6371]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Nov 21 16:10:01 xhz postfix/pickup[1262]: 43FA6400D0: uid=0 from=<root>
Nov 21 16:10:01 xhz postfix/cleanup[4088]: 43FA6400D0: message-id=<20211121081001.43FA6400D0@xhz.localdomain>
Nov 21 16:10:01 xhz postfix/qmgr[1263]: 43FA6400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:10:01 xhz postfix/local[1784]: 43FA6400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:10:01 xhz postfix/cleanup[4088]: 4772F400F2: message-id=<20211121081001.4772F400F2@xhz.localdomain>
Nov 21 16:10:01 xhz postfix/bounce[4090]: 43FA6400D0: sender non-delivery notification: 4772F400F2
Nov 21 16:10:01 xhz postfix/qmgr[1263]: 4772F400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:10:01 xhz postfix/qmgr[1263]: 43FA6400D0: removed
Nov 21 16:10:01 xhz postfix/local[1781]: 4772F400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:10:01 xhz postfix/qmgr[1263]: 4772F400F2: removed
Nov 21 16:10:09 xhz crontab[6439]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6444]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6446]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6447]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6449]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6450]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6452]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6453]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6455]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6456]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6458]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6501]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6506]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6508]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6509]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6511]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6512]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6514]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6515]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6517]: (postgres) REPLACE (postgres)
Nov 21 16:10:09 xhz crontab[6518]: (postgres) LIST (postgres)
Nov 21 16:10:09 xhz crontab[6520]: (postgres) REPLACE (postgres)
Nov 21 16:11:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:11:01 xhz systemd[1]: Started Session 21 of user root.
-- Subject: Unit session-21.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-21.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:11:01 xhz CROND[6854]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:11:01 xhz postfix/pickup[1262]: 5009D400D0: uid=0 from=<root>
Nov 21 16:11:01 xhz postfix/cleanup[4088]: 5009D400D0: message-id=<20211121081101.5009D400D0@xhz.localdomain>
Nov 21 16:11:01 xhz postfix/qmgr[1263]: 5009D400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:11:01 xhz postfix/local[1784]: 5009D400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:11:01 xhz postfix/cleanup[4088]: 53C46400AB: message-id=<20211121081101.53C46400AB@xhz.localdomain>
Nov 21 16:11:01 xhz postfix/bounce[4090]: 5009D400D0: sender non-delivery notification: 53C46400AB
Nov 21 16:11:01 xhz postfix/qmgr[1263]: 53C46400AB: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:11:01 xhz postfix/qmgr[1263]: 5009D400D0: removed
Nov 21 16:11:01 xhz postfix/local[1781]: 53C46400AB: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:11:01 xhz postfix/qmgr[1263]: 53C46400AB: removed
Nov 21 16:12:01 xhz systemd[1]: Started Session 22 of user root.
-- Subject: Unit session-22.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-22.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:12:01 xhz CROND[7208]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:12:01 xhz postfix/pickup[1262]: 5C59F400D0: uid=0 from=<root>
Nov 21 16:12:01 xhz postfix/cleanup[4088]: 5C59F400D0: message-id=<20211121081201.5C59F400D0@xhz.localdomain>
Nov 21 16:12:01 xhz postfix/qmgr[1263]: 5C59F400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:12:01 xhz postfix/local[1784]: 5C59F400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:12:01 xhz postfix/cleanup[4088]: 5F9B6400F2: message-id=<20211121081201.5F9B6400F2@xhz.localdomain>
Nov 21 16:12:01 xhz postfix/bounce[4090]: 5C59F400D0: sender non-delivery notification: 5F9B6400F2
Nov 21 16:12:01 xhz postfix/qmgr[1263]: 5F9B6400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:12:01 xhz postfix/qmgr[1263]: 5C59F400D0: removed
Nov 21 16:12:01 xhz postfix/local[1781]: 5F9B6400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:12:01 xhz postfix/qmgr[1263]: 5F9B6400F2: removed
Nov 21 16:12:09 xhz crontab[7273]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7278]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7280]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7281]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7283]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7284]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7286]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7287]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7289]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7290]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7292]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7337]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7342]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7344]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7345]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7347]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7348]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7350]: (postgres) REPLACE (postgres)
Nov 21 16:12:09 xhz crontab[7351]: (postgres) LIST (postgres)
Nov 21 16:12:09 xhz crontab[7353]: (postgres) REPLACE (postgres)
Nov 21 16:12:10 xhz crontab[7354]: (postgres) LIST (postgres)
Nov 21 16:12:10 xhz crontab[7356]: (postgres) REPLACE (postgres)
Nov 21 16:12:59 xhz systemd[1]: Starting Cleanup of Temporary Directories...
-- Subject: Unit systemd-tmpfiles-clean.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit systemd-tmpfiles-clean.service has begun starting up.
Nov 21 16:12:59 xhz systemd[1]: Started Cleanup of Temporary Directories.
-- Subject: Unit systemd-tmpfiles-clean.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit systemd-tmpfiles-clean.service has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:13:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:13:01 xhz systemd[1]: Started Session 23 of user root.
-- Subject: Unit session-23.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-23.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:13:01 xhz CROND[7691]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:13:01 xhz postfix/pickup[1262]: 68F40400C3: uid=0 from=<root>
Nov 21 16:13:01 xhz postfix/cleanup[4088]: 68F40400C3: message-id=<20211121081301.68F40400C3@xhz.localdomain>
Nov 21 16:13:01 xhz postfix/qmgr[1263]: 68F40400C3: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:13:01 xhz postfix/local[1784]: 68F40400C3: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:13:01 xhz postfix/cleanup[4088]: 6CBB8400D0: message-id=<20211121081301.6CBB8400D0@xhz.localdomain>
Nov 21 16:13:01 xhz postfix/bounce[4090]: 68F40400C3: sender non-delivery notification: 6CBB8400D0
Nov 21 16:13:01 xhz postfix/qmgr[1263]: 6CBB8400D0: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:13:01 xhz postfix/qmgr[1263]: 68F40400C3: removed
Nov 21 16:13:01 xhz postfix/local[1781]: 6CBB8400D0: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:13:01 xhz postfix/qmgr[1263]: 6CBB8400D0: removed
Nov 21 16:14:01 xhz systemd[1]: Started Session 24 of user root.
-- Subject: Unit session-24.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-24.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:14:01 xhz CROND[8047]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:14:01 xhz postfix/pickup[1262]: 753F3400D0: uid=0 from=<root>
Nov 21 16:14:01 xhz postfix/cleanup[4088]: 753F3400D0: message-id=<20211121081401.753F3400D0@xhz.localdomain>
Nov 21 16:14:01 xhz postfix/qmgr[1263]: 753F3400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:14:01 xhz postfix/local[1784]: 753F3400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:14:01 xhz postfix/cleanup[4088]: 78B2F400F2: message-id=<20211121081401.78B2F400F2@xhz.localdomain>
Nov 21 16:14:01 xhz postfix/bounce[4090]: 753F3400D0: sender non-delivery notification: 78B2F400F2
Nov 21 16:14:01 xhz postfix/qmgr[1263]: 753F3400D0: removed
Nov 21 16:14:01 xhz postfix/qmgr[1263]: 78B2F400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:14:01 xhz postfix/local[1781]: 78B2F400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:14:01 xhz postfix/qmgr[1263]: 78B2F400F2: removed
Nov 21 16:14:10 xhz crontab[8112]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8117]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8119]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8120]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8122]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8123]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8125]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8126]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8128]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8129]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8131]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8174]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8179]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8181]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8182]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8184]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8185]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8187]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8188]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8190]: (postgres) REPLACE (postgres)
Nov 21 16:14:10 xhz crontab[8191]: (postgres) LIST (postgres)
Nov 21 16:14:10 xhz crontab[8193]: (postgres) REPLACE (postgres)
Nov 21 16:15:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:15:01 xhz systemd[1]: Started Session 25 of user root.
-- Subject: Unit session-25.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-25.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:15:01 xhz CROND[8526]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:15:01 xhz postfix/pickup[1262]: 818EB400D0: uid=0 from=<root>
Nov 21 16:15:01 xhz postfix/cleanup[4088]: 818EB400D0: message-id=<20211121081501.818EB400D0@xhz.localdomain>
Nov 21 16:15:01 xhz postfix/qmgr[1263]: 818EB400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:15:01 xhz postfix/local[1784]: 818EB400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:15:01 xhz postfix/cleanup[4088]: 85456400F2: message-id=<20211121081501.85456400F2@xhz.localdomain>
Nov 21 16:15:01 xhz postfix/bounce[4090]: 818EB400D0: sender non-delivery notification: 85456400F2
Nov 21 16:15:01 xhz postfix/qmgr[1263]: 85456400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:15:01 xhz postfix/qmgr[1263]: 818EB400D0: removed
Nov 21 16:15:01 xhz postfix/local[1781]: 85456400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:15:01 xhz postfix/qmgr[1263]: 85456400F2: removed
Nov 21 16:16:01 xhz systemd[1]: Started Session 26 of user root.
-- Subject: Unit session-26.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-26.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:16:01 xhz CROND[8882]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:16:01 xhz postfix/pickup[1262]: 8D1D6400AB: uid=0 from=<root>
Nov 21 16:16:01 xhz postfix/cleanup[4088]: 8D1D6400AB: message-id=<20211121081601.8D1D6400AB@xhz.localdomain>
Nov 21 16:16:01 xhz postfix/qmgr[1263]: 8D1D6400AB: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:16:01 xhz postfix/local[1784]: 8D1D6400AB: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:16:01 xhz postfix/cleanup[4088]: 90A4A400F2: message-id=<20211121081601.90A4A400F2@xhz.localdomain>
Nov 21 16:16:01 xhz postfix/bounce[4090]: 8D1D6400AB: sender non-delivery notification: 90A4A400F2
Nov 21 16:16:01 xhz postfix/qmgr[1263]: 8D1D6400AB: removed
Nov 21 16:16:01 xhz postfix/qmgr[1263]: 90A4A400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:16:01 xhz postfix/local[1781]: 90A4A400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:16:01 xhz postfix/qmgr[1263]: 90A4A400F2: removed
Nov 21 16:16:10 xhz systemd[1]: Time has been changed
-- Subject: Time change
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The system clock has been changed to REALTIME microseconds after January 1st, 1970.
Nov 21 16:16:11 xhz crontab[8958]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[8963]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[8965]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[8966]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[8968]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[8969]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[8971]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[8972]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[8974]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[8975]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[8977]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[9020]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[9025]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[9027]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[9028]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[9030]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[9031]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[9033]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[9034]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[9036]: (postgres) REPLACE (postgres)
Nov 21 16:16:11 xhz crontab[9037]: (postgres) LIST (postgres)
Nov 21 16:16:11 xhz crontab[9039]: (postgres) REPLACE (postgres)
Nov 21 16:17:02 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:17:02 xhz systemd[1]: Started Session 27 of user root.
-- Subject: Unit session-27.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-27.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:17:02 xhz CROND[9362]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:17:02 xhz postfix/pickup[1262]: 672E1400D0: uid=0 from=<root>
Nov 21 16:17:02 xhz postfix/cleanup[4088]: 672E1400D0: message-id=<20211121081702.672E1400D0@xhz.localdomain>
Nov 21 16:17:02 xhz postfix/qmgr[1263]: 672E1400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:17:02 xhz postfix/local[1784]: 672E1400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.04, delays=0.01/0/0/0.02, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:17:02 xhz postfix/cleanup[4088]: 6F306400F2: message-id=<20211121081702.6F306400F2@xhz.localdomain>
Nov 21 16:17:02 xhz postfix/bounce[4090]: 672E1400D0: sender non-delivery notification: 6F306400F2
Nov 21 16:17:02 xhz postfix/qmgr[1263]: 6F306400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:17:02 xhz postfix/qmgr[1263]: 672E1400D0: removed
Nov 21 16:17:02 xhz postfix/local[1781]: 6F306400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:17:02 xhz postfix/qmgr[1263]: 6F306400F2: removed
Nov 21 16:18:01 xhz systemd[1]: Started Session 28 of user root.
-- Subject: Unit session-28.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-28.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:18:01 xhz CROND[9716]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:18:01 xhz postfix/pickup[1262]: 72ABC400C3: uid=0 from=<root>
Nov 21 16:18:01 xhz postfix/cleanup[4088]: 72ABC400C3: message-id=<20211121081801.72ABC400C3@xhz.localdomain>
Nov 21 16:18:01 xhz postfix/qmgr[1263]: 72ABC400C3: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:18:01 xhz postfix/local[1784]: 72ABC400C3: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:18:01 xhz postfix/cleanup[4088]: 76102400F2: message-id=<20211121081801.76102400F2@xhz.localdomain>
Nov 21 16:18:01 xhz postfix/bounce[4090]: 72ABC400C3: sender non-delivery notification: 76102400F2
Nov 21 16:18:01 xhz postfix/qmgr[1263]: 72ABC400C3: removed
Nov 21 16:18:01 xhz postfix/qmgr[1263]: 76102400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:18:01 xhz postfix/local[1781]: 76102400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:18:01 xhz postfix/qmgr[1263]: 76102400F2: removed
Nov 21 16:18:11 xhz crontab[9794]: (postgres) LIST (postgres)
Nov 21 16:18:11 xhz crontab[9799]: (postgres) LIST (postgres)
Nov 21 16:18:11 xhz crontab[9801]: (postgres) REPLACE (postgres)
Nov 21 16:18:11 xhz crontab[9802]: (postgres) LIST (postgres)
Nov 21 16:18:11 xhz crontab[9804]: (postgres) REPLACE (postgres)
Nov 21 16:18:11 xhz crontab[9805]: (postgres) LIST (postgres)
Nov 21 16:18:11 xhz crontab[9807]: (postgres) REPLACE (postgres)
Nov 21 16:18:11 xhz crontab[9808]: (postgres) LIST (postgres)
Nov 21 16:18:11 xhz crontab[9810]: (postgres) REPLACE (postgres)
Nov 21 16:18:11 xhz crontab[9811]: (postgres) LIST (postgres)
Nov 21 16:18:11 xhz crontab[9813]: (postgres) REPLACE (postgres)
Nov 21 16:18:12 xhz crontab[9856]: (postgres) LIST (postgres)
Nov 21 16:18:12 xhz crontab[9861]: (postgres) LIST (postgres)
Nov 21 16:18:12 xhz crontab[9863]: (postgres) REPLACE (postgres)
Nov 21 16:18:12 xhz crontab[9864]: (postgres) LIST (postgres)
Nov 21 16:18:12 xhz crontab[9866]: (postgres) REPLACE (postgres)
Nov 21 16:18:12 xhz crontab[9867]: (postgres) LIST (postgres)
Nov 21 16:18:12 xhz crontab[9869]: (postgres) REPLACE (postgres)
Nov 21 16:18:12 xhz crontab[9870]: (postgres) LIST (postgres)
Nov 21 16:18:12 xhz crontab[9872]: (postgres) REPLACE (postgres)
Nov 21 16:18:12 xhz crontab[9873]: (postgres) LIST (postgres)
Nov 21 16:18:12 xhz crontab[9875]: (postgres) REPLACE (postgres)
Nov 21 16:19:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:19:01 xhz systemd[1]: Started Session 29 of user root.
-- Subject: Unit session-29.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-29.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:19:01 xhz CROND[10197]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:19:01 xhz postfix/pickup[1262]: 7F1F2400D0: uid=0 from=<root>
Nov 21 16:19:01 xhz postfix/cleanup[4088]: 7F1F2400D0: message-id=<20211121081901.7F1F2400D0@xhz.localdomain>
Nov 21 16:19:01 xhz postfix/qmgr[1263]: 7F1F2400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:19:01 xhz postfix/local[1784]: 7F1F2400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:19:01 xhz postfix/cleanup[4088]: 826B7400F2: message-id=<20211121081901.826B7400F2@xhz.localdomain>
Nov 21 16:19:01 xhz postfix/bounce[4090]: 7F1F2400D0: sender non-delivery notification: 826B7400F2
Nov 21 16:19:01 xhz postfix/qmgr[1263]: 826B7400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:19:01 xhz postfix/qmgr[1263]: 7F1F2400D0: removed
Nov 21 16:19:01 xhz postfix/local[1781]: 826B7400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:19:01 xhz postfix/qmgr[1263]: 826B7400F2: removed
Nov 21 16:19:04 xhz sshd[10214]: Accepted password for root from 1.85.38.10 port 17754 ssh2
Nov 21 16:19:04 xhz systemd-logind[644]: New session 30 of user root.
-- Subject: A new session 30 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A new session with the ID 30 has been created for the user root.
-- 
-- The leading process of the session is 10214.
Nov 21 16:19:04 xhz systemd[1]: Started Session 30 of user root.
-- Subject: Unit session-30.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-30.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:19:04 xhz sshd[10214]: pam_unix(sshd:session): session opened for user root by (uid=0)
Nov 21 16:19:04 xhz sshd[10214]: pam_lastlog(sshd:session): corruption detected in /var/log/btmp
Nov 21 16:20:01 xhz systemd[1]: Started Session 32 of user root.
-- Subject: Unit session-32.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-32.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:20:01 xhz systemd[1]: Started Session 31 of user root.
-- Subject: Unit session-31.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-31.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:20:01 xhz CROND[10573]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:20:01 xhz CROND[10574]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Nov 21 16:20:01 xhz postfix/pickup[1262]: 8D080400D0: uid=0 from=<root>
Nov 21 16:20:01 xhz postfix/cleanup[4088]: 8D080400D0: message-id=<20211121082001.8D080400D0@xhz.localdomain>
Nov 21 16:20:01 xhz postfix/qmgr[1263]: 8D080400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:20:01 xhz postfix/local[1784]: 8D080400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.03, delays=0.02/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:20:01 xhz postfix/cleanup[4088]: 91EAE400C3: message-id=<20211121082001.91EAE400C3@xhz.localdomain>
Nov 21 16:20:01 xhz postfix/qmgr[1263]: 91EAE400C3: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:20:01 xhz postfix/bounce[4090]: 8D080400D0: sender non-delivery notification: 91EAE400C3
Nov 21 16:20:01 xhz postfix/qmgr[1263]: 8D080400D0: removed
Nov 21 16:20:01 xhz postfix/local[1781]: 91EAE400C3: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:20:01 xhz postfix/qmgr[1263]: 91EAE400C3: removed
Nov 21 16:20:12 xhz crontab[10658]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10664]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10666]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10667]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10669]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10670]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10672]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10674]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10676]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10677]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10679]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10726]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10731]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10733]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10734]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10736]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10737]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10739]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10740]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10742]: (postgres) REPLACE (postgres)
Nov 21 16:20:12 xhz crontab[10743]: (postgres) LIST (postgres)
Nov 21 16:20:12 xhz crontab[10745]: (postgres) REPLACE (postgres)
Nov 21 16:21:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:21:01 xhz systemd[1]: Started Session 33 of user root.
-- Subject: Unit session-33.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-33.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:21:01 xhz CROND[11058]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:21:01 xhz postfix/pickup[1262]: 997F8400C3: uid=0 from=<root>
Nov 21 16:21:01 xhz postfix/cleanup[4088]: 997F8400C3: message-id=<20211121082101.997F8400C3@xhz.localdomain>
Nov 21 16:21:01 xhz postfix/qmgr[1263]: 997F8400C3: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:21:01 xhz postfix/local[1784]: 997F8400C3: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:21:01 xhz postfix/cleanup[4088]: 9D240400F2: message-id=<20211121082101.9D240400F2@xhz.localdomain>
Nov 21 16:21:01 xhz postfix/bounce[4090]: 997F8400C3: sender non-delivery notification: 9D240400F2
Nov 21 16:21:01 xhz postfix/qmgr[1263]: 9D240400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:21:01 xhz postfix/qmgr[1263]: 997F8400C3: removed
Nov 21 16:21:01 xhz postfix/local[1781]: 9D240400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:21:01 xhz postfix/qmgr[1263]: 9D240400F2: removed
Nov 21 16:22:01 xhz systemd[1]: Started Session 34 of user root.
-- Subject: Unit session-34.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-34.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:22:01 xhz CROND[11419]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:22:01 xhz postfix/pickup[1262]: A5B11400D0: uid=0 from=<root>
Nov 21 16:22:01 xhz postfix/cleanup[4088]: A5B11400D0: message-id=<20211121082201.A5B11400D0@xhz.localdomain>
Nov 21 16:22:01 xhz postfix/qmgr[1263]: A5B11400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:22:01 xhz postfix/local[1784]: A5B11400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:22:01 xhz postfix/cleanup[4088]: A888F400F2: message-id=<20211121082201.A888F400F2@xhz.localdomain>
Nov 21 16:22:01 xhz postfix/qmgr[1263]: A888F400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:22:01 xhz postfix/bounce[4090]: A5B11400D0: sender non-delivery notification: A888F400F2
Nov 21 16:22:01 xhz postfix/qmgr[1263]: A5B11400D0: removed
Nov 21 16:22:01 xhz postfix/local[1781]: A888F400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:22:01 xhz postfix/qmgr[1263]: A888F400F2: removed
Nov 21 16:22:12 xhz crontab[11504]: (postgres) LIST (postgres)
Nov 21 16:22:12 xhz crontab[11509]: (postgres) LIST (postgres)
Nov 21 16:22:12 xhz crontab[11511]: (postgres) REPLACE (postgres)
Nov 21 16:22:12 xhz crontab[11512]: (postgres) LIST (postgres)
Nov 21 16:22:12 xhz crontab[11514]: (postgres) REPLACE (postgres)
Nov 21 16:22:12 xhz crontab[11515]: (postgres) LIST (postgres)
Nov 21 16:22:12 xhz crontab[11517]: (postgres) REPLACE (postgres)
Nov 21 16:22:12 xhz crontab[11518]: (postgres) LIST (postgres)
Nov 21 16:22:12 xhz crontab[11520]: (postgres) REPLACE (postgres)
Nov 21 16:22:12 xhz crontab[11521]: (postgres) LIST (postgres)
Nov 21 16:22:12 xhz crontab[11523]: (postgres) REPLACE (postgres)
Nov 21 16:22:13 xhz crontab[11566]: (postgres) LIST (postgres)
Nov 21 16:22:13 xhz crontab[11572]: (postgres) LIST (postgres)
Nov 21 16:22:13 xhz crontab[11574]: (postgres) REPLACE (postgres)
Nov 21 16:22:13 xhz crontab[11576]: (postgres) LIST (postgres)
Nov 21 16:22:13 xhz crontab[11578]: (postgres) REPLACE (postgres)
Nov 21 16:22:13 xhz crontab[11579]: (postgres) LIST (postgres)
Nov 21 16:22:13 xhz crontab[11581]: (postgres) REPLACE (postgres)
Nov 21 16:22:13 xhz crontab[11582]: (postgres) LIST (postgres)
Nov 21 16:22:13 xhz crontab[11584]: (postgres) REPLACE (postgres)
Nov 21 16:22:13 xhz crontab[11585]: (postgres) LIST (postgres)
Nov 21 16:22:13 xhz crontab[11587]: (postgres) REPLACE (postgres)
Nov 21 16:23:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:23:01 xhz systemd[1]: Started Session 35 of user root.
-- Subject: Unit session-35.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-35.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:23:01 xhz CROND[11882]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:23:01 xhz postfix/pickup[1262]: B2532400D0: uid=0 from=<root>
Nov 21 16:23:01 xhz postfix/cleanup[4088]: B2532400D0: message-id=<20211121082301.B2532400D0@xhz.localdomain>
Nov 21 16:23:01 xhz postfix/qmgr[1263]: B2532400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:23:01 xhz postfix/local[1784]: B2532400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:23:01 xhz postfix/cleanup[4088]: B65A5400F2: message-id=<20211121082301.B65A5400F2@xhz.localdomain>
Nov 21 16:23:01 xhz postfix/qmgr[1263]: B65A5400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:23:01 xhz postfix/bounce[4090]: B2532400D0: sender non-delivery notification: B65A5400F2
Nov 21 16:23:01 xhz postfix/qmgr[1263]: B2532400D0: removed
Nov 21 16:23:01 xhz postfix/local[1781]: B65A5400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:23:01 xhz postfix/qmgr[1263]: B65A5400F2: removed
Nov 21 16:24:01 xhz systemd[1]: Started Session 36 of user root.
-- Subject: Unit session-36.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-36.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:24:01 xhz CROND[12241]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:24:01 xhz postfix/pickup[1262]: BE8F7400D0: uid=0 from=<root>
Nov 21 16:24:01 xhz postfix/cleanup[4088]: BE8F7400D0: message-id=<20211121082401.BE8F7400D0@xhz.localdomain>
Nov 21 16:24:01 xhz postfix/qmgr[1263]: BE8F7400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:24:01 xhz postfix/local[1784]: BE8F7400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:24:01 xhz postfix/cleanup[4088]: C2CD5400F2: message-id=<20211121082401.C2CD5400F2@xhz.localdomain>
Nov 21 16:24:01 xhz postfix/bounce[4090]: BE8F7400D0: sender non-delivery notification: C2CD5400F2
Nov 21 16:24:01 xhz postfix/qmgr[1263]: C2CD5400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:24:01 xhz postfix/qmgr[1263]: BE8F7400D0: removed
Nov 21 16:24:01 xhz postfix/local[1781]: C2CD5400F2: to=<root@xhz.localdomain>, relay=local, delay=0.02, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:24:01 xhz postfix/qmgr[1263]: C2CD5400F2: removed
Nov 21 16:24:13 xhz crontab[12324]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12329]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12331]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12332]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12334]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12335]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12337]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12338]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12340]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12341]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12343]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12390]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12396]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12398]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12399]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12401]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12402]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12404]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12405]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12407]: (postgres) REPLACE (postgres)
Nov 21 16:24:13 xhz crontab[12408]: (postgres) LIST (postgres)
Nov 21 16:24:13 xhz crontab[12410]: (postgres) REPLACE (postgres)
Nov 21 16:25:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:25:01 xhz systemd[1]: Started Session 37 of user root.
-- Subject: Unit session-37.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-37.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:25:01 xhz CROND[12720]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:25:01 xhz postfix/pickup[1262]: CCF36400C3: uid=0 from=<root>
Nov 21 16:25:01 xhz postfix/cleanup[4088]: CCF36400C3: message-id=<20211121082501.CCF36400C3@xhz.localdomain>
Nov 21 16:25:01 xhz postfix/qmgr[1263]: CCF36400C3: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:25:01 xhz postfix/local[1784]: CCF36400C3: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:25:01 xhz postfix/cleanup[4088]: D03EA400D0: message-id=<20211121082501.D03EA400D0@xhz.localdomain>
Nov 21 16:25:01 xhz postfix/bounce[4090]: CCF36400C3: sender non-delivery notification: D03EA400D0
Nov 21 16:25:01 xhz postfix/qmgr[1263]: D03EA400D0: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:25:01 xhz postfix/qmgr[1263]: CCF36400C3: removed
Nov 21 16:25:01 xhz postfix/local[1781]: D03EA400D0: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:25:01 xhz postfix/qmgr[1263]: D03EA400D0: removed
Nov 21 16:26:01 xhz systemd[1]: Started Session 38 of user root.
-- Subject: Unit session-38.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-38.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:26:01 xhz CROND[13076]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:26:01 xhz postfix/pickup[1262]: D9670400D0: uid=0 from=<root>
Nov 21 16:26:01 xhz postfix/cleanup[4088]: D9670400D0: message-id=<20211121082601.D9670400D0@xhz.localdomain>
Nov 21 16:26:01 xhz postfix/qmgr[1263]: D9670400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:26:01 xhz postfix/local[1784]: D9670400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:26:01 xhz postfix/cleanup[4088]: DCCB6400F2: message-id=<20211121082601.DCCB6400F2@xhz.localdomain>
Nov 21 16:26:01 xhz postfix/qmgr[1263]: DCCB6400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:26:01 xhz postfix/bounce[4090]: D9670400D0: sender non-delivery notification: DCCB6400F2
Nov 21 16:26:01 xhz postfix/qmgr[1263]: D9670400D0: removed
Nov 21 16:26:01 xhz postfix/local[1781]: DCCB6400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:26:01 xhz postfix/qmgr[1263]: DCCB6400F2: removed
Nov 21 16:26:13 xhz crontab[13164]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13169]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13171]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13172]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13174]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13175]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13177]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13178]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13180]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13181]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13183]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13226]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13232]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13234]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13235]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13237]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13238]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13240]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13241]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13243]: (postgres) REPLACE (postgres)
Nov 21 16:26:13 xhz crontab[13244]: (postgres) LIST (postgres)
Nov 21 16:26:13 xhz crontab[13246]: (postgres) REPLACE (postgres)
Nov 21 16:27:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:27:01 xhz systemd[1]: Started Session 39 of user root.
-- Subject: Unit session-39.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-39.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:27:01 xhz CROND[13560]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:27:01 xhz postfix/pickup[1262]: E606B400D0: uid=0 from=<root>
Nov 21 16:27:01 xhz postfix/cleanup[4088]: E606B400D0: message-id=<20211121082701.E606B400D0@xhz.localdomain>
Nov 21 16:27:01 xhz postfix/qmgr[1263]: E606B400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:27:01 xhz postfix/local[1784]: E606B400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.03, delays=0.02/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:27:01 xhz postfix/cleanup[4088]: ECD3F400F2: message-id=<20211121082701.ECD3F400F2@xhz.localdomain>
Nov 21 16:27:01 xhz postfix/bounce[4090]: E606B400D0: sender non-delivery notification: ECD3F400F2
Nov 21 16:27:01 xhz postfix/qmgr[1263]: ECD3F400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:27:01 xhz postfix/qmgr[1263]: E606B400D0: removed
Nov 21 16:27:01 xhz postfix/local[1781]: ECD3F400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:27:01 xhz postfix/qmgr[1263]: ECD3F400F2: removed
Nov 21 16:28:01 xhz systemd[1]: Started Session 40 of user root.
-- Subject: Unit session-40.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-40.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:28:01 xhz CROND[13916]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:28:01 xhz postfix/pickup[1262]: F2636400AB: uid=0 from=<root>
Nov 21 16:28:01 xhz postfix/cleanup[4088]: F2636400AB: message-id=<20211121082801.F2636400AB@xhz.localdomain>
Nov 21 16:28:01 xhz postfix/qmgr[1263]: F2636400AB: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:28:02 xhz postfix/local[1784]: F2636400AB: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:28:02 xhz postfix/cleanup[4088]: 01D69400F2: message-id=<20211121082802.01D69400F2@xhz.localdomain>
Nov 21 16:28:02 xhz postfix/bounce[4090]: F2636400AB: sender non-delivery notification: 01D69400F2
Nov 21 16:28:02 xhz postfix/qmgr[1263]: 01D69400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:28:02 xhz postfix/qmgr[1263]: F2636400AB: removed
Nov 21 16:28:02 xhz postfix/local[1781]: 01D69400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:28:02 xhz postfix/qmgr[1263]: 01D69400F2: removed
Nov 21 16:28:14 xhz crontab[14005]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14010]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14012]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14013]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14015]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14016]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14018]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14019]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14021]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14022]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14024]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14067]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14072]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14074]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14075]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14077]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14078]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14080]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14081]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14083]: (postgres) REPLACE (postgres)
Nov 21 16:28:14 xhz crontab[14084]: (postgres) LIST (postgres)
Nov 21 16:28:14 xhz crontab[14086]: (postgres) REPLACE (postgres)
Nov 21 16:28:54 xhz sshd[14352]: Accepted password for root from 1.85.38.10 port 39299 ssh2
Nov 21 16:28:54 xhz systemd-logind[644]: New session 41 of user root.
-- Subject: A new session 41 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A new session with the ID 41 has been created for the user root.
-- 
-- The leading process of the session is 14352.
Nov 21 16:28:54 xhz systemd[1]: Started Session 41 of user root.
-- Subject: Unit session-41.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-41.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:28:54 xhz sshd[14352]: pam_unix(sshd:session): session opened for user root by (uid=0)
Nov 21 16:28:54 xhz sshd[14352]: pam_lastlog(sshd:session): corruption detected in /var/log/btmp
Nov 21 16:29:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:29:02 xhz systemd[1]: Started Session 42 of user root.
-- Subject: Unit session-42.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-42.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:29:02 xhz CROND[14411]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:29:02 xhz postfix/pickup[1262]: 0CCCE400D0: uid=0 from=<root>
Nov 21 16:29:02 xhz postfix/cleanup[4088]: 0CCCE400D0: message-id=<20211121082902.0CCCE400D0@xhz.localdomain>
Nov 21 16:29:02 xhz postfix/qmgr[1263]: 0CCCE400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:29:02 xhz postfix/local[1784]: 0CCCE400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.03, delays=0.02/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:29:02 xhz postfix/cleanup[4088]: 1090D400C3: message-id=<20211121082902.1090D400C3@xhz.localdomain>
Nov 21 16:29:02 xhz postfix/bounce[4090]: 0CCCE400D0: sender non-delivery notification: 1090D400C3
Nov 21 16:29:02 xhz postfix/qmgr[1263]: 1090D400C3: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:29:02 xhz postfix/qmgr[1263]: 0CCCE400D0: removed
Nov 21 16:29:02 xhz postfix/local[1781]: 1090D400C3: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:29:02 xhz postfix/qmgr[1263]: 1090D400C3: removed
Nov 21 16:30:01 xhz systemd[1]: Started Session 43 of user root.
-- Subject: Unit session-43.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-43.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:30:01 xhz systemd[1]: Started Session 44 of user root.
-- Subject: Unit session-44.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-44.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:30:01 xhz CROND[14757]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Nov 21 16:30:01 xhz CROND[14758]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:30:01 xhz postfix/pickup[1262]: 1ABFD400D0: uid=0 from=<root>
Nov 21 16:30:01 xhz postfix/cleanup[4088]: 1ABFD400D0: message-id=<20211121083001.1ABFD400D0@xhz.localdomain>
Nov 21 16:30:01 xhz postfix/qmgr[1263]: 1ABFD400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:30:01 xhz postfix/local[1784]: 1ABFD400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.03, delays=0.01/0/0/0.02, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:30:01 xhz postfix/cleanup[4088]: 22084400F2: message-id=<20211121083001.22084400F2@xhz.localdomain>
Nov 21 16:30:01 xhz postfix/bounce[14767]: 1ABFD400D0: sender non-delivery notification: 22084400F2
Nov 21 16:30:01 xhz postfix/qmgr[1263]: 22084400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:30:01 xhz postfix/qmgr[1263]: 1ABFD400D0: removed
Nov 21 16:30:01 xhz postfix/local[1781]: 22084400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:30:01 xhz postfix/qmgr[1263]: 22084400F2: removed
Nov 21 16:30:14 xhz crontab[14862]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14867]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14869]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14870]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14872]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14873]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14875]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14876]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14878]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14879]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14881]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14925]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14930]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14932]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14933]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14935]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14936]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14938]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14939]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14941]: (postgres) REPLACE (postgres)
Nov 21 16:30:14 xhz crontab[14942]: (postgres) LIST (postgres)
Nov 21 16:30:14 xhz crontab[14944]: (postgres) REPLACE (postgres)
Nov 21 16:31:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:31:01 xhz systemd[1]: Started Session 45 of user root.
-- Subject: Unit session-45.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-45.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:31:01 xhz CROND[15243]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:31:01 xhz postfix/pickup[1262]: 26C59400D0: uid=0 from=<root>
Nov 21 16:31:01 xhz postfix/cleanup[4088]: 26C59400D0: message-id=<20211121083101.26C59400D0@xhz.localdomain>
Nov 21 16:31:01 xhz postfix/qmgr[1263]: 26C59400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:31:01 xhz postfix/local[1784]: 26C59400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:31:01 xhz postfix/cleanup[4088]: 29E56400F2: message-id=<20211121083101.29E56400F2@xhz.localdomain>
Nov 21 16:31:01 xhz postfix/bounce[14767]: 26C59400D0: sender non-delivery notification: 29E56400F2
Nov 21 16:31:01 xhz postfix/qmgr[1263]: 29E56400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:31:01 xhz postfix/qmgr[1263]: 26C59400D0: removed
Nov 21 16:31:01 xhz postfix/local[1781]: 29E56400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:31:01 xhz postfix/qmgr[1263]: 29E56400F2: removed
Nov 21 16:31:30 xhz systemd[1]: Time has been changed
-- Subject: Time change
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The system clock has been changed to REALTIME microseconds after January 1st, 1970.
Nov 21 16:32:01 xhz auditd[608]: Audit daemon rotating log files
Nov 21 16:32:01 xhz systemd[1]: Started Session 46 of user root.
-- Subject: Unit session-46.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-46.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:32:01 xhz CROND[15611]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:32:01 xhz postfix/pickup[1262]: CB0C9400D0: uid=0 from=<root>
Nov 21 16:32:01 xhz postfix/cleanup[4088]: CB0C9400D0: message-id=<20211121083201.CB0C9400D0@xhz.localdomain>
Nov 21 16:32:01 xhz postfix/qmgr[1263]: CB0C9400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:32:01 xhz postfix/local[1784]: CB0C9400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.05, delays=0.04/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:32:01 xhz postfix/cleanup[4088]: CDFEA400F2: message-id=<20211121083201.CDFEA400F2@xhz.localdomain>
Nov 21 16:32:01 xhz postfix/qmgr[1263]: CDFEA400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:32:01 xhz postfix/bounce[14767]: CB0C9400D0: sender non-delivery notification: CDFEA400F2
Nov 21 16:32:01 xhz postfix/qmgr[1263]: CB0C9400D0: removed
Nov 21 16:32:01 xhz postfix/local[1781]: CDFEA400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:32:01 xhz postfix/qmgr[1263]: CDFEA400F2: removed
Nov 21 16:32:14 xhz crontab[15702]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15707]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15709]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15710]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15712]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15713]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15715]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15716]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15718]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15719]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15721]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15764]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15769]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15771]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15772]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15774]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15775]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15777]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15778]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15780]: (postgres) REPLACE (postgres)
Nov 21 16:32:14 xhz crontab[15781]: (postgres) LIST (postgres)
Nov 21 16:32:14 xhz crontab[15783]: (postgres) REPLACE (postgres)
Nov 21 16:32:18 xhz polkitd[638]: Registered Authentication Agent for unix-process:15838:206923 (system bus name :1.114 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 21 16:32:18 xhz polkitd[638]: Unregistered Authentication Agent for unix-process:15838:206923 (system bus name :1.114, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Nov 21 16:32:29 xhz polkitd[638]: Registered Authentication Agent for unix-process:15914:208058 (system bus name :1.115 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 21 16:32:29 xhz systemd[1]: Starting PostgreSQL 10 database server...
-- Subject: Unit postgresql-10.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit postgresql-10.service has begun starting up.
Nov 21 16:32:29 xhz postmaster[15926]: 2021-11-21 08:32:29.974 GMT [15926] LOG:  unrecognized configuration parameter "tcpip_socket" in file "/mydata/pg10data/postgresql.conf" line 62
Nov 21 16:32:29 xhz postmaster[15926]: 2021-11-21 08:32:29.974 GMT [15926] FATAL:  configuration file "/mydata/pg10data/postgresql.conf" contains errors
Nov 21 16:32:29 xhz systemd[1]: postgresql-10.service: main process exited, code=exited, status=1/FAILURE
Nov 21 16:32:29 xhz systemd[1]: Failed to start PostgreSQL 10 database server.
-- Subject: Unit postgresql-10.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit postgresql-10.service has failed.
-- 
-- The result is failed.
Nov 21 16:32:29 xhz systemd[1]: Unit postgresql-10.service entered failed state.
Nov 21 16:32:29 xhz systemd[1]: postgresql-10.service failed.
Nov 21 16:32:29 xhz polkitd[638]: Unregistered Authentication Agent for unix-process:15914:208058 (system bus name :1.115, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Nov 21 16:33:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:33:01 xhz systemd[1]: Started Session 47 of user root.
-- Subject: Unit session-47.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-47.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:33:01 xhz CROND[16092]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:33:01 xhz postfix/pickup[1262]: D77D7400D0: uid=0 from=<root>
Nov 21 16:33:01 xhz postfix/cleanup[4088]: D77D7400D0: message-id=<20211121083301.D77D7400D0@xhz.localdomain>
Nov 21 16:33:01 xhz postfix/qmgr[1263]: D77D7400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:33:01 xhz postfix/local[1784]: D77D7400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:33:01 xhz postfix/cleanup[4088]: DAAE7400F2: message-id=<20211121083301.DAAE7400F2@xhz.localdomain>
Nov 21 16:33:01 xhz postfix/bounce[14767]: D77D7400D0: sender non-delivery notification: DAAE7400F2
Nov 21 16:33:01 xhz postfix/qmgr[1263]: D77D7400D0: removed
Nov 21 16:33:01 xhz postfix/qmgr[1263]: DAAE7400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:33:01 xhz postfix/local[1781]: DAAE7400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:33:01 xhz postfix/qmgr[1263]: DAAE7400F2: removed
Nov 21 16:34:01 xhz systemd[1]: Started Session 48 of user root.
-- Subject: Unit session-48.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-48.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:34:01 xhz CROND[16450]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:34:01 xhz postfix/pickup[1262]: E4582400C3: uid=0 from=<root>
Nov 21 16:34:01 xhz postfix/cleanup[4088]: E4582400C3: message-id=<20211121083401.E4582400C3@xhz.localdomain>
Nov 21 16:34:01 xhz postfix/qmgr[1263]: E4582400C3: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:34:01 xhz postfix/local[1784]: E4582400C3: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:34:01 xhz postfix/cleanup[4088]: E826D400F2: message-id=<20211121083401.E826D400F2@xhz.localdomain>
Nov 21 16:34:01 xhz postfix/bounce[14767]: E4582400C3: sender non-delivery notification: E826D400F2
Nov 21 16:34:01 xhz postfix/qmgr[1263]: E826D400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:34:01 xhz postfix/qmgr[1263]: E4582400C3: removed
Nov 21 16:34:01 xhz postfix/local[1781]: E826D400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:34:01 xhz postfix/qmgr[1263]: E826D400F2: removed
Nov 21 16:34:15 xhz crontab[16549]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16554]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16556]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16557]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16559]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16560]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16562]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16563]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16565]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16566]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16568]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16611]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16616]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16618]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16619]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16621]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16622]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16624]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16625]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16627]: (postgres) REPLACE (postgres)
Nov 21 16:34:15 xhz crontab[16628]: (postgres) LIST (postgres)
Nov 21 16:34:15 xhz crontab[16630]: (postgres) REPLACE (postgres)
Nov 21 16:35:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:35:01 xhz systemd[1]: Started Session 49 of user root.
-- Subject: Unit session-49.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-49.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:35:01 xhz CROND[16936]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:35:01 xhz postfix/pickup[1262]: F0DAD400D0: uid=0 from=<root>
Nov 21 16:35:01 xhz postfix/cleanup[4088]: F0DAD400D0: message-id=<20211121083501.F0DAD400D0@xhz.localdomain>
Nov 21 16:35:01 xhz postfix/qmgr[1263]: F0DAD400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:35:02 xhz postfix/local[1784]: F0DAD400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:35:02 xhz postfix/cleanup[4088]: 01066400F2: message-id=<20211121083502.01066400F2@xhz.localdomain>
Nov 21 16:35:02 xhz postfix/bounce[14767]: F0DAD400D0: sender non-delivery notification: 01066400F2
Nov 21 16:35:02 xhz postfix/qmgr[1263]: 01066400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:35:02 xhz postfix/qmgr[1263]: F0DAD400D0: removed
Nov 21 16:35:02 xhz postfix/local[1781]: 01066400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:35:02 xhz postfix/qmgr[1263]: 01066400F2: removed
Nov 21 16:36:02 xhz systemd[1]: Started Session 50 of user root.
-- Subject: Unit session-50.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-50.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:36:02 xhz CROND[17293]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:36:02 xhz postfix/pickup[1262]: 08EE0400C3: uid=0 from=<root>
Nov 21 16:36:02 xhz postfix/cleanup[4088]: 08EE0400C3: message-id=<20211121083602.08EE0400C3@xhz.localdomain>
Nov 21 16:36:02 xhz postfix/qmgr[1263]: 08EE0400C3: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:36:02 xhz postfix/local[1784]: 08EE0400C3: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:36:02 xhz postfix/cleanup[4088]: 0CA30400F2: message-id=<20211121083602.0CA30400F2@xhz.localdomain>
Nov 21 16:36:02 xhz postfix/bounce[14767]: 08EE0400C3: sender non-delivery notification: 0CA30400F2
Nov 21 16:36:02 xhz postfix/qmgr[1263]: 0CA30400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:36:02 xhz postfix/qmgr[1263]: 08EE0400C3: removed
Nov 21 16:36:02 xhz postfix/local[1781]: 0CA30400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:36:02 xhz postfix/qmgr[1263]: 0CA30400F2: removed
Nov 21 16:36:15 xhz crontab[17390]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17395]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17397]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17398]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17400]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17401]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17403]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17404]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17406]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17407]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17409]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17452]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17457]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17459]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17460]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17462]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17463]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17465]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17466]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17468]: (postgres) REPLACE (postgres)
Nov 21 16:36:15 xhz crontab[17470]: (postgres) LIST (postgres)
Nov 21 16:36:15 xhz crontab[17472]: (postgres) REPLACE (postgres)
Nov 21 16:37:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:37:01 xhz systemd[1]: Started Session 51 of user root.
-- Subject: Unit session-51.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-51.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:37:01 xhz CROND[17769]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:37:01 xhz postfix/pickup[1262]: 1560E400D0: uid=0 from=<root>
Nov 21 16:37:01 xhz postfix/cleanup[4088]: 1560E400D0: message-id=<20211121083701.1560E400D0@xhz.localdomain>
Nov 21 16:37:01 xhz postfix/qmgr[1263]: 1560E400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:37:01 xhz postfix/local[1784]: 1560E400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.04, delays=0.03/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:37:01 xhz postfix/cleanup[4088]: 1DF45400F2: message-id=<20211121083701.1DF45400F2@xhz.localdomain>
Nov 21 16:37:01 xhz postfix/bounce[14767]: 1560E400D0: sender non-delivery notification: 1DF45400F2
Nov 21 16:37:01 xhz postfix/qmgr[1263]: 1DF45400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:37:01 xhz postfix/qmgr[1263]: 1560E400D0: removed
Nov 21 16:37:01 xhz postfix/local[1781]: 1DF45400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:37:01 xhz postfix/qmgr[1263]: 1DF45400F2: removed
Nov 21 16:38:01 xhz systemd[1]: Started Session 52 of user root.
-- Subject: Unit session-52.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-52.scope has finished starting up.
-- 
-- The start-up result is done.
Nov 21 16:38:01 xhz CROND[18124]: (root) CMD (cd /root; mkdir 1)
Nov 21 16:38:01 xhz postfix/pickup[1262]: 21E15400D0: uid=0 from=<root>
Nov 21 16:38:01 xhz postfix/cleanup[4088]: 21E15400D0: message-id=<20211121083801.21E15400D0@xhz.localdomain>
Nov 21 16:38:01 xhz postfix/qmgr[1263]: 21E15400D0: from=<root@xhz.localdomain>, size=769, nrcpt=1 (queue active)
Nov 21 16:38:01 xhz postfix/local[1784]: 21E15400D0: to=<root@xhz.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:38:01 xhz postfix/cleanup[4088]: 25872400F2: message-id=<20211121083801.25872400F2@xhz.localdomain>
Nov 21 16:38:01 xhz postfix/bounce[14767]: 21E15400D0: sender non-delivery notification: 25872400F2
Nov 21 16:38:01 xhz postfix/qmgr[1263]: 25872400F2: from=<>, size=2610, nrcpt=1 (queue active)
Nov 21 16:38:01 xhz postfix/qmgr[1263]: 21E15400D0: removed
Nov 21 16:38:01 xhz postfix/local[1781]: 25872400F2: to=<root@xhz.localdomain>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Nov 21 16:38:01 xhz postfix/qmgr[1263]: 25872400F2: removed
Nov 21 16:38:16 xhz crontab[18232]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18238]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18240]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18241]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18243]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18247]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18249]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18250]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18252]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18253]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18255]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18299]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18304]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18306]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18307]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18309]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18310]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18312]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18313]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18315]: (postgres) REPLACE (postgres)
Nov 21 16:38:16 xhz crontab[18316]: (postgres) LIST (postgres)
Nov 21 16:38:16 xhz crontab[18318]: (postgres) REPLACE (postgres)
Nov 21 16:39:01 xhz crond[654]: (postgres) RELOAD (/var/spool/cron/postgres)
Nov 21 16:39:01 xhz systemd[1]: Started Session 53 of user root.
  • 然后也可以总结一下,就是说,我们的项目是突然蹦了,在过程中我们也发现了有极个别线程会占用CPU快到100%。单节点在运行一段时间后,CPU 的使用会飙升,一旦飙升,一般怀疑某个业务逻辑的计算量太大,或者是触发了死循环(比如著名的 HashMap 高并发引起的死循环),但排查到最后其实是 GC 的问题
    • 那咱们一般咱们除了直接问题出现直接导致项目崩了,客户用不了了给咱们反馈回来了之外,肯定要通过什么监视工具、或者啥监视页面啥的监视到问题。所以,这里呢,一般有JVM性能分析神器-VisualVM,脱颖而出
      • 也可以看这一篇,这一篇也比较详细:VisualVM工具
        java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第1张图片
        java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第2张图片
        java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第3张图片
        java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第4张图片
    • 所以先进行如下排查,排查步骤如下
      • (1)使用 top 命令查询资源占用情况查找到使用 CPU 最多的某个进程记录它的 pid。使用 Shift + P 快捷键可以按 CPU 的使用率进行排序。
        java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第5张图片
        • 也可以使用top -p PID命令,查询指定PID的资源占用情况
      • (2)再次使用 top 命令,加 -H 参数**,查看某个进程中使用 CPU 最多的某个线程**,记录线程的 ID。【top -Hp $pid
        • 也可以使用ps -mp PID -o THREAD,tid,time命令,查询该进程的线程情况
        • 也可以使用ps -mp PID -o THREAD,tid,time | sort -rn命令,将该进程下的线程按资源使用情况倒序展示:
      • (3)使用 printf 函数,将十进制的 tid 转化成十六进制printf %x $tid
        • 使用printf “%x\n” PID命令,将PID转为十六进制的TID,之所以需要将PID转为十六进制是因为在堆栈信息中,PID是以十六进制形式存在的。
      • (4)使用 jstack 命令,查看 Java 进程的线程栈jstack $pid >$pid.log
        • 使用jstack PID | grep TID -A 100命令,查询堆栈信息
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第6张图片
        • 除此之外,我们还可以使用jinfo和jstat命令来查询 Java 进程的启动参数以及 GC 情况
          • 使用jinfo PID命令,查询启动参数
            java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第7张图片
          • 使用jstat -gcutil PID 1000命令,查询 GC 情况:
            java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第8张图片
      • (5)使用 less 命令查看生成的文件,并查找刚才转化的十六进制 tid,找到发生问题的线程上下文。less $pid.log
        • 我们在 jstack 日志搜关键字DEAD,以及中找到了 CPU 使用最多的几个线程id。可以看到问题发生的根源,是我们的堆已经满了,但是又没有发生 OOM,于是 GC 进程就一直在那里回收,回收的效果又非常一般,造成 CPU 升高应用假死。接下来的具体问题排查,就需要把内存 dump 一份下来,使用 MAT 等工具分析具体原因了
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第9张图片
          • 基本概念:
            java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第10张图片
          • MAT分析 dump文件:
            • 内存溢出排除方法思路:
              • 看GC日志 126719K->126719K(126720K) [回收前后内存大小不变或一直增长无减少波动,则可能有内存溢出问题]
              • 生成堆内存 heap dump(某个时间点jvm中所有活跃的对象的堆内存快照)
              • MAT查看: 过滤列举对象(传入或传出的对象引用)、分组排序显示
                java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第11张图片
            • MAT分析dump具体步骤如下:
              • 1.MAT打开dump文件:MAT打开dump文件,生成分析报告: File > Open Heap Dump > Leak Suspects Report
                java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第12张图片
                • 1.Overview:Overview概要信息,比如空间大小、类的数量、对象实例数量、类加载器等等
                  java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第13张图片
                  java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第14张图片
              • 2.Histogram :Overview > Actions > The Histogram (查看堆栈中java类对象[Objects]个数、[Shallow Heap]individual objects此类对象占用大小、[Retained Heap]关联对象占用大小)
                java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第15张图片
                java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第16张图片
                java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第17张图片
              • 3.Dominator Tree:支配树,分析对象的引用关系。 对象内存占用&占比。dominate_tree -> 对象调用堆栈树-查找内存占用最高对象(Retained Heap倒叙排序) -> Paths to GC Roots -> exclude all phantom/weak/soft etc.reference(排除所有虚弱软引用) -查找GC Root线程-> 定位未释放内存代码段
                • Actions > dominator_tree (查看堆中内存占用最高的对象的线程调用堆栈) -> 对象调用堆栈树-查找内存占用最高对象(Retained Heap倒叙排序) -> Paths to GC Roots -> exclude all phantom/weak/soft etc.reference (排除所有虚弱软引用) -查找GC Root线程 -> 查找未释放的内存占用最高的代码逻辑段(很可能是产生内存溢出代码)
                  java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第18张图片
                  java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第19张图片
              • 4.Leak Suspects --自动分析内存溢出可疑点:MAT插件会给出一份可疑的分析报告,非常方便,我们只需结合源代码稍加分析到底哪个Problem才是引发问题真正原因所在。点点看 Details,若有问题你很容易发现你熟悉的代码段。
                java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第20张图片
              • 5.OQL全称为Object Query Language,类似于SQL语句的查询语言,能够用来查询当前内存中满足指定条件的所有的对象。
              • 6.对比dump 堆栈文件:因为我们这个例子很简单,可以通过上面的方法来找到内存泄漏的原因,但是复杂的情况就需要通过对比hpof文件来进行分析了。
                • 使用步骤为:操作应用,生成第一个hpof文件。进行一段时间操作,再生成第二个hpof文件用MAT打开这两个hpof文件。将第一个和第二个hpof文件的Dominator Tree或者Histogram添加到Compare Basket中,之后选中2个文件对比即可
                  java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第21张图片
                  java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第22张图片
    • MAT:全称 Eclipse Memory Analysis Tools 是一个分析 Java堆数据的专业工具:jvm内存溢出/内存泄漏问题分析定位神器,可以计算出内存中对象的实例数量、占用空间大小、引用关系等,看看是谁阻止了垃圾收集器的回收工作,从而定位内存泄漏的原因
      • 什么时候会用到MAT?
        • OutOfMemoryError的时候,触发Full GC,但空间却回收不了,引发内存溢出
        • java应用服务器系统异常,比如load负载飙高,io异常,或者线程死锁等,都可能通过分析堆中的内存对象来定位原因
  • 线上故障排查经验:
    • bug分为系统级别和业务级别bug:
      java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第23张图片
      • 如CPU爆满、服务不可用、甚至服务器宕机等都属于系统级别的bug,重点在于如何迅速解决。
        • 如果是CPU100%,那是由哪个线程,哪个类,甚至是哪个方法导致的?
          • CPU高负载,甚至100%,可以用perf来查看,perf是linux的性能分析工具,核心作用之一就是用来查看热点函数的分布情况。
          • 某一进程存在异常嫌疑,想快速知道它的状态?大部分情况下通过系统告警就可以知道大概问题所在。如发生消息堆积我们就该怀疑消息生产者和消费者的状态,这个时候就要具体去查看消息队列这一进程。可以使用一些轻量级的linux命令,如ps:ps -ef | grep xxx
          • vmstat是Virtual Meomory Statistics(虚拟内存统计)的缩写。它是一个用于监控内存和磁盘使用情况的工具,但是也可以用来查看CPU的一些指标,如中断次数等。使用它可以查看内存使用的详细信息和磁盘的读/写情况。
        • 若是业务流程正常但是部分服务性能拉跨,那么如何快速定位到问题在哪儿?
      • 业务级别bug
        • 出现了业务bug那就纯纯的是开发或测试的锅了。
          • bug确定后第一步一定是先看日志,只要你写需求的时候日志打的全,一般出现了问题日志或者告警都会第一时间推送。通过日志我们可以定位到bug对应代码的位置
          • 第二步就是:看数据,数据是业务应用的核心。若通过日志和页面表现查看到你的主流程是没有问题的,那么下一步就是要确定表的数据是否有问题,数据存在bug的表现会是各方面的,可能是用户反馈,也可能是流程错误,这要取决于你表的设计。我记得项目中有时候会由于下位机的缘故出现空数据,然后就会由于某条逻辑出现bug,系统就崩了【线上数据是重中之重,当你决定要修复数据,在处理之前一定要做好备份,这样起码可以保证事情不会变的更糟。一般情况下修改线上数据这种活都需要你写好SQL,然后经过leader审批再交给DBA来操作,一定不要干出删库跑路这种事哟。】
          • 假设验证了你数据是OK的,那么问题就极大可能出现在了代码层面。修改业务bug最重要的是要将bug点修改掉并且保证其它业务还能正常运行,这是牵一发而动全身的事情,否则bug只会越改越多。
    • 如果一个实例发生了问题,根据情况选择,要不要着急去重启。如果出现的CPU、内存飙高或者日志里出现了OOM异常。第一步是隔离,第二步是保留现场,第三步才是问题排查
      • 隔离:就是把你的这台机器从请求列表里摘除,比如把 nginx 相关的权重设成零
      • 现场保留:查看比如 CPU、系统内存等,通过历史状态可以体现一个趋势性问题,而这些信息的获取一般依靠监控系统的协作。
        java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第24张图片
      • 保留信息:
        • 系统当前网络连接:ss -antp > $DUMP_DIR/ss.dump 2>&1。后续的处理,可通过查看各种网络连接状态的梳理,来排查 TIME_WAIT 或者 CLOSE_WAIT,或者其他连接过高的问题,非常有用
          • 使用 ss 命令而不是 netstat 的原因,是因为 netstat 在网络连接非常多的情况下,执行非常缓慢
        • 网络状态统计:netstat -s > $DUMP_DIR/netstat-s.dump 2>&1,它能够按照各个协议进行统计输出,对把握当时整个网络状态,有非常大的作用。
          • sar -n DEV 1 2 > $DUMP_DIR/sar-traffic.dump 2>&1,在一些速度非常高的模块上,比如 Redis、Kafka,就经常发生跑满网卡的情况。表现形式就是网络通信非常缓慢
        • 进程资源:lsof -p $PID > $DUMP_DIR/lsof-$PID.dump通过查看进程,能看到打开了哪些文件,可以以进程的维度来查看整个资源的使用情况,包括每条网络连接、每个打开的文件句柄。同时,也可以很容易的看到连接到了哪些服务器、使用了哪些资源。这个命令在资源非常多的情况下,输出稍慢,请耐心等待。
        • CPU 资源:主要用于输出当前系统的 CPU 和负载,便于事后排查。
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第25张图片
        • I/O 资源:iostat -x > $DUMP_DIR/iostat.dump 2>&1。一般,以计算为主的服务节点,I/O 资源会比较正常,但有时也会发生问题,比如日志输出过多,或者磁盘问题等。此命令可以输出每块磁盘的基本性能信息,用来排查 I/O 问题。
        • 内存问题:free -h > $DUMP_DIR/free.dump 2>&1。free 命令能够大体展现操作系统的内存概况,这是故障排查中一个非常重要的点,比如 SWAP 影响了 GC,SLAB 区挤占了 JVM 的内存
        • 其他全局
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第26张图片
        • 进程快照、dump 堆信息
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第27张图片
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第28张图片
          • kill -3 $PID:有时候,jstack 并不能够运行,有很多原因,比如 Java 进程几乎不响应了等之类的情况。我们会尝试 向进程发送 kill -3 信号,这个信号将会打印 jstack 的 trace 信息到日志文件中,是 jstack 的一个替补方案
          • gcore -o $DUMP_DIR/core P I D :对于 j m a p 无法执行的问题,也有替补,那就是 G D B 组件中的 g c o r e ,将会生成一个 c o r e 文件。我们可以使用如下的命令去生成 d u m p : PID:对于 jmap 无法执行的问题,也有替补,那就是 GDB 组件中的 gcore,将会生成一个 core 文件。我们可以使用如下的命令去生成 dump: PID:对于jmap无法执行的问题,也有替补,那就是GDB组件中的gcore,将会生成一个core文件。我们可以使用如下的命令去生成dump{JDK_BIN}jhsdb jmap --exe ${JDK}java --core $DUMP_DIR/core --binaryheap
        • 内存泄漏的现象
          java基础巩固-宇宙第一AiYWM:为了维持生计,做项目经验之~SSM项目错误集锦Part3(项目蹦+pg数据库坏+100%-->线上故障排查经验【业务bug第一步一定是先看日志,写好日志】)~整起_第29张图片

巨人的肩膀:
库森老师关于线上故障排查的经验
深入理解Java虚拟机
极简架构老师的文章

你可能感兴趣的:(数据库,java,postgresql)