Mysql 非正常关闭，启动异常

2023-11-01T08:21:22.574390Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 13014483
2023-11-01T08:21:22.574401Z 0 [Note] InnoDB: Database was not shutdown normally!
2023-11-01T08:21:22.574470Z 0 [Note] InnoDB: Starting crash recovery.
2023-11-01 16:21:22 0x7f8f2d4ed700  InnoDB: Assertion failure in thread 140252917192448 in file os0file.cc line 3124

[root@sealos-k8s-node-08 ~]# docker logs dae5f49046b8
2023-11-01 16:21:21+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01 16:21:22+08:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-01 16:21:22+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01T08:21:22.487449Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-11-01T08:21:22.497761Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-x7b75.lower-test
2023-11-01T08:21:22.497865Z 0 [Note] mysqld (mysqld 5.7.37) starting as process 1 ...
2023-11-01T08:21:22.509062Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-x7b75.lower-test
2023-11-01T08:21:22.515635Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-x7b75.lower-test
2023-11-01T08:21:22.517729Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-11-01T08:21:22.517753Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-11-01T08:21:22.517757Z 0 [Note] InnoDB: Uses event mutexes
2023-11-01T08:21:22.517761Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-11-01T08:21:22.517763Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-11-01T08:21:22.517766Z 0 [Note] InnoDB: Using Linux native AIO
2023-11-01T08:21:22.518076Z 0 [Note] InnoDB: Number of pools: 1
2023-11-01T08:21:22.518206Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-11-01T08:21:22.520016Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-11-01T08:21:22.528689Z 0 [Note] InnoDB: Completed initialization of buffer pool
2023-11-01T08:21:22.531479Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-11-01T08:21:22.565931Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2023-11-01T08:21:22.574332Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 13014474
2023-11-01T08:21:22.574390Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 13014483
2023-11-01T08:21:22.574401Z 0 [Note] InnoDB: Database was not shutdown normally!
2023-11-01T08:21:22.574470Z 0 [Note] InnoDB: Starting crash recovery.
2023-11-01 16:21:22 0x7f8f2d4ed700  InnoDB: Assertion failure in thread 140252917192448 in file os0file.cc line 3124
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:21:22 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68197 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0x7f8f47dafb6c]
mysqld(handle_fatal_signal+0x501)[0x7f8f476aff31]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f8f46f75730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f8f46a507bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f8f46a3b535]
mysqld(+0x6c3df1)[0x7f8f47676df1]
mysqld(_Z18os_file_flush_funci+0x20c)[0x7f8f47fcb43c]
mysqld(_Z9fil_flushm+0x320)[0x7f8f48186b10]
mysqld(_Z15log_io_completeP11log_group_t+0x459)[0x7f8f47fa4f69]
mysqld(_Z12fil_aio_waitm+0xe0)[0x7f8f481803e0]
mysqld(io_handler_thread+0xa8)[0x7f8f4807b158]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f8f46f6afa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8f46b124cf]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.


-bash-4.2# kubectl  get po | grep mysql
dbapi-mysql-m6xlq                                           0/1     CrashLoopBackOff         1 (69s ago)       14s
hive-mysql-metastore-qs6r2                                  1/1     Running                  2 (24d ago)       226d
mysql-hive-metastore-glusterfs-4lxjf                        1/1     Running                  0                 7d6h

innodb_force_recovery=1

http://blog.chinaunix.net/uid-26950862-id-5593732.html

-bash-4.2# cat mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  mysqld.cnf: |-
    [mysqld]
    pid-file        = /var/run/mysqld/mysqld.pid
    socket          = /var/run/mysqld/mysqld.sock
    datadir         = /var/lib/mysql
    #log-error      = /var/log/mysql/error.log
    ## By default we only accept connections from localhost
    ##bind-address   = 127.0.0.1
    ## Disabling symbolic-links is recommended to prevent assorted security risks
    #symbolic-links=0
    symbolic-links=0
    log-bin=/var/lib/mysql/mysql-bin
    server-id=1

    group_concat_max_len = 10240
    # 最大睡眠时间
    wait_timeout=300
    # 超时时间设置
    interactive_timeout = 500
    #### 
    innodb_force_recovery=1
-bash-4.2# cat mysql-dbapi.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    app: dbapi-mysql
  name: dbapi-mysql
spec:
  replicas: 1
  selector:
    app: dbapi-mysql
  template:
    metadata:
      labels:
        app: dbapi-mysql
    spec:
      nodeSelector:
        doris: doris-fe
      containers:
        - name: dbapi-mysql
          image: docker-registry-node:5000/mysql:5.7.37
          ports:
          - containerPort: 3306
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: MYSQL_ROOT_PASSWORD
              value: "123456"
          volumeMounts:
            - name: mysql-configmap-volume
              mountPath: /etc/mysql/mysql.conf.d
            - name: mysql-volume
              mountPath: /var/lib/mysql
      volumes:
      - name: mysql-volume
        persistentVolumeClaim:
          claimName: mysql-dbapi-pvc
      - name: mysql-configmap-volume
        configMap:
          name: mysql-config
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: dbapi-mysql-svc
  name: dbapi-mysql-svc
spec:
  type: NodePort
  ports:
  - port: 3306
    targetPort: 3306
    nodePort: 32310
    name: mysql-port
  selector:
    app: dbapi-mysql

Errcode: 30 - Read-only file system

-bash-4.2# kubectl  logs dbapi-mysql-m6xlq
2023-11-01 17:15:27+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01 17:15:28+08:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-01 17:15:28+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01T09:15:28.563686Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-11-01T09:15:28.589541Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-m6xlq.lower-test
2023-11-01T09:15:28.589666Z 0 [Note] mysqld (mysqld 5.7.37-log) starting as process 1 ...
2023-11-01T09:15:28.604726Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-m6xlq.lower-test
2023-11-01T09:15:28.615831Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-m6xlq.lower-test
mysqld: File '/var/lib/mysql/mysql-bin.index' not found (Errcode: 30 - Read-only file system)
2023-11-01T09:15:28.628082Z 0 [ERROR] Aborting

2023-11-01T09:15:28.628112Z 0 [Note] Binlog end
2023-11-01T09:15:28.631272Z 0 [Note] mysqld: Shutdown complete

挂载节点经常性的断电重启、Errcode: 30 - Read-only file system 重启挂载节点重启处理了

-bash-4.2# kubectl  get po | grep mysql
dbapi-mysql-bcjmp                                           1/1     Running                  0                 90s
...

延展

尝试执行 update 命令，报错如下 Operation not allowed when innodb_forced_recovery > 0， 去除这个innodb_force_recovery=1配置