因参数innodb_undo_directory的配置问题，导致xtrabackup备份失败

软件环境描述

centos7.3
mysql5.7.29 二进制安装包
xtrabckup2.4.18 二进制安装包

在进行数据库物理备份的时候，遇到如下报错

2020-03-03 20:24:49 0x7f324f9cb740  InnoDB: Assertion failure in thread 139854060762944 in file srv0start.cc line 928
InnoDB: Failing assertion: dir
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:24:49 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2c)[0xd972dc]
innobackupex(handle_fatal_signal+0x262)[0xa4a4c2]
/lib64/libpthread.so.0(+0xf5f0)[0x7f324f5be5f0]
/lib64/libc.so.6(gsignal+0x37)[0x7f324db17337]
/lib64/libc.so.6(abort+0x148)[0x7f324db18a28]
innobackupex[0x7180cb]
innobackupex(_Z25srv_undo_tablespaces_initbbmPm+0x1b85)[0xb3cd95]
innobackupex[0x7469de]
innobackupex(_Z22xtrabackup_backup_funcv+0xf12)[0x748cd2]
innobackupex(main+0xe15)[0x725495]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f324db03505]
innobackupex[0x73d3e4]

Please report a bug at https://jira.percona.com/projects/PXB

排查过程

1.未检查配置文件的情况下，怀疑是xtrabckup和mysql的版本不兼容的问题，
更换xtrabackup的版本和mysql的小版本，都不能备份成功（依然报错）

2.使用rpm安装mysql社区版本，使用xtrabackup2.4备份，成功，此时才怀疑是配置文件的问题，
使用二进制安装程序下mysqld初始化新端口，使用最简单的my.cnf配置（修改datadir）,备份成功

3.最终确定是配置文件的问题，于是排查配置文件，发现innodb_undo_directory参数配置了，
于是怀疑是这个参数的问题（安装脚本里并没有创建这个目录）

4.经过几次修改innodb_undo_directory参数的位置并进行备份测试，确定是由于该参数的配置目录实际上并不存在而导致以上报错

实验过程简单记录

1.innodb_undo_directory的位置并不存在时
innodb_undo_directory = /data/mysql8899/nodir/

备份--失败
rm -rf /data/backup_xtrabackup/*
innobackupex --defaults-file=/data/mysql8899/mycnf/my_8899.cnf --user=root --password=Root#2018 --host=127.0.0.1 --port=8899 --socket=/data/mysql8899/data/mysql8899.sock /data/backup_xtrabackup/

InnoDB: Number of pools: 1
200304 06:53:19 >> log scanned up to (2624040)
2020-03-04 06:53:19 0x7f3b42dd0740  InnoDB: Assertion failure in thread 139892501579584 in file srv0start.cc line 928
InnoDB: Failing assertion: dir



2.innodb_undo_directory的位置存在时
innodb_undo_directory = /data/mysql8899/undolog/

手动创建这个目录 
mkdir -p /data/mysql8899/undolog/ (实际并未赋予mysql属组）

备份--成功
rm -rf /data/backup_xtrabackup/*
innobackupex --defaults-file=/data/mysql8899/mycnf/my_8899.cnf --user=root --password=Root#2018 --host=127.0.0.1 --port=8899 --socket=/data/mysql8899/data/mysql8899.sock /data/backup_xtrabackup/

问题总结梳理

1.遇到类似的问题，首先要相信软件本身，是可以解决常规问题的。
不能盲目猜测（如果一开始就相信，xtrbackup软件 和 二进制安装程序是没有问题的，那这个问题的排查过程就会大大的缩减）。
2.对于my.cnf的配置项，要有敬畏之心。
不能拿来主义，每个参数都要熟知于心。
遇到新的参数，一定要做足测试，才可以使用（至少备份、恢复、不同集群架构、自动化运维、数据归档处理、mysql周边辅助工具以及在percona分支、mariadb下的这些场景下，都要进行）

参数说明
innodb_undo_directory
InnoDB undo日志所在表空间的物理路径。和innodb_undo_logs、innodb_undo_tablespaces参数配合，来设置undo日志的路径，默认路径是数据文件路径。

Undo tablespaces contain undo logs, which are collections of undo log records that contain information about how to undo the latest change by a transaction to a clustered index record. Undo logs exist within undo log segments, which are contained within rollback segments. The innodb_rollback_segments variable defines the number of rollback segments allocated to each undo tablespace.
Undo logs can be stored in one or more undo tablespaces instead of the system tablespace. This layout differs from the default configuration in which undo logs reside in the system tablespace. The I/O patterns for undo logs make undo tablespaces good candidates for SSD storage, while keeping the system tablespace on hard disk storage.
The number of undo tablespaces used by InnoDB is controlled by the innodb_undo_tablespaces configuration option. This option can only be configured when initializing the MySQL instance. It cannot be changed afterward.

另外，需要注意这个参数在后续的版本将会移除

参考网址
https://dev.mysql.com/doc/refman/5.7/en/innodb-undo-tablespaces.html