mtr MySQL Test Framework Components

MySQL Test Framework Components

Table of Contents

• Test Framework System Requirements
• How to Report Bugs from Tests in the MySQL Test Suite

The MySQL test framework consists of programs that run tests, and directories and files used by those programs.
MySQL测试框架由运行测试的程序以及这些程序使用的目录和文件组成。

Test Framework Programs

The MySQL test framework uses several programs:
MySQL测试框架使用了以下几个程序:

• The mysql-test-run.pl Perl script is the main application used to run the test suite. It invokes mysqltest to run individual test cases.
  mysql-test-run.pl Perl脚本是用于运行测试套件的主要应用程序。它调用mysqltest来运行单独的测试用例。

• mysqltest runs test cases.
  Mysqltest运行测试用例。

• The mysql_client_test program is used for testing aspects of the MySQL client API that cannot be tested using mysqltest and its test language.
  mysql_client_test程序用于测试MySQL客户端API的某些方面，这些方面不能使用mysqltest及其测试语言进行测试。

• The mysql-stress-test.pl Perl script performs stress-testing of the MySQL server.
  MySQL -stress-test.pl Perl脚本执行MySQL服务器的压力测试。

• A unit-testing facility is provided so that individual unit test programs can be created for storage engines and plugins.
  它提供了一个单元测试工具，这样就可以为存储引擎和插件创建单独的单元测试程序。

Test suite programs can be found in these locations:
测试套件程序可以在以下位置找到:

• For a source distribution, mysqltest is in the client directory. For a binary distribution, it is in the MySQL bin directory.
  对于源发行版，mysqltest在客户端目录中。对于二进制发行版，它在MySQL bin目录下。

• For a source distribution, mysql_client_test is in the tests directory. For a binary distribution, it is in the MySQL bin directory.
  对于源分发版，mysql_client_test位于tests目录中。对于二进制发行版，它在MySQL bin目录下。

• The other programs are located in the mysql-test directory. For a source distribution, mysql-test is found under the source tree root. For a binary distribution, the location of mysql-test depends on the layout used for the distribution format.
  其他程序位于mysql-test目录中。对于源代码发行版，mysql-test位于源代码树的根目录下。对于二进制发行版，mysql-test的位置取决于发行版格式所使用的布局。

Test Framework Directories and Files

The test suite is located in the mysql-test directory, which contains the following components:
测试套件位于mysql-test目录下，包含以下组件:

• The mysql-test-run.pl and mysql-stress-test.pl programs that are used for running tests.
  用于运行测试的mysql-test-run.pl和mysql-stress-test.pl程序。

• The t directory contains test case input files. A test case file might also have additional files associated with it.
  t目录包含测试用例输入文件。一个测试用例文件可能还有其他与之相关的文件。

  • A file name of the form test_name.test is a test case file for a test named test_name. For example, subquery.test is the test case file for the test named subquery.
    格式为test_name.test是一个名为test_name的测试的测试用例文件。例如,subquery.test是名为subquery的测试的测试用例文件。

  • A file name of the form test_name-master.opt provides options to associate with the named test case. mysql-test-run.pl restarts the server with the options given in the file if the options are different from those required for the currently running server.
    test_name-master.opt提供了与命名的测试用例相关联的选项。如果选项与当前运行的服务器所需的选项不同，那么Mysql-test-run.pl将使用文件中给出的选项重新启动服务器。

    Note that the -master.opt file is used for the “main” server of a test, even if no replication is involved.
    注意-master.opt文件用于测试的“主”服务器，即使不涉及复制。

  • A file name of the form test_name-slave.opt provides slave options.
    test_name-slave.opt提供从选项。

  • Server options which need to be passed during initialization of the server can be passed in both test_name-master.opt and test_name-slave.opt. Each bootstrap variable must be passed as the value of a --initialize option so that mysql-test-run.pl can recognize that the variable should be used during server initialization, remove the existing data directory, and initialize a new data directory with the options provided. mysql-test-run.pl will also start the server afterwards, with the bootstrap options, and other options specified.
    在服务器初始化期间需要传递的服务器选项可以在test_name-master.opt 和 test_name-slave.opt中。每个引导变量都必须作为——initialize选项的值传递，以便mysql-test-run.pl能够识别在服务器初始化期间应该使用这个变量，删除现有的数据目录，并使用提供的选项初始化一个新的数据目录。pl之后也会启动服务器，并指定引导选项和其他选项。

  • A file name of the form test_name-client.opt provides mysqltest client options.
    test_name-client.opt提供 client options。

  • A file name of the form test_name.cnf contains additional entries for the config file to be used for this test.
    test_name.cnf包含增加的配置内容用于测试。

    • During a test run, each server is started with a different defaults-group-suffix value and this suffix value can be used with a group name to specify server specific options. For example, section [mysqld.1] is used to specify options for mysqld server with suffix value ".1".
      在测试运行期间，每个服务器都使用不同的默认-group-suffix值启动，这个后缀值可以与组名一起使用，以指定特定于服务器的选项。例如，section [mysqld。1]用于指定mysqld服务器的选项，后缀值为“。1”。

    • The [mysqltest] section is used to specify options for mysqltest client.
      [mysqltest]的作用是为mysqltest客户端指定选项。

    • The [ENV] section is used to specify environment variables for a test case.
      [ENV]部分用于为测试用例指定环境变量。

    • To include the options mentioned in another .cnf file, !include path_to_cnf_file is used. Path to the included .cnf should either be an absolute path or a relative path from current location or from mysql-test directory.
      要包含另一个.cnf文件中提到的选项，需要使用!include path_to_cnf_file。包含的.cnf的路径应该是来自当前位置或mysql-test目录的绝对路径或相对路径。

    Here is a sample .cnf file.

    !include include/default_my.cnf
    
    [mysqld.1]
    Options for server mysqld.1
    
    [mysqld.2]
    Options for server mysqld.2
    
    [mysqltest]
    ps-protocol
    
    ...
    ...
    ...
    
    [ENV]
    SERVER_MYPORT_1= @mysqld.1.port
    SERVER_MYPORT_2= @mysqld.2.port
    

  • A file name of the form test_name.combinations provides section of options for each test run.
    一个test_name.combinations形式的文件名为每次测试运行提供了部分选项。

  • A file name of the form suite.opt provides server options to all the tests within the suite it belongs to.
    suite.opt 提供服务端options给所有测试。

    If a test run is started with more than one server, the options specified in a suite.opt are used by all of them. It is also possible to pass the server options needed during the server initialization in the suite.opt file.
    如果使用多个服务器启动测试运行，则在套件中指定选项。它们都使用Opt。还可以在套件中传递服务器初始化期间所需的suite.opt文件。

    The options mentioned in the suite.opt file are overridden by those found in a test_name-master.opt file or in a test_name-slave.opt file.
    suite.opt文件被test_name-master.opt或test_name-slave.opt的内容覆盖。

  • A file name of the form test_name-master.sh is a shell script that will be executed before the server is started for the named test case. There may also be a test_name-slave.sh which is executed before the slave server is started.
    一个名为test_name-master.sh的文件名是一个shell脚本，它将在为指定的测试用例启动服务器之前执行。可能还有一个test_name-slave.sh，在从服务器启动之前执行。

    These files will not work on Windows and may therefore be replaced with a more portable mechanism in the future.
    这些文件将不能在Windows上工作，因此可能在未来被一种更可移植的机制取代。

  • The disabled.def file contains information about deferred/disabled tests. When a test is failing because of a bug in the server and you want it to be ignored by mysql-test-run.pl, list the test in this file.
    disabled.def 包含有关延迟/禁用测试的信息。当一个测试由于服务器中的错误而失败，并且您希望mysql-test-run.pl忽略它时，请在这个文件中列出测试。

    The format of a line in the disabled.def file looks like this, where fields are separated by one or more spaces.
    def文件中的行格式如下所示，其中字段由一个或多个空格分隔。

    suite_name.test_name [@platform|@!platform] : # []
    

    Example:

    main.1st @linux : BUG#18980 Test fails randomly
    

    suite_name is the suite name. test_name is the test case name.
    Suite_name是套件名。test_name是测试用例的名称。

    An additional element after the test name and before the colon you may add a @platform to have the test disabled only on a specific platform. Basic OS names as reported by ^O在Perl，或'windows'是支持的。另外,@ !平台将在除指定平台之外的所有平台上禁用测试。

    BUG#nnnnn or WL#nnnn indicates the bug or the WL related to the test that causes it to fail (and thus requires it to be disabled).
    BUG#nnnnn或WL#nnnn表示导致测试失败(因此需要禁用它)的BUG或WL。

    The comment text following the BUG#nnnnn or the WL#nnnn is not part of the mandatory syntax for this file, mysql-test-run.pl will not actually care what is written here. Length of a comment text must not be more than 80 characters.
    BUG#nnnnn或WL#nnnn后面的注释文本不是这个文件的强制语法的一部分，mysql-test-run.pl实际上并不关心这里写了什么。注释文本的长度不能超过80个字符。

    A comment line can be written in the file by beginning the line with a “#” character.
    注释行可以通过以“#”字符开头写入文件。

• The r directory contains test case result files:
  r目录包含测试用例结果文件:

  • A file name of the form test_name.result is the expected result for the named test case. A file r/test_name.result is the output that corresponds to the input in the test case file t/test_name.test.
    test_name.result 是指定测试用例的预期结果。一个r/test_name.result 是对应于测试用例文件t/test_name.test中的输入的输出。

  • A file name of the form test_name.reject contains output for the named test case if the test fails due to output mismatch (but not it it fails for other reasons).
    格式为test_name.Reject包含指定测试用例的输出，如果测试由于输出不匹配而失败(但它不是因为其他原因而失败)。

  For a test case that succeeds, the .result file represents both the expected and actual result. For a test case that fails, the .result file represents the expected result, and the .reject file represents the actual result.
  对于成功的测试用例，.result文件代表了预期的和实际的结果。对于失败的测试用例，.result文件表示预期结果，.reject文件表示实际结果。

  If a .reject file is created because a test fails, mysql-test-run.pl removes the file later the next time the test succeeds.
  如果因为测试失败而创建了.reject文件，那么mysql-test-run.pl将在下次测试成功时删除该文件。

  When --check-testcases option is enabled, a test case not having its corresponding .result file is marked as failed by MTR. See mysql-test-run.pl.
  当——check testcases选项被启用时，没有相应的.result文件的测试用例会被MTR标记为失败。看到mysql-test-run.pl。

• The include directory contains files that are included by test case files using the source command. These include files encapsulate operations of varying complexity into a single file so that you can perform the operations in a single step. See Using Include Files to Simplify Test Cases.
  include目录包含使用source命令的测试用例文件所包含的文件。这些文件将不同复杂性的操作封装到单个文件中，以便您可以在单个步骤中执行这些操作。请参阅使用包含文件来简化测试用例。

• The lib directory contains library files used by mysql-test-run.pl, and database initialization SQL code.
  lib目录包含mysql-test-run.pl使用的库文件和数据库初始化SQL代码。

• The std_data directory contains data files used by some of the tests.
  std_data目录包含一些测试使用的数据文件。

• The var directory is used during test runs for various kinds of files: log files, temporary files, trace files, Unix socket files for the servers started during the tests, and so forth. This directory cannot be shared by simultaneous test runs.
  var目录在测试运行期间用于各种文件:日志文件、临时文件、跟踪文件、用于测试期间启动的服务器的Unix套接字文件，等等。此目录不能由同时测试运行共享。

• The suite directory contains a set of subdirectories containing named test suites. Each subdirectory represents a test suite with the same name.
  套件目录包含一组子目录，其中包含已命名的测试套件。每个子目录表示一个同名的测试套件。

  A test suite directory contains the following components:
  测试套件目录包含以下组件:

  • The t directory contains test cases.
    t目录中包含测试用例。

  • The r directory contains test case result files.
    r目录中包含测试用例结果文件。

  • The include directory contains files that are included by the test cases belonging to that suite.
    include目录包含属于该套件的测试用例所包含的文件。

  • A file name of the form combinations provides section of options for each test run. See Controlling the Binary Log Format Used for an Entire Test Run.
    表单组合的文件名为每次测试运行提供了部分选项。请参见控制整个测试运行使用的二进制日志格式。

  • A file name of the form my.cnf contains additional entries for the config file used by all the tests within the suite it belongs to.
    cnf格式的文件名包含配置文件的附加条目，该配置文件由其所属套件中的所有测试使用。

    A suite specific my.cnf file is overridden by a test_name.cnf file.
    特定于套件的my.cnf文件被test_name.cnf文件覆盖。

• The collections directory contains collections of test runs that are run during integration and release testing of MySQL. They are not directly useful outside this context, but need to be part of the source repository and are included for reference.
  collections目录包含MySQL集成和发布测试期间运行的测试运行的集合。它们在此上下文之外没有直接的用处，但需要成为源存储库的一部分，并被包含以供参考。

Unit test-related files are located in the unittest directory. Additional files specific to storage engines and plugins may be present under the subdirectories of the storage or plugin directories.
与单元测试相关的文件位于unittest目录中。特定于存储引擎和插件的附加文件可能存在于存储或插件目录的子目录下。

Test Execution and Evaluation

There are a number of targets in the top-level Makefile that can be used to run sets of tests. make test only runs unit tests. Other targets run subsets of the tests, or run tests with specific options for the test programs. Have a look at the Makefile to see what targets are available.
顶级Makefile中有许多目标，可用于运行测试集。Make测试只运行单元测试。其他目标运行测试的子集，或使用测试程序的特定选项运行测试。看看Makefile，看看有哪些目标可用。

A “test case” is a single file. The case might contain multiple individual test commands. If any individual command fails, the entire test case is considered to fail. Note that “fail” means “does not produce the expected result.” It does not necessarily mean “executes without error,” because some tests are written precisely to verify that an illegal statement does in fact produce an error. In such an instance, if the statement executes successfully without producing the expected error, that is considered failure of the test.
“测试用例”是单个文件。这个案例可能包含多个单独的测试命令。如果任何单独的命令失败，则认为整个测试用例失败。注意，“失败”意味着“不会产生预期的结果”。它并不一定意味着“执行时没有错误”，因为一些测试被精确地编写来验证非法语句确实产生了错误。在这种情况下，如果语句成功执行而没有产生预期的错误，则认为测试失败。

Test case output (the test result) consists of:
测试用例输出(测试结果)包括:

• Input SQL statements and their output. Each statement is written to the result followed by its output. Columns in output resulting from SQL statements are separated by tab characters.
  输入SQL语句及其输出。每条语句都被写入结果，然后输出。SQL语句产生的输出中的列由制表符分隔。

• The result from mysqltest commands such as echo and exec. The commands themselves are not echoed to the result, only their output.
  mysqltest命令的结果，例如echo和exec。命令本身不响应结果，只响应它们的输出。

The disable_query_log and enable_query_log commands control logging of input SQL statements. The disable_result_log and enable_result_log commands control logging of SQL statement results, and warning or error messages resulting from those statements.
disable_query_log和enable_query_log命令控制输入SQL语句的日志记录。disable_result_log和enable_result_log命令控制SQL语句结果的日志记录，以及由这些语句产生的警告或错误消息。

mysqltest reads a test case file from its standard input by default. The --test-file or -x option can be given to name a test case file explicitly.
Mysqltest默认从它的标准输入读取一个测试用例文件。可以指定——test-file或-x选项来显式地命名测试用例文件。

mysqltest writes test case output to the standard output by default. The --result-file or -R option can be used to indicate the location of the result file. That option, together with the --record option, determine how mysqltest treats the test actual and expected results for a test case:
Mysqltest默认将测试用例输出写入标准输出。可以使用——result-file或-R选项来指示结果文件的位置。这个选项，连同——record选项，决定了mysqltest如何处理测试用例的实际和预期结果:

• If the test produces no results, mysqltest exits with an error message to that effect, unless --result-file is given and this file is empty.
  如果测试没有产生结果，mysqltest将退出，并输出一条错误消息，除非给出了——result-file且该文件为空。

• Otherwise, if --result-file is not given, mysqltest sends test results to the standard output.
  否则，如果没有给出——result-file, mysqltest将把测试结果发送到标准输出。

• With --result-file but not --record, mysqltest reads the expected results from the given file and compares them with the actual results. If the results do not match, mysqltest writes a .reject file in the log directory and exits with an error. It will also print out a diff of the expected and actual result, provided it can find an appropriate diff tool to use.
  使用——result-file而不是——record, mysqltest从给定的文件中读取预期的结果，并将它们与实际结果进行比较。如果结果不匹配，mysqltest在日志目录中写入一个.reject文件，并以错误退出。它还将打印出预期和实际结果的差异，前提是它可以找到合适的差异工具来使用。

• With both --result-file and --record, mysqltest updates the given file by writing the actual test results to it. The file does not need to pre-exist.
  使用——result-file和——record, mysqltest通过向它写入实际的测试结果来更新给定的文件。该文件不需要预先存在。

mysqltest itself knows nothing of the t and r directories under the mysql-test directory. The use of files in those directories is a convention that is used by mysql-test-run.pl, which invokes mysqltest with the appropriate options for each test case to tell mysqltest where to read input and write output.
Mysqltest本身对mysql-test目录下的t和r目录一无所知。这些目录中文件的使用是mysql-test-run.pl使用的一种约定，它调用mysqltest，并为每个测试用例提供适当的选项，以告诉mysqltest从哪里读取输入和写入输出。