PreparedStatement重新认知(1)——它真的预编译吗

起因

最近在阅读数据库连接池相关的书籍，书中有一小节提到了Statement与PreparedStatement的区别，并指出使用PreparedStatement会对SQL进行预编译，并将预编译的SQL存储下来，下次直接使用，提高效率。与之相对的是Statement，它需要频繁进行编译，从这个角度而言，PreparedStatement会比Statement更快，但事实真的是这样的吗？书中并没有对此二者进行深入阐述，只是以喂食般给出了上述结论。笔者一直对PreparedStatement的工作原理比较模糊，它是怎么进行的预编译，印象中有说是在客户端进行的编译，也有说在数据库端进行的编译。因此，为了搞清楚PreparedStatement的工作原理，查阅相关资料，并将学习结果记录下来并做分享

本文将围绕下述两个问题进行展开：

PreparedStatement是真的更快吗？
预编译真的发生了吗？
如果真的发生了预编译，是在客户端还是在数据库端发生？

案例探索

为了下文更好地表述，事先约定一下环境与术语

数据库指的是关系型数据库，在本文中特指MySQL
客户端是相对于数据库而言的说法，因为对于数据库端而言，任何连接它的应用程序都可以称为数据库客户端，包括Java程序，在本文中指的是Java代码，或者是JDBC-MySQL驱动程序(mysql-connector-java)
MySQL数据库版本: 5.6.39 社区版
mysql-connector-java 版本: 5.1.47
JDK版本: Oracle 1.8.0_192

如果要做性能测试，出于JVM的工作机制，有经验的选手一般会考虑到"代码预热"，一般做法是在main方法中for循环(如10万次)调用一个方法使一段代码"热"起来，达到JIT编译门槛，使这段"热"代码编译成为机器码，以达到最高的执行效率。但这种预热法对于测量的结果没有太大指导意义，正确的姿势应该是采用JMH(Java Microbenchmark Harness)。
具体可参考R大(RednaxelaFX)在知乎的回答，传送门

因此，本文的案例也会采用JMH进行测试

测试Statement的查询性能，JMH测试代码如下:

@State(Scope.Thread)
public class StatementTest {
    Connection connection;
    Statement statement;

    @Setup
    public void init() throws Exception {
        String url = "jdbc:mysql:///test";
        connection = DriverManager.getConnection(url, "root", "xxx");
        statement = connection.createStatement();
    }

    @TearDown
    public void close() throws Exception {
        if (statement != null) {
            statement.close();
        }
        if (connection != null) {
            connection.close();
        }
    }

    @Benchmark
    // 预热2次，每次10秒
    @Warmup(iterations = 2, time = 10)
    // 预热完成后测量3次，每次10秒
    @Measurement(iterations = 3, time = 10)
    public ResultSet m() throws SQLException {
        return statement.executeQuery("select * from foo where id = 1");
    }

    public static void main(String[] args) throws Exception {
        Options opt = new OptionsBuilder()
                .forks(2) // 测两轮
                .build();

        new Runner(opt).run();
    }
}

上面这段JMH代码大概的含义是: 一共测两轮，每轮一开始会预热2次(每次10秒)，接着开始3次正式开始测量(每次10秒)，测量的是statement.executeQuery("select * from foo where id = 1");的性能
。测量结果如下:

# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration   1: 6383.500 ops/s
# Warmup Iteration   2: 9198.667 ops/s
Iteration   1: 9853.260 ops/s
Iteration   2: 9484.938 ops/s
Iteration   3: 7762.880 ops/s

# Run progress: 50.00% complete, ETA 00:00:52
# Fork: 2 of 2
# Warmup Iteration   1: 8876.019 ops/s
# Warmup Iteration   2: 8821.313 ops/s
Iteration   1: 9678.910 ops/s
Iteration   2: 9688.882 ops/s
Iteration   3: 9805.789 ops/s


Result "com.example.demo.StatementTest.m":
  9379.110 ±(99.9%) 2248.990 ops/s [Average]
  (min, avg, max) = (7762.880, 9379.110, 9853.260), stdev = 802.012
  CI (99.9%): [7130.120, 11628.100] (assumes normal distribution)


# Run complete. Total time: 00:01:43

Benchmark         Mode  Cnt     Score      Error  Units
StatementTest.m  thrpt    6  9379.110 ± 2248.990  ops/s

Process finished with exit code 0

测试一般情况下，PrepareStatement的查询性能(大多数日常开发的姿势)

@State(Scope.Thread)
public class PrepareStatementTest {
    Connection connection;
    PreparedStatement preparedStatement;

    @Setup
    public void init() throws Exception {
        // 只变更如下链接
        String url = "jdbc:mysql:///test";
        connection = DriverManager.getConnection(url, "root", "xxx");
        // 由Statement变成PreparedStatement
        preparedStatement = connection.prepareStatement("select * from foo where id = ?");
    }

    @TearDown
    public void close() throws Exception {
        if (preparedStatement != null) {
            preparedStatement.close();
        }
        if (connection != null) {
            connection.close();
        }
    }

    @Benchmark
    @Warmup(iterations = 2, time = 10)
    @Measurement(iterations = 3, time = 10)
    public ResultSet m() throws SQLException {
        preparedStatement.setLong(1, 1);
        return preparedStatement.executeQuery();
    }

    public static void main(String[] args) throws Exception {
        Options opt = new OptionsBuilder()
                .forks(2)
                .build();

        new Runner(opt).run();
    }
}

这段JMH测试代码中，改变的地方只是从Statement变成了PreparedStatement，别的不变，JMH测量结果如下:

# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration   1: 8779.791 ops/s
# Warmup Iteration   2: 9724.843 ops/s
Iteration   1: 9320.048 ops/s
Iteration   2: 8324.464 ops/s
Iteration   3: 10007.290 ops/s

# Run progress: 50.00% complete, ETA 00:00:51
# Fork: 2 of 2
# Warmup Iteration   1: 9068.252 ops/s
# Warmup Iteration   2: 9750.551 ops/s
Iteration   1: 9485.021 ops/s
Iteration   2: 9450.123 ops/s
Iteration   3: 9780.915 ops/s


Result "com.example.demo.PrepareStatementTest.m":
  9394.643 ±(99.9%) 1628.668 ops/s [Average]
  (min, avg, max) = (8324.464, 9394.643, 10007.290), stdev = 580.799
  CI (99.9%): [7765.975, 11023.312] (assumes normal distribution)


# Run complete. Total time: 00:01:42

Benchmark                Mode  Cnt     Score      Error  Units
PrepareStatementTest.m  thrpt    6  9394.643 ± 1628.668  ops/s

Process finished with exit code 0

在PreparedStatement的JMH测试代码中，URL参数添加useServerPrepStmts=true，即jdbc:mysql:///test?useServerPrepStmts=true，其它不变，JMH测试结果如下:

# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration   1: 9659.413 ops/s
# Warmup Iteration   2: 9296.215 ops/s
Iteration   1: 8734.479 ops/s
Iteration   2: 9609.639 ops/s
Iteration   3: 9683.444 ops/s

# Run progress: 50.00% complete, ETA 00:00:51
# Fork: 2 of 2
# Warmup Iteration   1: 10151.456 ops/s
# Warmup Iteration   2: 10179.692 ops/s
Iteration   1: 9989.025 ops/s
Iteration   2: 10158.410 ops/s
Iteration   3: 10662.958 ops/s


Result "com.example.demo.PrepareStatementTest.m":
  9806.326 ±(99.9%) 1814.637 ops/s [Average]
  (min, avg, max) = (8734.479, 9806.326, 10662.958), stdev = 647.117
  CI (99.9%): [7991.689, 11620.963] (assumes normal distribution)


# Run complete. Total time: 00:01:43

Benchmark                Mode  Cnt     Score      Error  Units
PrepareStatementTest.m  thrpt    6  9806.326 ± 1814.637  ops/s

Process finished with exit code 0

在PreparedStatement的JMH测试代码中，URL参数添加cachePrepStmts=true，即jdbc:mysql:///test?cachePrepStmts=true，其它不变，JMH测试结果如下:

# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration   1: 9725.255 ops/s
# Warmup Iteration   2: 10058.296 ops/s
Iteration   1: 10081.576 ops/s
Iteration   2: 10064.490 ops/s
Iteration   3: 10185.449 ops/s

# Run progress: 50.00% complete, ETA 00:00:52
# Fork: 2 of 2
# Warmup Iteration   1: 9757.321 ops/s
# Warmup Iteration   2: 10143.745 ops/s
Iteration   1: 10142.830 ops/s
Iteration   2: 10127.477 ops/s
Iteration   3: 10113.163 ops/s


Result "com.example.demo.PrepareStatementTest.m":
  10119.164 ±(99.9%) 121.980 ops/s [Average]
  (min, avg, max) = (10064.490, 10119.164, 10185.449), stdev = 43.499
  CI (99.9%): [9997.184, 10241.144] (assumes normal distribution)


# Run complete. Total time: 00:01:43

Benchmark                Mode  Cnt      Score     Error  Units
PrepareStatementTest.m  thrpt    6  10119.164 ± 121.980  ops/s

Process finished with exit code 0

在PreparedStatement的JMH测试代码中，URL参数添加useServerPrepStmts=true&cachePrepStmts=true，即jdbc:mysql:///test?useServerPrepStmts=true&cachePrepStmts=true，其它不变，JMH测试结果如下:

# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration   1: 10303.083 ops/s
# Warmup Iteration   2: 10785.386 ops/s
Iteration   1: 10780.442 ops/s
Iteration   2: 10755.745 ops/s
Iteration   3: 10794.132 ops/s

# Run progress: 50.00% complete, ETA 00:00:51
# Fork: 2 of 2
# Warmup Iteration   1: 10416.622 ops/s
# Warmup Iteration   2: 10658.629 ops/s
Iteration   1: 10657.114 ops/s
Iteration   2: 10706.986 ops/s
Iteration   3: 10646.870 ops/s


Result "com.example.demo.PrepareStatementTest.m":
  10723.548 ±(99.9%) 176.565 ops/s [Average]
  (min, avg, max) = (10646.870, 10723.548, 10794.132), stdev = 62.965
  CI (99.9%): [10546.983, 10900.114] (assumes normal distribution)


# Run complete. Total time: 00:01:43

Benchmark                Mode  Cnt      Score     Error  Units
PrepareStatementTest.m  thrpt    6  10723.548 ± 176.565  ops/s

Process finished with exit code 0

把五次测试结果归纳一下：

Statement: 9379.110
PreparedStatement: 9394.643
PreparedStatement useServerPrepStmts: 9806.326
PreparedStatement cachePrepStmts: 10119.164
PreparedStatement useServerPrepStmts cachePrepStmts: 10723.548

可以看到，本案例中使用Statement与PreparedStatement(不添加useServerPrepStmts、cachePrepStmts)，吞吐量并无太大差别

PreparedStatement是真的更快吗？
答：不一定，至少在本案例中，二者相差无几，除去误差，基本认为二者是一致的。在大多数项目中，与关系型数据库打交道都会直接或间接使用到PreparedStatement，但很少会在连接参数中添加useServerPrepStmts与cachePrepStmts，此时，在效率上，与Statement并无二致

useServerPrepStmts、cachePrepStmts这两个参数非常重要，是解答后两个问题的关键。

先说useServerPrepStmts，这个参数是让数据库端支持prepared statements，即预编译。也就是说，如果连接参数中没有添加这个属性，数据库端压根就不会进行预编译。摘抄MySQL官网两段话:

Changes in MySQL Connector/J 3.1.0: Added useServerPrepStmts property (default false). The driver will use server-side prepared statements when the server version supports them (4.1 and newer) when this property is set to true. It is currently set to false by default until all bind/fetch functionality has been implemented. Currently only DML prepared statements are implemented for 4.1 server-side prepared statements.

Upgrading from MySQL Connector/J 3.0 to 3.1: Server-side Prepared Statements: Connector/J 3.1 will automatically detect and use server-side prepared statements when they are available (MySQL server version 4.1.0 and newer).

这里有两个关键信息：

MySQL 4.1+才支持数据库端的预编译，之前的版本并不支持
客户端(mysql-connector-java)版本必须>= 3.1.0

如果客户端版本>= 3.1.0，且数据库版本>=4.1.0，那么客户端与数据库端连接时会自动开启数据库端的预编译

但是，客户端版本自5.0.5之后，客户端与数据库端连接时不再自动开启数据库端的预编译

Changes in MySQL Connector/J 5.0.5: Important change: Due to a number of issues with the use of server-side prepared statements, Connector/J 5.0.5 has disabled their use by default. The disabling of server-side prepared statements does not affect the operation of the connector in any way.

To enable server-side prepared statements, add the following configuration property to your connector string: useServerPrepStmts=true

敲重点：因为数据库端预编译太多BUG，mysql-connector-java>=5.0.5不再自动开启数据库端预编译，如果非要开启，可以在连接参数中添加useServerPrepStmts=true属性。

我们使用的客户端版本为5.1.47，在JMH中添加了useServerPrepStmts=true开启数据库端的预编译，较未开启之前性能提升4.4%(9394.643->9806.326)。

预编译真的发生了吗？答: 在未添加useServerPrepStmts=true属性之前，数据库端的预编译并没有发生，添加之后，开启了数据库端的预编译能力

如果真的发生了预编译，是在客户端还是在数据库端发生？答: 未添加useServerPrepStmts=true属性之前，是在客户端进行了预编译，数据库端没有；而添加属性之后，数据库端也开启了预编译

源码分析

接下来看一下获取连接时(DriverManager.getConnection(url, user, password);)对于useServerPrepStmts属性的处理

// com.mysql.jdbc.ConnectionImpl#initializePropsFromServer
private void initializePropsFromServer() throws SQLException {
    // ...(省略)

    //
    // Users can turn off detection of server-side prepared statements
    // getUseServerPreparedStmts() 用于检测客户端版本是否>=3.1.0，以及连接是否配置useServerPrepStmts=true
    // versionMeetsMinimum(4, 1, 0)要求数据库端版本 >= 4.1.0
    if (getUseServerPreparedStmts() && versionMeetsMinimum(4, 1, 0)) {
        // 此属性为true，才会开启数据库端预编译
        this.useServerPreparedStmts = true;

        if (versionMeetsMinimum(5, 0, 0) && !versionMeetsMinimum(5, 0, 3)) {
        // 5.0.0 <= MySQL数据库端版本 < 5.0.3，也不支持(或许是有BUG)
            this.useServerPreparedStmts = false; // 4.1.2+ style prepared
            // statements
            // don't work on these versions
        }
    }

    // ...(省略)
}

public boolean getUseServerPreparedStmts() {
    return this.detectServerPreparedStmts.getValueAsBoolean();
}

 // Think really long and hard about changing the default for this many, many applications have come to be acustomed to the latency profile of preparing stuff client-side, rather than prepare (round-trip), execute (round-trip), close (round-trip).
 // 如果没有设置useServerPrepStmts属性，默认值为false
 // 自3.1.0版本开始
private BooleanConnectionProperty detectServerPreparedStmts = new BooleanConnectionProperty("useServerPrepStmts", false,
        Messages.getString("ConnectionProperties.useServerPrepStmts"), "3.1.0", MISC_CATEGORY, Integer.MIN_VALUE);

从源码可以看出，开启数据库端预编译需要同时满足以下条件

MySQL客户端版本 >=3.1.0
MySQL服务端版本 >=4.1，且版本号不能是[5.0.0, 5.0.3)
设置连接属性useServerPrepStmts = true

基本上与官网介绍是吻合，至于版本号不能是[5.0.0, 5.0.3)，这点并没有在官网看到

接着通过Connection创建PreparedStatement

PreparedStatement preparedStatement = connection.prepareStatement("select * from foo where id = ?");

// com.mysql.jdbc.ConnectionImpl#prepareStatement(java.lang.String, int, int)

public java.sql.PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency) throws SQLException {
    synchronized (getConnectionMutex()) {
        checkClosed();

        //
        // FIXME: Create warnings if can't create results of the given type or concurrency
        //
        PreparedStatement pStmt = null;

        boolean canServerPrepare = true;

        String nativeSql = getProcessEscapeCodesForPrepStmts() ? nativeSQL(sql) : sql;

        // useServerPreparedStmts赋值过程上面已经分析
        // getEmulateUnsupportedPstmts(): 如果驱动检测到服务端不支持预编译，是否要启用客户端的预编译来代替，默认是true
        if (this.useServerPreparedStmts && getEmulateUnsupportedPstmts()) {
            // 根据SQL去判断服务端是否支持预编译，因为有的SQL例如调用存储过程的命令`call`，`create table`是不支持预编译的，因此需要将canServerPrepare属性置为false
            canServerPrepare = canHandleAsServerPreparedStatement(nativeSql);
        }

        if (this.useServerPreparedStmts && canServerPrepare) {
            // 进入此分支代表启用数据库端的预编译
            // cachePrepStmts参数值如果为true则代表需要将PrepStmts缓存起来，默认是false
            if (this.getCachePreparedStatements()) {
                // cachePrepStmts = true
                synchronized (this.serverSideStatementCache) {
                    pStmt = this.serverSideStatementCache.remove(new CompoundCacheKey(this.database, sql));

                    if (pStmt != null) {
                        ((com.mysql.jdbc.ServerPreparedStatement) pStmt).setClosed(false);
                        pStmt.clearParameters();
                    }

                    if (pStmt == null) {
                        try {
                            pStmt = ServerPreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database, resultSetType,
                                    resultSetConcurrency);
                            if (sql.length() < getPreparedStatementCacheSqlLimit()) {
                                ((com.mysql.jdbc.ServerPreparedStatement) pStmt).isCached = true;
                            }

                            pStmt.setResultSetType(resultSetType);
                            pStmt.setResultSetConcurrency(resultSetConcurrency);
                        } catch (SQLException sqlEx) {
                            // Punt, if necessary
                            if (getEmulateUnsupportedPstmts()) {
                                pStmt = (PreparedStatement) clientPrepareStatement(nativeSql, resultSetType, resultSetConcurrency, false);

                                if (sql.length() < getPreparedStatementCacheSqlLimit()) {
                                    this.serverSideStatementCheckCache.put(sql, Boolean.FALSE);
                                }
                            } else {
                                throw sqlEx;
                            }
                        }
                    }
                }
            } else {
                // cachePrepStmts = false
                try {
                    pStmt = ServerPreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database, resultSetType, resultSetConcurrency);

                    pStmt.setResultSetType(resultSetType);
                    pStmt.setResultSetConcurrency(resultSetConcurrency);
                } catch (SQLException sqlEx) {
                    // Punt, if necessary
                    if (getEmulateUnsupportedPstmts()) {
                        pStmt = (PreparedStatement) clientPrepareStatement(nativeSql, resultSetType, resultSetConcurrency, false);
                    } else {
                        throw sqlEx;
                    }
                }
            }
        } else {
            // 使用客户端的预编译
            // 客户端的预编译也可以开启缓存功能(cachePrepStmts)
            pStmt = (PreparedStatement) clientPrepareStatement(nativeSql, resultSetType, resultSetConcurrency, false);
        }

        return pStmt;
    }
}

客户端PreparedStatement的实现类: com.mysql.jdbc.JDBC42PreparedStatement

服务端PreparedStatement的实现类: com.mysql.jdbc.JDBC42ServerPreparedStatement

cachePrepStmts=true是开启PreparedStatement的缓存功能，该缓存由MySQL驱动实现，同时支持客户端的PreparedStatement与服务端PreparedStatement，但对两端的缓存实现方式不一样，一是缓存的时机不同，二是缓存的value不同

客户端的缓存，是在创建客户端PrepareStatement的时候进行缓存的，缓存以nativeSql为key，ParseInfo为value

public java.sql.PreparedStatement clientPrepareStatement(String sql, int resultSetType, int resultSetConcurrency, boolean processEscapeCodesIfNeeded)
        throws SQLException {
    checkClosed();

    String nativeSql = processEscapeCodesIfNeeded && getProcessEscapeCodesForPrepStmts() ? nativeSQL(sql) : sql;

    PreparedStatement pStmt = null;

    if (getCachePreparedStatements()) {
        // 开启缓存
        PreparedStatement.ParseInfo pStmtInfo = this.cachedPreparedStatementParams.get(nativeSql);

        if (pStmtInfo == null) {
            pStmt = com.mysql.jdbc.PreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database);
            // nativeSql为key，ParseInfo为value
            this.cachedPreparedStatementParams.put(nativeSql, pStmt.getParseInfo());
        } else {
            pStmt = com.mysql.jdbc.PreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database, pStmtInfo);
        }
    } else {
        pStmt = com.mysql.jdbc.PreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database);
    }

    pStmt.setResultSetType(resultSetType);
    pStmt.setResultSetConcurrency(resultSetConcurrency);

    return pStmt;
}

服务端的缓存，是在com.mysql.jdbc.ServerPreparedStatement#close时进行缓存的，以CompoundCacheKey(封装了catalog与originalSql)为key，pstmt为value

// com.mysql.jdbc.ServerPreparedStatement#close

public void close() throws SQLException {
    MySQLConnection locallyScopedConn = this.connection;

    if (locallyScopedConn == null) {
        return; // already closed
    }

    synchronized (locallyScopedConn.getConnectionMutex()) {
        if (this.isCached && isPoolable() && !this.isClosed) {
            clearParameters();
            this.isClosed = true;
            // 缓存
            this.connection.recachePreparedStatement(this);
            return;
        }

        this.isClosed = false;
        realClose(true, true);
    }
}

public void recachePreparedStatement(ServerPreparedStatement pstmt) throws SQLException {
    synchronized (getConnectionMutex()) {
        if (getCachePreparedStatements() && pstmt.isPoolable()) {
            synchronized (this.serverSideStatementCache) {
                // 以CompoundCacheKey为key，pstmt为value
                Object oldServerPrepStmt = this.serverSideStatementCache.put(new CompoundCacheKey(pstmt.currentCatalog, pstmt.originalSql), pstmt);
                if (oldServerPrepStmt != null && oldServerPrepStmt != pstmt) {
                    ((ServerPreparedStatement) oldServerPrepStmt).isCached = false;
                    ((ServerPreparedStatement) oldServerPrepStmt).setClosed(false);
                    ((ServerPreparedStatement) oldServerPrepStmt).realClose(true, true);
                }
            }
        }
    }
}

总结

本文对PreparedStatement在MySQL下的工作机制进行了探究，并使用JMH编写性能测试代码，得出了它并不比Statement更快的结论，并且对"预编译"的概念进行了阐述，概念包含两个方面：客户端的预编译与数据库端的预编译

由于大多数据情况下数据库连接参数中并不会配置useServerPrepStmts = true，此时应用程序工作在客户端的预编译模式下，性能与Statement相比未有明显提高，尽管开启服务端预编译能提升吞吐量，但该方式存在过多的BUG，在生产环境中仍然不建议开启，避免采坑

从MySQL官网以及互联网上几乎找不到关于cachePrepStmts的BUG，暂且可以粗略认为该功能BUG较少，或者影响可以忽略不计，因此可以尝试配置cachePrepStmts=true，即开启PreparedStatement的缓存，达到提升吞吐量的目的

在不配置useServerPrepStmts 、cachePrepStmts的情况下，PreparedStatement并不比Statement更快，是否意味着可以使用Statement代替PreparedStatement？实则不然，因为PreparedStatement还有一个非常重要的特性是Statement所不具备的: 防止SQL注入

PreparedStatement是如何防止SQL注入呢？且听下回分解

导读: PreparedStatement重新认知(2)——防止SQL注入

PreparedStatement重新认知(1)——它真的预编译吗

起因

案例探索

源码分析

总结

你可能感兴趣的:(PreparedStatement重新认知(1)——它真的预编译吗)