db_bili
db_bili

Mysql中copying to tmp table事件的相关处理

环境说明：

 版本
' Ver 8.42 Distrib 5.5.27, for Linux on x86_64
Table 为 innodb 引擎

1.原始sql的执行计划

(这条sql导致了mysql的性能问题，当它并发多条执行的时候影响其它正常sql的执行，大量sending data以及copying to tmp table等待)
mysql>  set profiling=1;
Query OK, 0 rows affected (0.00 sec)

mysql>  show profiles;
Empty set (0.00 sec)

mysql> explain
    ->  SELECT a.tid, a.subject,a.dateline,a.fid,b.message
    -> FROM forum_thread AS a
    -> LEFT JOIN forum_post_8 AS b ON a.tid = b.tid
    -> WHERE a.highlight!=0 AND b.first=1 AND a.fid=113 AND b.invisible=0
    -> ORDER BY a.dateline DESC
    -> LIMIT 0,5;
+----+-------------+-------+--------+------------------------------+-----------+---------+-------------+-------+----------------------------------------------+
| id | select_type | table | type   | possible_keys                | key       | key_len | ref         | rows  | Extra                                        |
+----+-------------+-------+--------+------------------------------+-----------+---------+-------------+-------+----------------------------------------------+
|  1 | SIMPLE      | b     | ref    | invisible,displayorder,first | invisible | 1       | const       | 97042 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | a     | eq_ref | PRIMARY,displayorder,typeid  | PRIMARY   | 8       | forum.b.tid |     1 | Using where                                  |
+----+-------------+-------+--------+------------------------------+-----------+---------+-------------+-------+----------------------------------------------+
2 rows in set (1.29 sec)

mysql>  SELECT a.tid, a.subject,a.dateline,a.fid,b.message
    -> FROM forum_thread AS a
    -> LEFT JOIN forum_post_8 AS b ON a.tid = b.tid
    -> WHERE a.highlight!=0 AND b.first=1 AND a.fid=113 AND b.invisible=0
    -> ORDER BY a.dateline DESC
    -> LIMIT 0,5;



执行这条SQL后，总耗时为 52s 查看具体步骤的耗时
 
mysql> show profiles;
+----------+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Query_ID | Duration    | Query                                                                                                                                                                                                                                  |
+----------+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|        1 |  1.28875000 | explain
SELECT a.tid, a.subject,a.dateline,a.fid,b.message
FROM forum_thread AS a
LEFT JOIN forum_post_8 AS b ON a.tid = b.tid
WHERE a.highlight!=0 AND b.first=1 AND a.fid=113 AND b.invisible=0
ORDER BY a.dateline DESC
LIMIT 0,5 |
|        2 | 52.027014 | SELECT a.tid, a.subject,a.dateline,a.fid,b.message
FROM forum_thread AS a
LEFT JOIN forum_post_8 AS b ON a.tid = b.tid
WHERE a.highlight!=0 AND b.first=1 AND a.fid=113 AND b.invisible=0
ORDER BY a.dateline DESC
LIMIT 0,5           |                                                                                                                                                                                                               |
+----------+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

 
mysql> show profile cpu,block io,memory,swaps for query 2;
+--------------------------------+-----------+----------+------------+--------------+---------------+-------+
| Status                         | Duration  | CPU_user | CPU_system | Block_ops_in | Block_ops_out | Swaps |
+--------------------------------+-----------+----------+------------+--------------+---------------+-------+
| starting                       |  0.000019 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking query cache for query |  0.000083 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking permissions           |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking permissions           |  0.000005 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Opening tables                 |  0.000020 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| System lock                    |  0.000008 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000025 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| init                           |  0.000038 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| optimizing                     |  0.000015 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| statistics                     |  0.000113 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| preparing                      |  0.000025 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Creating tmp table             |  0.000162 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| executing                      |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
|  Copying to tmp table           | 52.027014  | 4.992241 |   1.324799 |            0 |             0 |     0 |
| Sorting result                 |  0.000066 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   |  0.000218 | 0.000000 |   0.000999 |            0 |             0 |     0 |
| end                            |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| removing tmp table             |  0.000093 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| end                            |  0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| query end                      |  0.000005 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| closing tables                 |  0.000010 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  |  0.000012 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  |  0.000026 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  |  0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| storing result in query cache  |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| logging slow query             |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| logging slow query             |  0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| cleaning up                    |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
+--------------------------------+-----------+----------+------------+--------------+---------------+-------+
31 rows in set (0.00 sec)

mysql>  show profile cpu,CONTEXT SWITCHES,PAGE FAULTS for query 2;
+--------------------------------+-----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
| Status                         | Duration  | CPU_user | CPU_system | Context_voluntary | Context_involuntary | Page_faults_major | Page_faults_minor |
+--------------------------------+-----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
| starting                       |  0.000019 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Waiting for query cache lock   |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking query cache for query |  0.000083 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking permissions           |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking permissions           |  0.000005 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Opening tables                 |  0.000020 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| System lock                    |  0.000008 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Waiting for query cache lock   |  0.000025 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| init                           |  0.000038 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| optimizing                     |  0.000015 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| statistics                     |  0.000113 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| preparing                      |  0.000025 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Creating tmp table             |  0.000162 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 5 |
| executing                      |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
|  Copying to tmp table           | 52.027014 | 4.992241 |   1.324799 |             17969 |                1828 |                 1 |            251431 |
| Sorting result                 |  0.000066 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                14 |
| Sending data                   |  0.000218 | 0.000000 |   0.000999 |                 0 |                   1 |                 0 |                 1 |
| end                            |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| removing tmp table             |  0.000093 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| end                            |  0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| query end                      |  0.000005 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| closing tables                 |  0.000010 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  |  0.000012 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Waiting for query cache lock   |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  |  0.000026 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 4 |
| Waiting for query cache lock   |  0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  |  0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| storing result in query cache  |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| logging slow query             |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| logging slow query             |  0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| cleaning up                    |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
+--------------------------------+-----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
31 rows in set (0.00 sec)

同时可以用iostat看到硬盘的IO   rkB/s  非常高
# iostat -x /dev/sda8  2 10
Linux 2.6.18-164.el5 (photo)    01/15/2013

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.68    0.00    0.29    0.44   98.59

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8        46.27   0.00 344.28  0.00 39323.38    0.00 19661.69     0.00   114.22     2.41    6.92   2.89  99.45

avg-cpu:  %user   %nice    %sys %iowait   %idle
           1.12    0.00    0.94   11.18   86.76

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8        31.84   1.00 327.86  1.00 32636.82   15.92 16318.41     7.96    99.29     2.30    7.02   3.02  99.45

avg-cpu:  %user   %nice    %sys %iowait   %idle
           1.50    0.00    1.06   10.94   86.50

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8        83.92   0.00 348.74  0.00 43907.54    0.00 21953.77     0.00   125.90     2.47    7.10   2.88 100.55

avg-cpu:  %user   %nice    %sys %iowait   %idle
           1.62    0.00    1.12   11.62   85.63

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8        64.18   0.00 365.17  0.00 49532.34    0.00 24766.17     0.00   135.64     2.42    6.63   2.72  99.45

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.50    0.00    0.88   11.44   87.19

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8         6.03   1.01 345.73  1.01 9543.72   16.08  4771.86     8.04    27.57     1.21    3.49   2.86  99.25




2.第一次改写SQL

mysql> explain 
select AA.*,b.message from (select a.tid, a.subject,a.dateline,a.fid
from forum_thread AS a
where  a.highlight!=0 AND a.fid =113)  AA
LEFT JOIN
forum_post_8 AS b ON AA.tid = b.tid
where b.first=1 and b.invisible=0
order by AA.dateline desc ;

+----+-------------+------------+------+------------------------------+--------------+---------+--------------+--------+----------------+
| id | select_type | table      | type | possible_keys                | key          | key_len | ref          | rows   | Extra          |
+----+-------------+------------+------+------------------------------+--------------+---------+--------------+--------+----------------+
|  1 | PRIMARY     | | ALL  | NULL                         | NULL         | NULL    | NULL         |     99 | Using filesort |
|  1 | PRIMARY     | b          | ref  | invisible,displayorder,first | displayorder | 9       | AA.tid,const |      1 | Using where    |
|  2 | DERIVED     | a          | ref  | displayorder,typeid          | typeid       | 8       |              | 108440 | Using where    |
+----+-------------+------------+------+------------------------------+--------------+---------+--------------+--------+----------------+

typeid的索引是(fid+typeid)的组合索引
执行改写后的SQL后查看执行耗时情况
mysql>  show profiles;
+----------+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Query_ID | Duration    | Query                                                                                                                                                                                                                               |
+----------+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|        1 |   20.36569700   | select * from (select a.tid, a.subject,a.dateline,a.fid
from forum_thread AS a
where  a.highlight!=0 AND a.fid =113)  AA
LEFT JOIN
forum_post_8 AS b ON AA.tid = b.tid
where b.first=1 and b.invisible=0
order by AA.dateline desc |
+----------+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql>   show profile for query 1;
mysql>  show profile cpu,block io,memory,swaps for query 1;
+--------------------------------+-----------+----------+------------+--------------+---------------+-------+
| Status                         | Duration  | CPU_user | CPU_system | Block_ops_in | Block_ops_out | Swaps |
+--------------------------------+-----------+----------+------------+--------------+---------------+-------+
| starting                       |  0.000025 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking query cache for query |  0.000127 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking permissions           |  0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking permissions           |  0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Opening tables                 |  4.817902 | 0.045993 |   0.010998 |            0 |             0 |     0 |
| System lock                    |  0.021866 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| optimizing                     |  0.000024 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| statistics                     |  0.507939 | 0.004000 |   0.002000 |            0 |             0 |     0 |
| preparing                      |  0.000038 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| executing                      |  0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   | 25.623515 | 1.406786 |   0.519921 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000006 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   |  0.000026 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| init                           |  0.000049 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| optimizing                     |  0.000018 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| statistics                     |  0.198725 | 0.001000 |   0.000000 |            0 |             0 |     0 |
| preparing                      |  0.000026 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| executing                      |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sorting result                 |  0.000062 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   |  0.998911 | 0.003999 |   0.004999 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000006 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   |  1.061145 | 0.001000 |   0.003000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000011 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   |  0.008693 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| end                            |  0.000007 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| query end                      |  0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| closing tables                 |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| removing tmp table             |  0.000007 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| closing tables                 |  0.000010 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  |  0.000019 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  |  0.000007 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  |  0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| storing result in query cache  |  0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| logging slow query             |  0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| logging slow query             |  0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| cleaning up                    |  0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
+--------------------------------+-----------+----------+------------+--------------+---------------+-------+
39 rows in set (0.00 sec)


mysql>  show profile cpu,CONTEXT SWITCHES,PAGE FAULTS for query 1;
+--------------------------------+-----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
| Status                         | Duration  | CPU_user | CPU_system | Context_voluntary | Context_involuntary | Page_faults_major | Page_faults_minor |
+--------------------------------+-----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
| starting                       |  0.000025 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| Waiting for query cache lock   |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking query cache for query |  0.000127 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                30 |
| checking permissions           |  0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking permissions           |  0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Opening tables                 |  4.817902 | 0.045993 |   0.010998 |               374 |                   3 |                 1 |               998 |
| System lock                    |  0.021866 | 0.000000 |   0.000000 |                 1 |                   0 |                 1 |                33 |
| optimizing                     |  0.000024 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                10 |
| statistics                     |  0.507939 | 0.004000 |   0.002000 |                37 |                   0 |                 1 |               120 |
| preparing                      |  0.000038 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                10 |
| executing                      |  0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| Sending data                   | 25.623515 | 1.406786 |   0.519921 |             13625 |                 158 |                 0 |             74911 |
| Waiting for query cache lock   |  0.000006 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   |  0.000026 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| init                           |  0.000049 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| optimizing                     |  0.000018 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| statistics                     |  0.198725 | 0.001000 |   0.000000 |                11 |                   0 |                 0 |                45 |
| preparing                      |  0.000026 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| executing                      |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sorting result                 |  0.000062 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 4 |
| Sending data                   |  0.998911 | 0.003999 |   0.004999 |                46 |                   0 |                 0 |               131 |
| Waiting for query cache lock   |  0.000006 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   |  1.061145 | 0.001000 |   0.003000 |                36 |                   0 |                 0 |               119 |
| Waiting for query cache lock   |  0.000011 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   |  0.008693 | 0.000000 |   0.000000 |                 2 |                   3 |                 0 |                 8 |
| end                            |  0.000007 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| query end                      |  0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| closing tables                 |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| removing tmp table             |  0.000007 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| closing tables                 |  0.000010 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  |  0.000019 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 2 |
| Waiting for query cache lock   |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  |  0.000007 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Waiting for query cache lock   |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  |  0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| storing result in query cache  |  0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| logging slow query             |  0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| logging slow query             |  0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| cleaning up                    |  0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
+--------------------------------+-----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
39 rows in set (0.00 sec)

同样在iostat看到 磁盘IO read 仍然比较高，但队列长度avgrq-sz比之前小。
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8        48.50   7.00 879.00  6.00 48036.00  104.00 24018.00    52.00    54.40     4.39    4.82   1.13 100.05

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.50    0.00    0.94   11.06   87.50

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8        15.00   0.00 524.00  0.00 20740.00    0.00 10370.00     0.00    39.58     3.41    6.69   1.91 100.05

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.12    0.00    0.75   11.62   87.51

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8         3.50   0.00 328.00  0.00 8108.00    0.00  4054.00     0.00    24.72     2.99    9.18   3.05 100.05

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.25    0.00    0.81   11.44   87.50

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda8         3.00   4.00 432.50  1.00 13496.00   40.00  6748.00    20.00    31.22     3.13    7.22   2.31 100.05

3.第二次改写SQL

对sql进行进一步分析

子查询的执行时间其实还有优化的余地，首先看了下子查询中的数据分别情况

mysql> select a.highlight,count(*) from forum_thread AS a group by a.highlight;
+-----------+----------+
| highlight | count(*) |
+-----------+----------+
|         0 |  1522463 |
|         1 |     5576 |
|         2 |       13 |
|         3 |        1 |
|         4 |        9 |
|         5 |        9 |
|         6 |        1 |
|         7 |        7 |
|         8 |        5 |
|        40 |        2 |
|        41 |       55 |
|        42 |       31 |
|        43 |        7 |
|        44 |       34 |
|        45 |       25 |
|        46 |        1 |
|        47 |        2 |
|        48 |        5 |
|        63 |        1 |
|        70 |        1 |
|       127 |        6 |
+-----------+----------+
21 rows in set (18.71 sec)
 
mysql> select fid,count(*) from forum_thread group by fid order by 2 desc;
+------+----------+
| fid  | count(*) |
+------+----------+
| 1233 |   370531 |
|  273 |   165592 |
|  268 |    69730 |
|  266 |    62790 |
|  255 |    49992 |
| 1118 |    47293 |
|  113 |    44690 |
|  147 |    44318 |
|  252 |    43084 |
|  253 |    35864 |
|  114 |    33344 |
| 1170 |    24051 |
| 1230 |    23133 |
|  269 |    21613 |
|  254 |    20629 |
|  256 |    19073 |
| 1182 |    17588 |

-----后面省略很多其它值

可以看到其实 highlight !=0 这个结果集是非常小的，如果有fid+highlight 的组合索引的话，子查询
select a.tid, a.subject,a.dateline,a.fid
from forum_thread AS a
where  a.highlight!=0 AND a.fid =113 
的索引范围扫描的结果集会非常小，比起原来只有fid的索引，需要回表过滤highlight=0的rows效率要高不少。
于是建立索引
ALTER TABLE `forum_thread`     ADD INDEX `fid_high` (`fid`, `highlight`);

----
这时第2步改写的sql的执行计划变为了：
mysql> explain  select AA.*,b.message from (select a.tid, a.subject,a.dateline,a.fid
    -> from forum_thread AS a
    -> where  a.highlight!=0 AND a.fid =113)  AA
    -> LEFT JOIN
    ->  forum_post_8 AS b ON AA.tid = b.tid
    -> where b.first=1 and b.invisible=0
    -> order by AA.dateline desc ;
+----+-------------+------------+-------+------------------------------+--------------+---------+--------------+------+----------------+
| id | select_type | table      | type  | possible_keys                | key          | key_len | ref          | rows | Extra          |
+----+-------------+------------+-------+------------------------------+--------------+---------+--------------+------+----------------+
|  1 | PRIMARY     | | ALL   | NULL                         | NULL         | NULL    | NULL         |   99 | Using filesort |
|  1 | PRIMARY     | b          | ref   | invisible,displayorder,first | displayorder | 9       | AA.tid,const |    1 | Using where    |
|  2 | DERIVED     | a          | range | displayorder,typeid,fid_high | fid_high     | 9       | NULL         |  100 | Using where    |
+----+-------------+------------+-------+------------------------------+--------------+---------+--------------+------+----------------+
3 rows in set (0.00 sec)

--执行一次这条sql后查看执行的具体时间耗时情况
mysql>  show profiles;
+----------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Query_ID | Duration   | Query                                                                                                                                                                                                                              |
+----------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|        1 | 2.07984400 | select * from (select a.tid, a.subject,a.dateline,a.fid
from forum_thread AS a
where  a.highlight>0 AND a.fid =113)  AA
LEFT JOIN
forum_post_8 AS b ON AA.tid = b.tid
where b.first=1 and b.invisible=0
order by AA.dateline desc |
+----------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

执行时间是2s
mysql>  show profile cpu,block io,memory,swaps for query 1;
+--------------------------------+----------+----------+------------+--------------+---------------+-------+
| Status                         | Duration | CPU_user | CPU_system | Block_ops_in | Block_ops_out | Swaps |
+--------------------------------+----------+----------+------------+--------------+---------------+-------+
| starting                       | 0.000022 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   | 0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking query cache for query | 0.000105 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking permissions           | 0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| checking permissions           | 0.000005 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Opening tables                 | 1.222125 | 0.027996 |   0.007999 |            0 |             0 |     0 |
| System lock                    | 0.000240 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| optimizing                     | 0.000013 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| statistics                     | 0.124458 | 0.001000 |   0.002000 |            0 |             0 |     0 |
| preparing                      | 0.000025 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| executing                      | 0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   | 0.450222 | 0.007999 |   0.009998 |            0 |             0 |     0 |
| Waiting for query cache lock   | 0.000006 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   | 0.000034 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| init                           | 0.000220 | 0.000999 |   0.000000 |            0 |             0 |     0 |
| optimizing                     | 0.000013 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| statistics                     | 0.025380 | 0.000000 |   0.001000 |            0 |             0 |     0 |
| preparing                      | 0.000024 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| executing                      | 0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sorting result                 | 0.000054 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   | 0.154623 | 0.003000 |   0.001000 |            0 |             0 |     0 |
| Waiting for query cache lock   | 0.000003 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   | 0.101302 | 0.003999 |   0.002000 |            0 |             0 |     0 |
| Waiting for query cache lock   | 0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Sending data                   | 0.000834 | 0.001000 |   0.000000 |            0 |             0 |     0 |
| end                            | 0.000063 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| query end                      | 0.000004 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| closing tables                 | 0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| removing tmp table             | 0.000006 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| closing tables                 | 0.000010 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  | 0.000014 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   | 0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  | 0.000012 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| Waiting for query cache lock   | 0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| freeing items                  | 0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| storing result in query cache  | 0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| logging slow query             | 0.000001 | 0.000000 |   0.000000 |            0 |             0 |     0 |
| cleaning up                    | 0.000002 | 0.000000 |   0.000000 |            0 |             0 |     0 |
+--------------------------------+----------+----------+------------+--------------+---------------+-------+
38 rows in set (0.00 sec)
mysql>   show profile cpu,CONTEXT SWITCHES,PAGE FAULTS for query 1;
+--------------------------------+----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
| Status                         | Duration | CPU_user | CPU_system | Context_voluntary | Context_involuntary | Page_faults_major | Page_faults_minor |
+--------------------------------+----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
| starting                       | 0.000022 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Waiting for query cache lock   | 0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking query cache for query | 0.000105 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 4 |
| checking permissions           | 0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| checking permissions           | 0.000005 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Opening tables                 | 1.222125 | 0.027996 |   0.007999 |               378 |                   0 |                 0 |               986 |
| System lock                    | 0.000240 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 4 |
| optimizing                     | 0.000013 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| statistics                     | 0.124458 | 0.001000 |   0.002000 |                36 |                   0 |                 0 |               102 |
| preparing                      | 0.000025 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| executing                      | 0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   | 0.450222 | 0.007999 |   0.009998 |               158 |                   0 |                 0 |               480 |
| Waiting for query cache lock   | 0.000006 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   | 0.000034 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 2 |
| init                           | 0.000220 | 0.000999 |   0.000000 |                 0 |                   0 |                 0 |                 3 |
| optimizing                     | 0.000013 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| statistics                     | 0.025380 | 0.000000 |   0.001000 |                 9 |                   0 |                 0 |                37 |
| preparing                      | 0.000024 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| executing                      | 0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sorting result                 | 0.000054 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 3 |
| Sending data                   | 0.154623 | 0.003000 |   0.001000 |                46 |                   0 |                 0 |               136 |
| Waiting for query cache lock   | 0.000003 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   | 0.101302 | 0.003999 |   0.002000 |                31 |                   0 |                 0 |               115 |
| Waiting for query cache lock   | 0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Sending data                   | 0.000834 | 0.001000 |   0.000000 |                 1 |                   0 |                 0 |                 8 |
| end                            | 0.000063 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| query end                      | 0.000004 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| closing tables                 | 0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| removing tmp table             | 0.000006 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| closing tables                 | 0.000010 | 0.000000 |   0.000000 |                 1 |                   0 |                 0 |                 0 |
| freeing items                  | 0.000014 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| Waiting for query cache lock   | 0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  | 0.000012 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 1 |
| Waiting for query cache lock   | 0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| freeing items                  | 0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| storing result in query cache  | 0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| logging slow query             | 0.000001 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
| cleaning up                    | 0.000002 | 0.000000 |   0.000000 |                 0 |                   0 |                 0 |                 0 |
+--------------------------------+----------+----------+------------+-------------------+---------------------+-------------------+-------------------+
38 rows in set (0.00 sec)



最终这条sql的执行时间由最初的50s 优化到2s 


4.结论

处理Copying to tmp table除了调整mysql的参数tmp_table_size，（这个参数的意思具体见
http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_tmp_table_size）更重要的是优化sql，减少IO读写。其实调整tmp_table_size的意义也仅仅在于调整后尽量不写on-disk table，对于很高的read I/O 导致的性能问题帮助并不大。



根据mysql手册上面所说的
http://dev.mysql.com/doc/refman/5.5/en/general-thread-states.html
Copying to tmp table
The server is copying to a temporary table in memory.

当mysql出现Copying to tmp table的事件的时候首先mysql引擎将数据写入内存里的临时表，这个临时表的大小由参数tmp_table_size决定，
事实上另外一个参数max_heap_table_size 也会对实际的临时表的大小有影响，实际值会取这2个参数中最小的那个。
如果需要的的临时表大小超过了tmp_table_size，则写到硬盘上的"on-disk MyISAM table"上,这个时候iostat会出现比较高的wkB/s

当有多条sql都同时出现Copying to tmp table的时候，服务器的IO带来非常大的压力，这时基本上其它正常的sql都会由于IO的竞争变得非常慢，阻塞了其它sql执行，表现到应用层面上就是相关的页面打不开，数据查询显示不出来。大量的sending data以及copying to tmp table等待


关于Copying to tmp table的介绍的一篇文章
http://www.dbtuna.com/article/55/Copying_to_tmp_table_-_MySQL_thread_states




备注1：show profile 参数

ALL - displays all information 
BLOCK IO - displays counts for block input and output operations 
CONTEXT SWITCHES - displays counts for voluntary and involuntary context switches
IPC - displays counts for messages sent and received 
MEMORY - is not currently implemented 
PAGE FAULTS - displays counts for major and minor page faults 
SOURCE - displays the names of functions from the source code, together with the name and line number of the file in which the function occurs 
SWAPS - displays swap counts 
  
mysql> show variables like 'profiling_history_size'                    
    -> ;
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| profiling_history_size | 15    |
+------------------------+-------+
1 row in set (0.00 sec)

mysql> show variables like '%profiling%';
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| have_profiling         | YES   |
| profiling              | OFF   |
| profiling_history_size | 15    |
+------------------------+-------+
3 rows in set (0.00 sec)
 
mysql> set profiling=1; #开启

备注2. mysql profile 相关介绍

A voluntary context switch occurs when a thread gives up the CPU it is running on voluntarily, either by a specialized call (sched_yield() or similar) or by starting waiting (e.g. a blocking I/O call).

Contrast with the involuntary context switch.

An involuntary context switch means that the kernel takes away the CPU from a running process to assign it to another process. This can occur because:

  • The process was already running for too long.
  • A process with higher priority needs the CPU right now.
Technically this is done by a hardware interrupt that passes control to the kernel which then passes the control to another chosen process.
----------------------------------------------
page_fault

Minor

If the page is loaded in memory at the time the fault is generated, but is not marked in the memory management unit as being loaded in memory, then it is called a minor or soft page fault. The page fault handler in the operating system merely needs to make the entry for that page in the memory management unit point to the page in memory and indicate that the page is loaded in memory; it does not need to read the page into memory. This could happen if the memory is shared by different programs and the page is already brought into memory for other programs. The page could also have been removed from a process's Working Set, but not yet written to disk or erased, such as in operating systems that use Secondary Page Caching. For example, HP OpenVMS may remove a page that does not need to be written to disk (if it has remained unchanged since it was last read from disk, for example) and place it on a Free Page List if the working set is deemed too large. However, the page contents are not overwritten until the page is assigned elsewhere, meaning it is still available if it is referenced by the original process before being allocated. Since these faults do not involve disk latency, they are faster and less expensive than major page faults.


Major

If the page is not loaded in memory at the time the fault is generated, then it is called a major or hard page fault. The page fault handler in the operating system needs to find a free page in memory, or choose another non-free page in memory to be used for this page's data, which might be used by another process. In this latter case, the OS first needs to write out the data in that page if it hasn't already been written out since it was last modified, and mark that page as not being loaded into memory in its process page table. Once the page has thus been made available, the OS can read the data for the new page into the physical page, and then make the entry for that page in the memory management unit point to the page in memory and indicate that the page is loaded in memory.[clarification needed] Major faults are more expensive than minor page faults and add disk latency to the interrupted program's execution. This is the mechanism used by an operating system to increase the amount of program memory available on demand. The operating system delays loading parts of the program from disk until the program attempts to use it and the page fault is generated.


备注3.清理mysql缓存方法

（不停库没法完全清理mysql的缓存）

service mysqld stop;
sync;
echo 3 > /proc/sys/vm/drop_caches ;
service mysqld start;

你可能感兴趣的:(MySql)

  • mysql禁用远程登录 igotyback mysql
    去mysql库中的user表里，将host都改成localhost之后刷新权限FLUSHPRIVILEGES;
  • MYSQL面试系列-04 king01299 面试mysql面试
    MYSQL面试系列-0417.关于redolog和binlog的刷盘机制、redolog、undolog作用、GTID是做什么的？innodb_flush_log_at_trx_commit及sync_binlog参数意义双117.1innodb_flush_log_at_trx_commit该变量定义了InnoDB在每次事务提交时，如何处理未刷入（flush）的重做日志信息（redolog）。它
  • 关于Mysql 中 Row size too large (＞ 8126) 错误的解决和理解 秋刀prince mysqlmysql数据库
    提示：啰嗦一嘴，数据库的任何操作和验证前，一定要记得先备份！！！不会有错；文章目录问题发现一、问题导致的可能原因1、页大小2、行格式2.1compact格式2.2Redundant格式2.3Dynamic格式2.4Compressed格式3、BLOB和TEXT列二、解决办法1、修改页大小（不推荐）2、修改行格式3、修改数据类型为BLOB和TEXT列4、其他优化方式（可以参考使用）4.1合理设置数据
  • MongoDB知识概括 GeorgeLin98 持久层mongodb
    MongoDB知识概括MongoDB相关概念单机部署基本常用命令索引-IndexSpirngDataMongoDB集成副本集分片集群安全认证MongoDB相关概念业务应用场景：传统的关系型数据库（如MySQL），在数据操作的“三高”需求以及应对Web2.0的网站需求面前，显得力不从心。解释：“三高”需求：①Highperformance-对数据库高并发读写的需求。②HugeStorage-对海量数
  • 入门MySQL——查询语法练习 K_un
    前言：前面几篇文章为大家介绍了DML以及DDL语句的使用方法，本篇文章将主要讲述常用的查询语法。其实MySQL官网给出了多个示例数据库供大家实用查询，下面我们以最常用的员工示例数据库为准，详细介绍各自常用的查询语法。1.员工示例数据库导入官方文档员工示例数据库介绍及下载链接：https://dev.mysql.com/doc/employee/en/employees-installation.h
  • 博客网站制作教程 2401_85194651 javamaven
    首先就是技术框架：后端：Java+SpringBoot数据库：MySQL前端：Vue.js数据库连接：JPA(JavaPersistenceAPI)1.项目结构blog-app/├──backend/│├──src/main/java/com/example/blogapp/││├──BlogApplication.java││├──config/│││└──DatabaseConfig.java
  • ubuntu安装wordpress lissettecarlr
    1安装nginx网上安装方式很多，这就就直接用apt-get了apt-getinstallnginx不用启动啥，然后直接在浏览器里面输入IP:80就能看到nginx的主页了。如果修改了一些配置可以使用下列命令重启一下systemctlrestartnginx.service2安装mysql输入安装前也可以更新一下软件源，在安装过程中将会让你输入数据库的密码。sudoapt-getinstallmy
  • 计算机毕业设计PHP仓储综合管理系统（源码+程序+VUE+lw+部署） java毕设程序源码王哥 php课程设计vue.js
    该项目含有源码、文档、程序、数据库、配套开发软件、软件安装教程。欢迎交流项目运行环境配置：phpStudy+Vscode+Mysql5.7+HBuilderX+Navicat11+Vue+Express。项目技术：原生PHP++Vue等等组成，B/S模式+Vscode管理+前后端分离等等。环境需要1.运行环境：最好是小皮phpstudy最新版，我们在这个版本上开发的。其他版本理论上也可以。2.开发
  • You have an error in your SQL syntax； check the manual that corresponds to your MySQL server version 努力的菜鸟~ sql数据库
    YouhaveanerrorinyourSQLsyntax;checkthemanualthatcorrespondstoyourMySQLserverversionfortherightsyntaxtousenear‘IDENTIFIEDBY‘123456’WITHGRANTOPTION’atline1在mysql5.7之前GRANTALLPRIVILEGESON*.*TO'root'@'%'I
  • mysql学习教程，从入门到精通，TOP 和MySQL LIMIT 子句（15） 知识分享小能手 大数据数据库MySQLmysql学习oracle数据库开发语言adb大数据
    1、TOP和MySQLLIMIT子句内容在SQL中，不同的数据库系统对于限制查询结果的数量有不同的实现方式。TOP关键字主要用于SQLServer和Access数据库中，而LIMIT子句则主要用于MySQL、PostgreSQL（通过LIMIT/OFFSET语法）、SQLite等数据库中。下面将分别详细介绍这两个功能的语法、语句以及案例。1.1、TOP子句（SQLServer和Access）1.1
  • 鲲鹏 ARM 架构 麒麟 Lylin v10 安装 Nginx (离线) 焚木灵 arm开发架构nginx服务器
    最近做一个银行的项目，银行的服务器是鲲鹏ARM架构的服务器，并且是麒麟v10的系统，这里记录一下在无法访问外网安装Nginx的方法。其他文章：鲲鹏ARM架构麒麟Lylinv10安装Mysql8.3(离线)-CSDN博客鲲鹏ARM架构麒麟Lylinv10安装Node和NVM(离线)-CSDN博客鲲鹏ARM架构麒麟Lylinv10安装Pm2(离线)-CSDN博客鲲鹏ARM架构麒麟Lylinv10安装P
  • Kubernetes部署MySQL数据持久化 沫殇-MS KubernetesMySQL数据库kubernetesmysql容器
    一、安装配置NFS服务端1、安装nfs-kernel-server：sudoapt-yinstallnfs-kernel-server2、服务端创建共享目录#列出所有可用块设备的信息lsblk#格式化磁盘sudomkfs-text4/dev/sdb#创建一个目录：sudomkdir-p/data/nfs/mysql#更改目录权限：sudochown-Rnobody:nogroup/data/nfs
  • MySQL事务隔离级别和MVCC 简书徐小耳
    MySQL事务隔离级别和MVCC参考：https://mp.weixin.qq.com/s/Jeg8656gGtkPteYWrG5_Nw1.MVCC只对读已提交和可重复的读有效果，而未提交读和串行则无意义。2.每条记录都会有trx_id(事务修改记录的id）和roll_pointer是一个指针指向旧版本的undo日志链表（row_id不是必必要的，如果有主键存在就不需要了）3.版本链的头结点就是记
  • CentOS7 安装MySQL5.7.44 不要Null了 javacentosmysql
    1.下载mysql安装包，我放在百度网盘里(下方链接)链接：https://pan.baidu.com/s/1_Mn1XW_1mWdTV4mhnLG66A提取码：s31n2.首先看看以前是否安装过mysqlrpm-qa|grep-imysql如果已经安装过mysql会提示卸载mysqlrpm-emysql-…3.使用FinallShell或者Xftp进行上传放到/usr/local/mysql，没
  • 非关系型数据库 天秤-white nosql
    一、为什么要用Nosql1.单机MySQL的时代。一个基本的网站访问量一般不会太大，单个数据库完全足够。那时候更多使用的静态网页html，服务器根本没有太大压力。这时候网站的瓶颈是什么？-数据量如果太大，一个机器放不下。-数据量太大需要建立数据的索引（B+Tree），一个服务器内存放不下。-访问量读写混合，一个服务器承受不了。2.memcached缓存+MySQL+垂直拆分（读写分离）。网站80%
  • 六、全局锁和表锁：给表加个字段怎么有这么多阻碍 nieniemin
    数据库锁设计的初衷是处理并发问题。作为多用户共享的资源，当出现并发访问的时候，数据库需要合理地控制资源的访问规则。而锁就是用来实现这些访问规则的重要数据结构。根据加锁的范围，MySQL里面的锁大致可以分成全局锁、表级锁和行锁三类。6.1全局锁全局锁就是对整个数据库实例加锁。MySQL提供了一个加全局读锁的方法，命令是Flushtableswithreadlock(FTWRL)。当你需要让整个库处于
  • Kubernetes 自定义控制器开发 IT回忆录 Kubeneteskubernetes
    目录前言一、CRD二、创建数据库表（Mysql）二、控制器开发1.使用kubernetes的examplecontroller模板2.在controller.go中新增数据表监听方法3.修改tools工具生成资源对象结构体定义这里记录开发k8s控制器的一般方式，controller开发主要使用k8s提供的client-go库进行。前言Controller监听集群内部资源对象的变化，编辑资源对象(增
  • 【K8s】专题十一：Kubernetes 集群证书过期处理方法 行者Sun1989 Kuberneteskubernetes云原生容器
    本文内容均来自个人笔记并重新梳理，如有错误欢迎指正！如果对您有帮助，烦请点赞、关注、转发、订阅专栏！专栏订阅入口Linux专栏|Docker专栏|Kubernetes专栏往期精彩文章【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法（续）【Docker】MySQL源码构建Docker镜
  • MySQl篇（SQL - 基本介绍）（持续更新迭代） wclass-zhengge mysqlsql数据库
    目录一、简介二、SQL方言（分页查询为例）1.简介2.SQL方言大比拼2.1.Oracle2.1.1.使用ROWNUM实现分页查询2.1.2.使用ROW_NUMBER()实现分页查询2.2.MySQL2.3.PostgreSQL三、语法规范四、注释五、MySQL脚本中的标点符号一、简介1、SQL是结构化查询语言（StructureQueryLanguage），专门用来操作/访问关系型数据库的通用语
  • 跟着黑马学mysql（5） 小杜不吃糖 mysql数据库
    17.DQL-聚合函数DQL-聚合函数介绍将一列数据作为一个整体，进行纵向计算。常见聚合函数函数功能count统计数量max最大值min最小值avg平均值sum求和语法SELECT聚合函数(字段列表)FROM表名;注意：所有的null值不参与聚合函数的运算18.DQL-分组查询语法SELECT字段列表FROM表名[WHERE条件]GROUPBY分组字段名[HAVING分组后的过滤条件];where
  • 梧桐数据库（WuTongDB）：数据库技术中都有哪些常见的优化器 鲁鲁517 梧桐数据库梧桐数据库
    以下是一些常见的数据库优化器：1.CBO（Cost-BasedOptimizer）应用场景：广泛应用于关系型数据库中，如Oracle、PostgreSQL、MySQL等。工作原理：通过计算不同执行计划的代价（如CPU、I/O等资源消耗），选择最低代价的执行计划。代表数据库：Oracle、PostgreSQL、MySQL。特点：CBO使用统计信息（如表大小、索引分布）来评估查询的代价。2.RBO（R
  • MySQL日志 沉着冷静2024 MySQLmysql数据库
    MySQL日志文章目录MySQL日志MySQL三大日志binlog的三种格式redolog和binlog的区别和应用场景为什么崩溃恢复不用binlog而用redolog？redolog如何实现持久化redolog还能做什么？redolog的三种刷盘策略两阶段提交什么是？为什么？两阶段提交过程MySQL三大日志1.undologundolog是InnoDB存储引擎层的日志，实现了事务的原子性，主要用
  • MySQL锁 沉着冷静2024 MySQLmysql数据库
    MySQL锁文章目录MySQL锁MySQL中锁的分类创建索引时会锁表吗线上修改表结构会加什么锁Innodb存储引擎的行级锁有哪些Update语句中，不带where条件，加什么锁？MySQL实现乐观锁MySQL死锁MySQL死锁是怎么发生的？检查死锁如何避免死锁MySQL中锁的分类全局锁：主要用于全库逻辑备份表级锁：表锁、元数据锁、意向锁表锁：通过locktables语句对表进行加锁，它不仅限制其他
  • Python实现mysql命令行 xu-jssy pythonmysqladb
    一、源码importosimportpymysqldefsql_shell():password=input("EnterPassword:")#访问密码ifpassword.strip()!="yyds":print("Bye")return#清空控制台输出os.system("cls"ifos.name=="nt"else"clear")try:#连接到MySQL数据库conn=pymysql
  • mysql case 干货 lack倪酱
    这是我舍友@某人提供的是为了做酒店时判断每个楼栋有几间空房子的。房间表与数据楼栋表与数据查询楼栋房间总数查询空房间数当时就在想怎么把后一个单列的数据合并到前面那三列的数据里呢？又是百度，又是谷歌，都是什么“unitall”。。表示我要的不是这个。有点难为我小脑瓜，实在是对数据库不了解，只会日常操作。最后是我舍友想出了解决办法直接贴图好了，如果想要了解case是什么，你可以自行查资料。学习查东西也是
  • 计算机毕设Node.js+Vue校园易购二手交易平台（程序+LW+部署） Node程序源码强子 vue.js课程设计node.js
    项目运行环境配置：Node.js最新版+Vscode+Mysql5.7+HBuilderX+Navicat11+Vue。项目技术：Express框架+Node.js+Vue等等组成，B/S模式+Vscode管理+前后端分离等等。环境需要1.运行环境：最好是Nodejs最新版，我们在这个版本上开发的。其他版本理论上也可以。2.开发
  • MySQL 源码｜55 - 语法解析(V2)：基础查询语句（query_specification）｜V20240915 长行 MySQL源码mysql源码语法解析查询
    目录文档：MySQL源码｜源码剖析文档目录源码位置（版本=MySQL8.0.37）：sql/sql_yacc.yy前置文档：MySQL源码｜68-语法解析(V2)：LOCKING子句（锁定读取）｜V20240909MySQL源码｜39-语法解析(V2)：ORDERBY子句｜V20240814｜V20240912（第2版）MySQL源码｜40-语法解析(V2)：GROUPBY子句｜V20240814
  • 老生常谈：MySQL高可用架构 我有一头小花驴 mysql架构数据库
    引言“高可用”是互联网一个永恒的话题，先避开MySQL不谈，为了保证各种服务的高可用有几种常用的解决方案。服务冗余：把服务部署多份，当某个节点不可用时，切换到其他节点。服务冗余对于无状态的服务是相对容易的。服务备份：有些服务是无法同时存在多个运行时的，比如说：Nginx的反向代理，一些集群的leader节点。这时可以存在一个备份服务，处于随时待命状态。自动切换：服务冗余之后，当某个节点不可用时，要
  • mysql 隐秘后门_【技术分享】CVE-2016-5483：利用mysqldump备份可生成后门 Toby Dai mysql隐秘后门
    预估稿费：100RMB投稿方式：发送邮件至linwei#360.cn，或登陆网页版在线投稿前言mysqldump是用来创建MySQL数据库逻辑备份的一个常用工具。它在默认配置下可以生成一个.sql文件，其中包含创建/删除表和插入数据等。在导入转储文件的时候，攻击者可以通过制造恶意表名来实现任意SQL语句查询和shell命令执行的目的。另一个与之相关的漏洞利用场景可以参考。攻击场景攻击者已经能够访问
  • 处理绿盟科技安全评估的系统漏洞 自我修炼的小石头 数据库运维开发工具
    如下图一次扫描,中度风险39个,这个是必须要处理的.其中mysql占了36个,一看这个感觉整个人都不好了.但是最后解决的办法也很简单,就是升级.系统版本Redhat7.31.telnet因为要升级openssh,openssl,为了避免意外,首先要开启telnet服务,并把所有相关的包上传到服务器.1.1确认是否安装了telnet和xinetd（默认是没有安装）rpm-qatelnet*rpm-q
  • 关于旗正规则引擎下载页面需要弹窗保存到本地目录的问题 何必如此 jsp超链接文件下载窗口
    生成下载页面是需要选择“录入提交页面”，生成之后默认的下载页面<a>标签超链接为：<a href="<%=root_stimage%>stimage/image.jsp?filename=<%=strfile234%>&attachname=<%=java.net.URLEncoder.encode(file234filesourc
  • 【Spark九十八】Standalone Cluster Mode下的资源调度源代码分析 bit1129 cluster
    在分析源代码之前，首先对Standalone Cluster Mode的资源调度有一个基本的认识： 首先，运行一个Application需要Driver进程和一组Executor进程。在Standalone Cluster Mode下，Driver和Executor都是在Master的监护下给Worker发消息创建(Driver进程和Executor进程都需要分配内存和CPU，这就需要Maste
  • linux上独立安装部署spark daizj linux安装spark1.4部署
    下面讲一下linux上安装spark，以 Standalone Mode 安装   1）首先安装JDK   下载JDK：jdk-7u79-linux-x64.tar.gz  ，版本是1.7以上都行，解压 tar -zxvf jdk-7u79-linux-x64.tar.gz     然后配置 ~/.bashrc&nb
  • Java 字节码之解析一 周凡杨 java字节码javap
        一： Java 字节代码的组织形式   类文件 {     OxCAFEBABE ，小版本号，大版本号，常量池大小，常量池数组，访问控制标记，当前类信息，父类信息，实现的接口个数，实现的接口信息数组，域个数，域信息数组，方法个数，方法信息数组，属性个数，属性信息数组 } &nbs
  • java各种小工具代码 g21121 java
    1.数组转换成List import java.util.Arrays; Arrays.asList(Object[] obj); 2.判断一个String型是否有值 import org.springframework.util.StringUtils; if (StringUtils.hasText(str)) 3.判断一个List是否有值 import org.spring
  • 加快FineReport报表设计的几个心得体会 老A不折腾 finereport
    一、从远程服务器大批量取数进行表样设计时，最好按“列顺序”取一个“空的SQL语句”，这样可提高设计速度。否则每次设计时模板均要从远程读取数据，速度相当慢！！ 二、找一个富文本编辑软件（如NOTEPAD+）编辑SQL语句，这样会很好地检查语法。有时候带参数较多检查语法复杂时，结合FineReport中生成的日志，再找一个第三方数据库访问软件（如PL/SQL）进行数据检索，可以很快定位语法错误。
  • mysql linux启动与停止 墙头上一根草
    如何启动/停止/重启MySQL一、启动方式1、使用 service 启动：service mysqld start2、使用 mysqld 脚本启动：/etc/inint.d/mysqld start3、使用 safe_mysqld 启动：safe_mysqld&二、停止1、使用 service 启动：service mysqld stop2、使用 mysqld 脚本启动：/etc/inin
  • Spring中事务管理浅谈 aijuans spring事务管理
    Spring中事务管理浅谈 By Tony Jiang@2012-1-20 Spring中对事务的声明式管理 拿一个XML举例 [html]  view plain copy print ? <?xml version="1.0" encoding="UTF-8"?>&nb
  • php中隐形字符65279（utf-8的BOM头）问题 alxw4616
    php中隐形字符65279（utf-8的BOM头）问题 今天遇到一个问题. php输出JSON 前端在解析时发生问题:parsererror.   调试: 1.仔细对比字符串发现字符串拼写正确.怀疑是 非打印字符的问题. 2.逐一将字符串还原为unicode编码. 发现在字符串头的位置出现了一个 65279的非打印字符.        
  • 调用对象是否需要传递对象(初学者一定要注意这个问题) 百合不是茶 对象的传递与调用技巧
        类和对象的简单的复习,在做项目的过程中有时候不知道怎样来调用类创建的对象,简单的几个类可以看清楚,一般在项目中创建十几个类往往就不知道怎么来看   为了以后能够看清楚,现在来回顾一下类和对象的创建,对象的调用和传递(前面写过一篇)   类和对象的基础概念:   JAVA中万事万物都是类 类有字段(属性),方法,嵌套类和嵌套接
  • JDK1.5 AtomicLong实例 bijian1013 javathreadjava多线程AtomicLong
    JDK1.5 AtomicLong实例 类 AtomicLong 可以用原子方式更新的 long 值。有关原子变量属性的描述，请参阅 java.util.concurrent.atomic 包规范。AtomicLong 可用在应用程序中（如以原子方式增加的序列号），并且不能用于替换 Long。但是，此类确实扩展了 Number，允许那些处理基于数字类的工具和实用工具进行统一访问。  
  • 自定义的RPC的Java实现 bijian1013 javarpc
            网上看到纯java实现的RPC，很不错。         RPC的全名Remote Process Call，即远程过程调用。使用RPC，可以像使用本地的程序一样使用远程服务器上的程序。下面是一个简单的RPC 调用实例，从中可以看到RPC如何
  • 【RPC框架Hessian一】Hessian RPC Hello World bit1129 Hello world
    什么是Hessian The Hessian binary web service protocol makes web services usable without requiring a large framework, and without learning yet another alphabet soup of protocols. Because it is a binary p
  • 【Spark九十五】Spark Shell操作Spark SQL bit1129 shell
    在Spark Shell上，通过创建HiveContext可以直接进行Hive操作   1. 操作Hive中已存在的表   [hadoop@hadoop bin]$ ./spark-shell Spark assembly has been built with Hive, including Datanucleus jars on classpath Welcom
  • F5　往header加入客户端的ip ronin47
    when HTTP_RESPONSE {if {[HTTP::is_redirect]}{         HTTP::header replace Location [string map {:port/ /} [HTTP::header value Location]]HTTP::header replace Lo
  • java-61-在数组中，数字减去它右边(注意是右边)的数字得到一个数对之差. 求所有数对之差的最大值。例如在数组{2, 4, 1, 16, 7, 5, bylijinnan java
    思路来自： http://zhedahht.blog.163.com/blog/static/2541117420116135376632/ 写了个java版的 public class GreatestLeftRightDiff { /** * Q61.在数组中，数字减去它右边(注意是右边)的数字得到一个数对之差。 * 求所有数对之差的最大值。例如在数组
  • mongoDB 索引 开窍的石头 mongoDB索引
    在这一节中我们讲讲在mongo中如何创建索引       得到当前查询的索引信息      db.user.find(_id:12).explain();        cursor: basicCoursor 指的是没有索引  &
  • [硬件和系统]迎峰度夏 comsci 系统
      从这几天的气温来看，今年夏天的高温天气可能会维持在一个比较长的时间内    所以，从现在开始准备渡过炎热的夏天。。。。    每间房屋要有一个落地电风扇，一个空调(空调的功率和房间的面积有密切的关系)    坐的，躺的地方要有凉垫，床上要有凉席       电脑的机箱
  • 基于ThinkPHP开发的公司官网 cuiyadll 行业系统
    后端基于ThinkPHP，前端基于jQuery和BootstrapCo.MZ 企业系统 轻量级企业网站管理系统 运行环境:PHP5.3+, MySQL5.0 系统预览 系统下载：http://www.tecmz.com 预览地址：http://co.tecmz.com 各种设备自适应 响应式的网站设计能够对用户产生友好度，并且对于
  • Transaction and redelivery in JMS (JMS的事务和失败消息重发机制) darrenzhu jms事务承认MQacknowledge
    JMS Message Delivery Reliability and Acknowledgement Patterns http://wso2.com/library/articles/2013/01/jms-message-delivery-reliability-acknowledgement-patterns/ Transaction and redelivery in
  • Centos添加硬盘完全教程 dcj3sjt126com linuxcentoshardware
    Linux的硬盘识别: sda        表示第1块SCSI硬盘 hda       表示第1块IDE硬盘 scd0      表示第1个USB光驱 一般使用“fdisk -l”命
  • yii2 restful web服务路由 dcj3sjt126com PHPyii2
    路由 随着资源和控制器类准备，您可以使用URL如 http://localhost/index.php?r=user/create访问资源，类似于你可以用正常的Web应用程序做法。 在实践中，你通常要用美观的URL并采取有优势的HTTP动词。 例如，请求POST /users意味着访问user/create动作。 这可以很容易地通过配置urlManager应用程序组件来完成 如下所示
  • MongoDB查询(4)——游标和分页[八] eksliang mongodbMongoDB游标MongoDB深分页
    转载请出自出处：http://eksliang.iteye.com/blog/2177567 一、游标         数据库使用游标返回find的执行结果。客户端对游标的实现通常能够对最终结果进行有效控制，从shell中定义一个游标非常简单，就是将查询结果分配给一个变量（用var声明的变量就是局部变量），便创建了一个游标，如下所示： > var
  • Activity的四种启动模式和onNewIntent() gundumw100 android
    Android中Activity启动模式详解 　　在Android中每个界面都是一个Activity，切换界面操作其实是多个不同Activity之间的实例化操作。在Android中Activity的启动模式决定了Activity的启动运行方式。 　　Android总Activity的启动模式分为四种： Activity启动模式设置： <acti
  • 攻城狮送女友的CSS3生日蛋糕 ini htmlWebhtml5csscss3
    在线预览：http://keleyi.com/keleyi/phtml/html5/29.htm   代码如下：   <!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <title>攻城狮送女友的CSS3生日蛋糕-柯乐义<
  • 读源码学Servlet（1）GenericServlet 源码分析 jzinfo tomcatWebservlet网络应用网络协议
    Servlet API的核心就是javax.servlet.Servlet接口，所有的Servlet 类（抽象的或者自己写的）都必须实现这个接口。在Servlet接口中定义了5个方法，其中有3个方法是由Servlet 容器在Servlet的生命周期的不同阶段来调用的特定方法。     先看javax.servlet.servlet接口源码：  package
  • JAVA进阶：VO(DTO)与PO(DAO)之间的转换 snoopy7713 javaVOHibernatepo
      PO即 Persistence Object　　VO即 Value Object 　VO和PO的主要区别在于：　　VO是独立的Java Object。　　PO是由Hibernate纳入其实体容器（Entity Map）的对象，它代表了与数据库中某条记录对应的Hibernate实体，PO的变化在事务提交时将反应到实际数据库中。 　实际上，这个VO被用作Data Transfer
  • mongodb group by date 聚合查询日期 统计每天数据（信息量） qiaolevip 每天进步一点点学习永无止境mongodb纵观千象
    /* 1 */ { "_id" : ObjectId("557ac1e2153c43c320393d9d"), "msgType" : "text", "sendTime" : ISODate("2015-06-12T11:26:26.000Z")
  • java之18天 常用的类(一) Luob. MathDateSystemRuntimeRundom
    System类 import java.util.Properties; /** * System: * out:标准输出,默认是控制台 * in:标准输入,默认是键盘 * * 描述系统的一些信息 * 获取系统的属性信息:Properties getProperties(); * * * */ public class Sy
  • maven wuai maven
    1、安装maven：解压缩、添加M2_HOME、添加环境变量path 2、创建maven_home文件夹，创建项目mvn_ch01,在其下面建立src、pom.xml，在src下面简历main、test、main下面建立java文件夹 3、编写类，在java文件夹下面依照类的包逐层创建文件夹，将此类放入最后一级文件夹 4、进入mvn_ch01 4.1、mvn compile ,执行后会在
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他