sql like 优化

阅读更多

A:
最多使用ffs，其他没有什么好方法！
两边都有%，就不会走index了。。
如果表不是很大，可以考虑keep之。。

B:
这个问题首先是在TAOBAO DBA的BLOG上看到丹臣写的关于Like和INSTR的性能问题。不过他只是给出了结果。我对这个函数性能感到有趣，之前一直没有关注过，遂自己详细测试了下。

Oracle 9208:

SQL> select count(*) from item;

COUNT(*)

----------

2781806

SQL> select count(*) from item where item like '%A0';

COUNT(*)

----------

9036

Elapsed: 00:00:04.03

Execution Plan

----------------------------------------------------------

0 SELECT STATEMENT Optimizer=CHOOSE (Cost=1134 Card=1 Bytes=16

)

1 0 SORT (AGGREGATE)

2 1 INDEX (FAST FULL SCAN) OF 'PK_ITEM' (UNIQUE) (Cost=1134

Card=138698 Bytes=2219168)

Statistics

----------------------------------------------------------

0 recursive calls

0 db block gets

10369 consistent gets

0 physical reads

0 redo size

519 bytes sent via SQL*Net to client

656 bytes received via SQL*Net from client

2 SQL*Net roundtrips to/from client

0 sorts (memory)

0 sorts (disk)

1 rows processed

可以看到走了FFS, 10396 Consistent Gets, Elapsed tail=4.03

再看SUBSTR

SQL> select count(*) from item where substr(item,-2)='A0';

COUNT(*)

----------

9036

Elapsed: 00:00:00.84

Execution Plan

----------------------------------------------------------

0 SELECT STATEMENT Optimizer=CHOOSE (Cost=1134 Card=1 Bytes=16

)

1 0 SORT (AGGREGATE)

2 1 INDEX (FAST FULL SCAN) OF 'PK_ITEM' (UNIQUE) (Cost=1134

Card=27740 Bytes=443840)

Statistics

----------------------------------------------------------

0 recursive calls

0 db block gets

10369 consistent gets

0 physical reads

0 redo size

519 bytes sent via SQL*Net to client

656 bytes received via SQL*Net from client

2 SQL*Net roundtrips to/from client

0 sorts (memory)

0 sorts (disk)

1 rows processed

可以看到PLAN一样，Consistent Gets一样，Elapsed tail=0.84

SUBSTR的耗时只有LIKE的21%左右，鉴于PLAN和Consistent Gets一致，所以可以认为是SUBSTR的CPU Cost要小于LIKE。

NOT LIKE的情况和LIKE相似。

INSTR的情况也如SUBSTR

确实如丹臣所言，Oracle的函数有相当的优化。

但9i的Optimizer在计算Cost的时候是以IO为准，那么在以CPU为准的10G上呢，Cost会有什么差别？

下面在10203上测试

SQL> select count(*) from item where item like '%A0';

COUNT(*)

----------

9104

Elapsed: 00:00:03.03

Execution Plan

----------------------------------------------------------

Plan hash value: 642095792

--------------------------------------------------------------------------------

-

| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time

|

--------------------------------------------------------------------------------

-

| 0 | SELECT STATEMENT | | 1 | 15 | 2211 (3)| 00:00:27

|

| 1 | SORT AGGREGATE | | 1 | 15 | |

|

|* 2 | INDEX FAST FULL SCAN| PK_ITEM | 139K| 2037K| 2211 (3)| 00:00:27

|

--------------------------------------------------------------------------------

-

Predicate Information (identified by operation id):

---------------------------------------------------

2 - filter("ITEM" LIKE '%A0')

Statistics

----------------------------------------------------------

0 recursive calls

0 db block gets

10117 consistent gets

0 physical reads

0 redo size

516 bytes sent via SQL*Net to client

492 bytes received via SQL*Net from client

2 SQL*Net roundtrips to/from client

0 sorts (memory)

0 sorts (disk)

1 rows processed

SQL> select count(*) from item where substr(item,-2)='A0';

COUNT(*)

----------

9104

Elapsed: 00:00:01.24

Execution Plan

----------------------------------------------------------

Plan hash value: 642095792

--------------------------------------------------------------------------------

-

| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time

|

----------------------------------------------------------------------

-

| 0 | SELECT STATEMENT | | 1 | 15 | 2220 (3)| 00:00:27

|

| 1 | SORT AGGREGATE | | 1 | 15 | |

|

|* 2 | INDEX FAST FULL SCAN| PK_ITEM | 27819 | 407K| 2220 (3)| 00:00:27

|

--------------------------------------------------------------------------------

-

Predicate Information (identified by operation id):

---------------------------------------------------

2 - filter(SUBSTR("ITEM",-2)='A0')

Statistics

----------------------------------------------------------

0 recursive calls

0 db block gets

10117 consistent gets

0 physical reads

0 redo size

516 bytes sent via SQL*Net to client

492 bytes received via SQL*Net from client

2 SQL*Net roundtrips to/from client

0 sorts (memory)

0 sorts (disk)

1 rows processed

结果是SUBSTR-2220-elapsed 1.24，Like-2211-elapsed 3.03。

显然SUBSTR在CPU占用上要高那么一点， 但是其相对于Like仅 41% 的耗时，使得它完全可以取代Like在查询以XX结尾的SQL中的地位。

BTW，我在一台Idle Server上的测试表明，使用SUBSTR不仅在耗时上缩短，而且CPU使用率也较LIKE低(9.5%/12.5%)

类似的测试表明INSTR相对LIKE ‘%XX%’的优势。

（函数取代LIKE ‘XX%’就别想了，一个Index Range Scan相对FFS的优势太大了）


C:
SQL> select count(*) from t;

  COUNT(*)
----------
     51838
SQL> insert /*+append*/ into t select * from t;
已创建51838行。
SQL> commit;
提交完成。
SQL> insert /*+append*/ into t select * from t;
已创建103676行。
SQL> commit;
提交完成。
SQL> insert /*+append*/ into t select * from t;
已创建207352行。
SQL> commit;
提交完成。
SQL> insert /*+append*/ into t select * from t;
已创建414704行。
SQL> commit;
提交完成。
SQL> select count(*) from t;

  COUNT(*)
----------
    829408
SQL> set autot traceonly exp stat
SQL> select * from t where object_type like '%TYPE%';

已选择34064行。

执行计划
----------------------------------------------------------
Plan hash value: 1601196873
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |  2592 |   235K|   160   (2)| 00:00:02 |
|*  1 |  TABLE ACCESS FULL| T    |  2592 |   235K|   160   (2)| 00:00:02 |
--------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_TYPE" LIKE '%TYPE%')

统计信息
----------------------------------------------------------
          0  recursive calls
          0  db block gets
      13623  consistent gets
      11364  physical reads
          0  redo size
    1466572  bytes sent via SQL*Net to client
      25370  bytes received via SQL*Net from client
       2272  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      34064  rows processed
SQL> SQL> select /*+index(t,t_idx)*/ * from t where object_type like '%TYPE%';

已选择34064行。

执行计划
----------------------------------------------------------
Plan hash value: 3778778741
-------------------------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |  2592 |   235K|   260   (2)| 00:00:04 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T     |  2592 |   235K|   260   (2)| 00:00:04 |
|*  2 |   INDEX FULL SCAN           | T_IDX |  2592 |       |   146   (2)| 00:00:02 |
-------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("OBJECT_TYPE" LIKE '%TYPE%')

统计信息
----------------------------------------------------------
          1  recursive calls
          0  db block gets
      14131  consistent gets
       1982  physical reads
     290188  redo size
    1477872  bytes sent via SQL*Net to client
      25370  bytes received via SQL*Net from client
       2272  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      34064  rows processed
SQL> SQL>
SQL>
SQL>
SQL> select * from t where instr(object_type,'TYPE')>0;

已选择34064行。

执行计划
----------------------------------------------------------
Plan hash value: 1601196873
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |  2592 |   235K|   161   (3)| 00:00:02 |
|*  1 |  TABLE ACCESS FULL| T    |  2592 |   235K|   161   (3)| 00:00:02 |
--------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(INSTR("OBJECT_TYPE",'TYPE')>0)

统计信息
----------------------------------------------------------
          0  recursive calls
          0  db block gets
      13623  consistent gets
       9952  physical reads
          0  redo size
    1466572  bytes sent via SQL*Net to client
      25370  bytes received via SQL*Net from client
       2272  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      34064  rows processed
SQL> SQL> SELECT * from t where rowid in(select /*+index_ffs(t,t_idx)*/ rowid from t where object_type like '%TYPE%');
已选择34064行。

执行计划
----------------------------------------------------------
Plan hash value: 628352769
-------------------------------------------------------------------------------
| Id  | Operation             | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT      |       |  2592 |   288K|   197   (4)| 00:00:03 |
|*  1 |  HASH JOIN            |       |  2592 |   288K|   197   (4)| 00:00:03 |
|*  2 |   INDEX FAST FULL SCAN| T_IDX |  2592 | 54432 |    35   (6)| 00:00:01 |
|   3 |   TABLE ACCESS FULL   | T     | 51838 |  4707K|   161   (3)| 00:00:02 |
-------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access(ROWID=ROWID)
   2 - filter("OBJECT_TYPE" LIKE '%TYPE%')

统计信息
----------------------------------------------------------
          1  recursive calls
          0  db block gets
      17559  consistent gets
       9991  physical reads
          0  redo size
    1466572  bytes sent via SQL*Net to client
      25370  bytes received via SQL*Net from client
       2272  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      34064  rows processed



小结：

   1，目前看instr比rowid及全表报描的性能更高及index_ffs更高

    2,index_ffs虽强制走了索引，但成本最高

D:
free介绍的是一种方法；RudolfLu曾经也介绍过一种方法，就是根据搜索字段+rowid建立一个物理表，对这个物理表进行全表，相较ffs会更快一些，不过需要定期刷新物理表，适用于变化较少，查询要求实时性要求不高的环境。

其实也可以用Oracle的全文检索技术，可能会有更好的结果，看我以下的一个例子：

[php]

1.这是模糊查询的结果
SQL> select count(*) from jivemessage where subject like '%abc%';

  COUNT(*)
----------
        13

Elapsed: 00:00:15.31

2.这是全文检索的结果
SQL> select count(*) from jivemessage where contains(subject,'abc')>0;

  COUNT(*)
----------
        25

Elapsed: 00:00:00.03

3.模糊查询的内容
SQL> select subject from jivemessage where subject like '%abc%';

SUBJECT
--------------------------------------------------------------------------------
abc
测试发贴""abc
abc
Re: abc
Re: abc
Re: abc

初夜abc
abcd
abc
abc

SUBJECT
--------------------------------------------------------------------------------
哈哇abc
http;//baoxing.168abc.com

13 rows selected.

Elapsed: 00:00:03.29

4.也许这是我们更想要的结果
SQL> select subject from jivemessage where contains(subject,'abc')>0;

SUBJECT
--------------------------------------------------------------------------------
【游戏】 把你的名字的首字母用智能ABC打出，看能出来什么？
游戏——把你的名字首字母用智能ABC打出来
智能ABC暗藏杀机
ABC
ABC
ABC
ABC
ABC
ＡＢＣ
哈哇abc
abc

SUBJECT
--------------------------------------------------------------------------------
abc
ABC
振奋爱的激情方案ABC
智能ABC的错吗？
ABC全选
瓜果美容ABC
经典英文歌曲ABC，不好你拿版砖砍我，好就回帖顶一下！
初夜abc
Re: abc
Re: abc
Re: abc

SUBJECT
--------------------------------------------------------------------------------
abc
测试发贴""abc
abc

25 rows selected.

Elapsed: 00:00:00.04

D:
select * from foo where rowid in (select /*+index_ffs(foo ind_name) */rowid from foo where name like '%ddd%' )

e:
SELECT
       org.*
  FROM test_ffs org, (SELECT /*+no_merge index_ffs(test_ffs  object_name ) */
                             ROWID AS r, object_name
                        FROM test_ffs
                       WHERE object_name LIKE '%A%' ) tmp
WHERE org.ROWID = tmp.r ;
INDEX_FFS(table  index_name)所查询的列必须全部被索引才可使用


f:
1。eygle 的 全文检索法：
对于用于条件的column length 占TABLE的record length的大部分的，速度应该是最快的。如果用于条件的column length 占TABLE的record length的比例并不是很大，那样优势就不明显，毕竟要多安装和管理一个组件。

2。RudolfLu的新表法
和上面对应的是，如果用于条件的column length 占TABLE的record length的比例并不是很大，比较好用，如果用于条件的column length 占TABLE的record length的大部分的话，那就没什么用了，另外，要多维护一个table,还要考虑两个表同步的问题，不建议使用。

3。我的index_ffs + rowid 法
主要优劣势和 RudolfLu的新表法 接近，但是index可以自动更新。而且，在某种环境里，可能已经有相关的index,可能并不需要增加新的index.
另外，没做过测试，不知道相同数据量的index_ffs  和 FTS 性能区别。


g:
1、尽量不要使用 like '%..%'

2、对于 like '..%..' (不以 % 开头)，Oracle可以应用 colunm上的index

3、对于 like '%...' 的 (不以 % 结尾)，可以利用 reverse + function index 的形式，变化成 like '..%' 代码



建测试表和Index。

注意：重点在于带reverse的function index。同时，一定要使用CBO才行......



SQL> select reverse('123') from dual;

REVERSE('123')

--------------------------------

321

1 row selected.



SQL> create table test_like as select object_id,object_name from dba_objects;

Table created.



SQL> create index test_like__name on test_like(object_name);

Index created.



SQL> create index test_like__name_reverse on test_like(reverse(object_name));

Index created.



SQL> analyze table test_like compute statistics for table for all indexes;

Table analyzed.



SQL> set autot trace



--常量开头的like , 会利用index ，没问题......

SQL> select * from test_like where object_name like AS%';

Execution Plan

----------------------------------------------------------

0 SELECT STATEMENT Optimizer=CHOOSE (Cost=2 Card=655 Bytes=15720)

1 0 TABLE ACCESS (BY INDEX ROWID) OF 'TEST_LIKE' (Cost=2 Card=655Bytes=15720)

2 1 INDEX (RANGE SCAN) OF 'TEST_LIKE__NAME' (NON-UNIQUE) (Cost=2 Card=118)



-- 开头和结尾都是%,对不起，很难优化



SQL> select * from test_like where object_name like '%%';



Execution Plan

----------------------------------------------------------

0 SELECT STATEMENT Optimizer=CHOOSE (Cost=6 Card=655 Bytes=15720)

1 0 TABLE ACCESS (FULL) OF 'TEST_LIKE' (Cost=6 Card=655 ytes=15720)



-- 以常量结束，直接写的时候是不能应用index的

SQL> select * from test_like where object_name like '%S';

Execution Plan

----------------------------------------------------------

0 SELECT STATEMENT Optimizer=CHOOSE (Cost=6 Card=655 Bytes=15720)

1 0 TABLE ACCESS (FULL) OF 'TEST_LIKE' (Cost=6 Card=655 Bytes=15720)



--'以常量结束的，加个reverse 函数，又可以用上index了'

SQL> select * from test_like where reverse(object_name)like reverse('%AS');

Execution Plan

----------------------------------------------------------

0 SELECT STATEMENT Optimizer=CHOOSE (Cost=2 Card=655 Bytes=15720)

1 0 TABLE ACCESS (BY INDEX ROWID) OF 'TEST_LIKE' (Cost=2 Card=655 Bytes=15720)

2 1 INDEX (RANGE SCAN) OF 'TEST_LIKE__NAME_REVERSE' (NON-UNIQUE) (Cost=2 Card=118)