sysbench和lua的简单研究
因为精力有限，所以先研究Mysql，等Mysql有些精通，再研究TiDB或其他开源数据库。
先从sysbench-0.4.12.14开始 - https://dev.mysql.com/downloads/benchmarks.html
create database sbtest;
grant all privileges on sbtest.* to test@‘localhost’ identified by ‘Welcome1+’;
采用–debug=on把“世界”变得更清澈
./sysbench --num-threads=1 --max-requests=2 --test=oltp --debug=on --mysql-table-engine=innodb --oltp-num-tables=1 --oltp-table-size=10 --mysql-user=test --mysql-password=Welcome1+ prepare
./sysbench --num-threads=1 --max-requests=2 --test=oltp --debug=on --mysql-table-engine=innodb --oltp-num-tables=1 --oltp-table-size=10 --mysql-user=test --mysql-password=Welcome1+ run
./sysbench --num-threads=1 --max-requests=2 --test=oltp --debug=on --mysql-table-engine=innodb --oltp-num-tables=1 --oltp-table-size=10 --mysql-user=test --mysql-password=Welcome1+ cleanup
表：
CREATE TABLE sbtest (
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT ‘0’,
c char(120) NOT NULL DEFAULT ‘’,
pad char(60) NOT NULL DEFAULT ‘’,
PRIMARY KEY (id),
KEY k (k)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=latin1；
压测SQL：
SELECT c from sbtest where id=?
SELECT c from sbtest where id in (?,?,?,?,?,?,?,?,?,?)
SELECT c from sbtest where id between ? and ?
SELECT SUM(K) from sbtest where id between ? and ?
SELECT c from sbtest where id between ? and ? order by c
SELECT DISTINCT c from sbtest where id between ? and ? order by c
UPDATE sbtest set k=k+1 where id=?
UPDATE sbtest set c=? where id=?
DELETE from sbtest where id=?
INSERT INTO sbtest values(?,0,’ ',‘aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeeyyyyyyyyyy’)
BEGIN
COMMIT
**Note：每次request会顺序执行以上SQL，而且UPDATE/DELETE/INSERT会使用同一个ID.
还有一个sysbench 1.1.0（基于LUA）- https://github.com/akopytov/sysbench
先简单学习LUA -
./third_party/luajit/bin/luajit-2.1.0-beta3
LuaJIT 2.1.0-beta3 – Copyright © 2005-2017 Mike Pall. http://luajit.org/
JIT: ON SSE2 SSE3 SSE4.1 fold cse dce fwd dse narrow loop abc sink fuse
print “Hello,world!”
Hello,world!
function create_a_counter()local count=0
return function()
count = count + 1
return count
end
end
ct = create_a_counter()
print(ct())
1
print(ct())
2
print(ct())
3
print(create_a_counter())
function: 0x408aa1f8
print(create_a_counter())
function: 0x408aa388
print(create_a_counter())
function: 0x408aa518
print((create_a_counter())())
1
print((create_a_counter())())
1
print((create_a_counter())())
在oltp_common.lua有参数说明：
sysbench.cmdline.options = {
table_size =
{“Number of rows per table”, 10000},
range_size =
{“Range size for range SELECT queries”, 100},
tables =
{“Number of tables”, 1},
…
…
}
由于oltp_common.lua的封装，oltp_read_write.lua极为精简，实际上执行的SQL同sysbench-0.4.12.14相同或者类似
function event()
if not sysbench.opt.skip_trx then
begin()
end
execute_point_selects()
if sysbench.opt.range_selects then
execute_simple_ranges()
execute_sum_ranges()
execute_order_ranges()
execute_distinct_ranges()
end
execute_index_updates()
execute_non_index_updates()
execute_delete_inserts()
if not sysbench.opt.skip_trx then
commit()
end
check_reconnect()
end
sysbench将luajit编译成library，然后Link入sysbench执行程序，从而能够执行LUA脚本。
LUAJIT_CFLAGS = -I$(abs_top_builddir)/third_party/luajit/inc
LUAJIT_LDFLAGS = -rdynamic
LUAJIT_LIBS = $(abs_top_builddir)/third_party/luajit/lib/libluajit-5.1.a -ldl
LUAJIT_DIR = third_party/luajit
如果想看到更多LUA细节，可以试试——
./sysbench/bin/sysbench ./sysbench/share/sysbench/oltp_read_write.lua --threads=2 --events=100000 --verbosity=5 --debug=on --luajit-cmd=v --mysql-user=test --mysql-password=Welcome1+ --tables=10 --table_size=100000 prepare
./sysbench/bin/sysbench ./sysbench/share/sysbench/oltp_read_write.lua --threads=2 --events=100000 --verbosity=5 --debug=on --luajit-cmd=v --mysql-user=test --mysql-password=Welcome1+ --tables=10 --table_size=100000 run
./sysbench/bin/sysbench ./sysbench/share/sysbench/oltp_read_write.lua --threads=2 --events=100000 --verbosity=5 --debug=on --luajit-cmd=v --mysql-user=test --mysql-password=Welcome1+ --tables=10 --table_size=100000 cleanup
用pstack可以看到两个lua thread执行，从sb_lua.c-》lj_BC_FUNCC-》db_driver.c-》drv_mysql.c-》mysql-5.7.19 client.c
Thread 3 (Thread 0x7f8340135700 (LWP 23471)):
#0 0x00007f833edcca9b in recv () from /lib64/libpthread.so.0
#1 0x00007f833f953560 in recv (__flags=0, __n=16384, __buf=0x7f8338004890, __fd=3) at /usr/include/bits/socket2.h:44
#2 inline_mysql_socket_recv (src_file=0x7f833fa36af0 “/export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/vio/viosocket.c”, src_line=123, flags=0, n=16384, buf=0x7f8338004890, mysql_socket=…) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/include/mysql/psi/mysql_socket.h:823
#3 vio_read (vio=vio@entry=0x7f8338004670, buf=0x7f8338004890 “6”, size=size@entry=16384) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/vio/viosocket.c:123
#4 0x00007f833f9535e1 in vio_read_buff (vio=0x7f8338004670, buf=0x7f83380088c0 “m\377\a”, size=4) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/vio/viosocket.c:166
#5 0x00007f833f930fa3 in net_read_raw_loop (net=net@entry=0x7f8338001da0, count=4) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:672
#6 0x00007f833f931247 in net_read_packet_header (net=0x7f8338001da0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:762
#7 net_read_packet (net=0x7f8338001da0, complen=0x7f8340134b40) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:822
#8 0x00007f833f931fdc in my_net_read (net=net@entry=0x7f8338001da0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:899
#9 0x00007f833f9261eb in cli_safe_read_with_ok (mysql=mysql@entry=0x7f8338001da0, parse_ok=parse_ok@entry=0 ‘\000’, is_data_packet=is_data_packet@entry=0x0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:1055
#10 0x00007f833f9264ef in cli_safe_read (mysql=mysql@entry=0x7f8338001da0, is_data_packet=is_data_packet@entry=0x0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:1188
#11 0x00007f833f9274e8 in cli_read_query_result (mysql=0x7f8338001da0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:5148
#12 0x00007f833f929496 in mysql_real_query (mysql=mysql@entry=0x7f8338001da0, query=query@entry=0x7f833e257010 “INSERT INTO sbtest1(k, c, pad) VALUES(5017955, ‘77685457002-60313448154-93610712648-61821086734-59286509531-41304833915-45555035587-24665129955-62001379076-91492887087’, '61168108036-68976681387-94857”…, length=length@entry=524140) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:5244
#13 0x000000000041dcc3 in mysql_drv_query (sb_conn=0x7f8338001010, query=0x7f833e257010 “INSERT INTO sbtest1(k, c, pad) VALUES(5017955, ‘77685457002-60313448154-93610712648-61821086734-59286509531-41304833915-45555035587-24665129955-62001379076-91492887087’, '61168108036-68976681387-94857”…, len=524140, rs=0x7f8338001038) at drv_mysql.c:972
#14 0x000000000040f333 in db_query (con=0x7f8338001010, query=, len=) at db_driver.c:572
#15 0x000000000040fa9b in db_bulk_do_insert (is_last=0, con=0x7f8338001010) at db_driver.c:925
#16 db_bulk_insert_next (con=0x7f8338001010, query=0x408e6ca0 “(5038109, ‘91286458069-48444577525-22754756125-55351938521-74574342350-37047882073-65015116514-10185773698-64551382738-43276849468’, ‘57820263199-28773002066-72191491800-83583176414-53296704425’)”, query_len=195) at db_driver.c:900
#17 0x000000007e45ffe0 in ?? ()
#18 0x000000000042fdb0 in ?? ()
#19 0x00989680408e6c38 in ?? ()
#20 0x00007f8338001010 in ?? ()
#21 0x4153380740000000 in ?? ()
#22 0x00000000408ed1b0 in ?? ()
#23 0x408ed148408e6b20 in ?? ()
#24 0x40389f78408ed060 in ?? ()
#25 0x00000000408e6b28 in ?? ()
#26 0x415321a200000000 in ?? ()
#27 0x4193de40403899f8 in ?? ()
#28 0x000000004193de48 in ?? ()
#29 0x4039c330408fb488 in ?? ()
#30 0x00000000408fb490 in ?? ()
#31 0x0000000000432237 in lj_BC_FUNCC ()
#32 0x000000000042167d in lua_pcall ()
#33 0x00000000004158a3 in call_custom_command (L=0x40384378) at sb_lua.c:946
#34 cmd_worker_thread (arg=) at sb_lua.c:978
#35 0x00007f833edc5e25 in start_thread () from /lib64/libpthread.so.0
#36 0x00007f833eaf334d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f8340124700 (LWP 23472)):
#0 0x00007f833edcca9b in recv () from /lib64/libpthread.so.0
#1 0x00007f833f953560 in recv (__flags=0, __n=16384, __buf=0x7f8330003500, __fd=4) at /usr/include/bits/socket2.h:44
#2 inline_mysql_socket_recv (src_file=0x7f833fa36af0 “/export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/vio/viosocket.c”, src_line=123, flags=0, n=16384, buf=0x7f8330003500, mysql_socket=…) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/include/mysql/psi/mysql_socket.h:823
#3 vio_read (vio=vio@entry=0x7f83300032e0, buf=0x7f8330003500 “6”, size=size@entry=16384) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/vio/viosocket.c:123
#4 0x00007f833f9535e1 in vio_read_buff (vio=0x7f83300032e0, buf=0x7f8330007530 “m\377\a”, size=4) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/vio/viosocket.c:166
#5 0x00007f833f930fa3 in net_read_raw_loop (net=net@entry=0x7f8330000a10, count=4) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:672
#6 0x00007f833f931247 in net_read_packet_header (net=0x7f8330000a10) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:762
#7 net_read_packet (net=0x7f8330000a10, complen=0x7f8340123b40) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:822
#8 0x00007f833f931fdc in my_net_read (net=net@entry=0x7f8330000a10) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql/net_serv.cc:899
#9 0x00007f833f9261eb in cli_safe_read_with_ok (mysql=mysql@entry=0x7f8330000a10, parse_ok=parse_ok@entry=0 ‘\000’, is_data_packet=is_data_packet@entry=0x0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:1055
#10 0x00007f833f9264ef in cli_safe_read (mysql=mysql@entry=0x7f8330000a10, is_data_packet=is_data_packet@entry=0x0) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:1188
#11 0x00007f833f9274e8 in cli_read_query_result (mysql=0x7f8330000a10) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:5148
#12 0x00007f833f929496 in mysql_real_query (mysql=mysql@entry=0x7f8330000a10, query=query@entry=0x7f833ff80010 “INSERT INTO sbtest2(k, c, pad) VALUES(5021609, ‘72949005509-39489782487-32241234017-69795429734-17763294487-03262219785-64699673850-77482937981-05256446242-95097452333’, '38662339137-58629628167-30238”…, length=length@entry=524140) at /export/home2/pb2/build/sb_1-23948522-1498141788.79/rpm/BUILD/mysql-5.7.19/mysql-5.7.19/sql-common/client.c:5244
#13 0x000000000041dcc3 in mysql_drv_query (sb_conn=0x7f8330000940, query=0x7f833ff80010 “INSERT INTO sbtest2(k, c, pad) VALUES(5021609, ‘72949005509-39489782487-32241234017-69795429734-17763294487-03262219785-64699673850-77482937981-05256446242-95097452333’, '38662339137-58629628167-30238”…, len=524140, rs=0x7f8330000968) at drv_mysql.c:972
#14 0x000000000040f333 in db_query (con=0x7f8330000940, query=, len=) at db_driver.c:572
#15 0x000000000040fa9b in db_bulk_do_insert (is_last=0, con=0x7f8330000940) at db_driver.c:925
#16 db_bulk_insert_next (con=0x7f8330000940, query=0x41a516b8 “(5043090, ‘45151831184-17233725886-83677336142-18842943783-20928163012-77222558892-59376813101-18044662735-93821158803-47895647288’, ‘33715251347-27068205446-76113781579-57542690146-52207909892’)”, query_len=195) at db_driver.c:900
#17 0x000000007840ffe0 in ?? ()
#18 0x000000000042fdb0 in ?? ()
#19 0x0098968041a51650 in ?? ()
#20 0x00007f8330000940 in ?? ()
#21 0x41533ce480000000 in ?? ()
#22 0x0000000041a51608 in ?? ()
#23 0x41a513a041a514e0 in ?? ()
#24 0x4034bf7841a512b8 in ?? ()
#25 0x0000000041a514e8 in ?? ()
#26 0x41531c76c0000000 in ?? ()
#27 0x41a5e1b04034b9f8 in ?? ()
#28 0x0000000041a5e1b8 in ?? ()
#29 0x4035e3304083f0c0 in ?? ()
#30 0x000000004083f0c8 in ?? ()
#31 0x0000000000432237 in lj_BC_FUNCC ()
#32 0x000000000042167d in lua_pcall ()
#33 0x00000000004158a3 in call_custom_command (L=0x40346378) at sb_lua.c:946
#34 cmd_worker_thread (arg=) at sb_lua.c:978
#35 0x00007f833edc5e25 in start_thread () from /lib64/libpthread.so.0
#36 0x00007f833eaf334d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f8340107840 (LWP 23470)):
#0 0x00007f833edc6f57 in pthread_join () from /lib64/libpthread.so.0
#1 0x0000000000413f29 in sb_thread_join (retval=0x0, thread=) at sb_thread.c:146
#2 sb_thread_join_workers () at sb_thread.c:192
#3 0x00000000004166a3 in sb_lua_call_custom_command (name=) at sb_lua.c:998
#4 0x000000000040966a in main (argc=12, argv=) at sysbench.c:1528
以上pstack存在c先调用lua脚本，然后lua脚本调用c函数db_bulk_insert_next，机制是FFI Library - http://luajit.org/ext_ffi.html
这里细节复杂，时间有限，我懒得深入了，只简单粘贴几句英文——
The FFI library allows calling external C functions and using C data structures from pure Lua code.
The FFI library largely obviates the need to write tedious manual Lua/C bindings in C. No need to learn a separate binding language — it parses plain C declarations! These can be cut-n-pasted from C header files or reference manuals. It’s up to the task of binding large libraries without the need for dealing with fragile binding generators.
The FFI library is tightly integrated into LuaJIT (it’s not available as a separate module). The code generated by the JIT-compiler for accesses to C data structures from Lua code is on par with the code a C compiler would generate. Calls to C functions can be inlined in JIT-compiled code, unlike calls to functions bound via the classic Lua/C API.
LUA实现了一套语法，与c高度融合，有许多高级功能（比如并发），又有脚本的方便性和灵活性，还能Link入程序，性能又非常好，程序还非常精简，怪不得LUA才如此流行。所以，我为LUA高度点赞。