【原创】sqlite3数据库SQL error: database disk image is malformed问题探究（1）
【原创】sqlite3数据库SQL error: database disk image is malformed问题探究
Author: chad
Mail: [email protected]
本文可以自由转载，但转载请务必注明出处以及本声明信息。
《附件内为有“database disk image is malformed”错误的实验用数据库》
如下所示，实际使用过程中发现有的数据库问题千奇百怪，常见有如下现象：
- 数据库文件部分损坏，表现例如：4M的数据库只剩下1M大小，数据库中大部分表已经不存在，但是一部分表扔可正常访问。
- 数据库文件大小没有明显变化，但是如果使用PRAGMA integrity_check;命令进行检查会检查出问题。
- 数据库文件大小没有明显变化，并且使用PRAGMA integrity_check;命令检测ok，但是更新损坏的表时出现错误，查询等其他操作不会出错。如下例所示：
[root@Chad: /home]#sqlite3 /tiandao/data/terminal.db
SQLite version 3.6.1
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> PRAGMA integrity_check;
ok
sqlite> .tables
CurveDataTable event task1data
DayFreezeTable event_tmp task1mark
MeterEventData groupparam task2data
MonthFreezeTable keyuser task2mark
SysMaintenance localparam terminalactive
TerRMStateInfoTable logininfo terminalcascade
capacitor measuringcufe terminalgroup
carrier measuringlimit terminalparam
comportinfo measuringparam terminalpulse
data1config measuringpoint terminalrate
data1value mlogininfo terminalvi
data2config rate totalgroup
data2value readingmeter vpac
dcanalog readtime vpacvalue
dlogininfo system
sqlite> select * from DayFreezeTable where id=222;
222|0|||||||||||||||||||0
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3331;
SQL error: database disk image is malformed
sqlite> .exit如上所示，更新某一条记录时才会提示数据结构损坏。但是插入新的数据条目却正常，如下所示：
sqlite> .schema DayFreezeTable
CREATE TABLE DayFreezeTable(
id INTEGER PRIMARY KEY,
TimeScale date,
ForwardPowerInd blob,
ReactivePowerInd_1 blob,
ReversePowerInd blob,
ReactivePowerInd_2 blob,
OneQuadrantReactivePower blob,
TwoQuadrantReactivePower blob,
ThreeQuadrantReactivePower blob,
FourQuadrantReactivePower blob,
ForwardPower blob,
ReactivePower_1 blob,
ReversePower blob,
ReactivePower_2 blob,
ForwardActiveDemand blob,
ReverseActiveDemand blob,
ForwardReactiveDemand blob,
ReverseReactiveDemand blob,
PhasePowerTime blob,
PhaseDemandTime blob , cmdbitmap integer default(0));
sqlite> insert into DayFreezeTable values(1111111,22);
SQL error: table DayFreezeTable has 21 columns but 2 values were supplied
sqlite> select * from DayFreezeTable limit 2;
1|0|
.L*!*|
铑顋
|
|||||||||||||||0
2|0|||||||||||||||||||0
sqlite> insert into DayFreezeTable values(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
sqlite> select * from DayFreezeTable limit 2;
0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0
1|0|
.L*!*|
铑顋
|
|||||||||||||||0
sqlite> update DayFreezeTable set TimeScale=1 where id=0;
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3331;
SQL error: database disk image is malformed再一步试验如下：
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=331;
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=332;
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>100;
SQL error: database disk image is malformed
sqlite> select * from DayFreezeTable where TimeScale<>'2013-03-03' limit 3;
0|1|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0
1|0|
.L*!*|
铑顋
|
|||||||||||||||0
2|0|||||||||||||||||||0
sqlite> select * from DayFreezeTable where TimeScale<>'2013-03-03' and id > 331 limit 3;
332|2013-03-02|||||||||||||||||||0
333|0|||||||||||||||||||0
334|0|||||||||||||||||||0
sqlite> select * from DayFreezeTable where TimeScale<>'2013-03-03' and id > 100 limit 3;
101|0| )铑铑铑铑铑铑铑铑铑铑铑铑顋 )铑铑铑铑铑铑铑铑铑铑| )铑铑铑铑铑铑铑铑铑铑铑铑顋 )铑铑铑铑铑铑铑铑铑铑|||||||||||||||0
102|0| )铑铑铑铑铑铑铑铑铑铑铑铑顋 )铑铑铑铑铑铑铑铑铑铑| )铑铑铑铑铑铑铑铑铑铑铑铑顋 )铑铑铑铑铑铑铑铑铑铑|||||||||||||||0
103|0| )铑铑铑铑铑铑铑铑铑铑铑铑顋 )铑铑铑铑铑铑铑铑铑铑| )铑铑铑铑铑铑铑铑铑铑铑铑顋 )铑铑铑铑铑铑铑铑铑铑|||||||||||||||0上面的试验说明了2个问题：
UPDATE DayFreezeTable 中的 id=331; id=332; 以及很多个条目时，数据库没有报错，说明即便出现“database disk image is malformed”错误，也并不意味着整个数据库或数据表都有问题，有问题的仅仅是个别条目！
使用UPDATE DayFreezeTable SET TimeScale=’2013-03-02’ where id>100;进行测试时，该命令是执行很长一段时间后才报错的，报错后语句执行终止，然后使用select命令查询发现数据库并没有任何改变，这说明数据库有回滚机制，只有命令全部正常才能提交结果。
上面的试验中说明第3331行数据存在问题，如下：
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3331;
SQL error: database disk image is malformed再次基础上继续试验如下：
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>100 and id<3331;
sqlite> select * from DayFreezeTable where id > 3300 limit 3;
3301|2013-03-02|F|F|F|F|||||||||F|F|||||0
3302|2013-03-02|V|V|W|W|||||||||W|W|||||0
3303|2013-03-02|F|F|F|F|||||||||F|F|||||0
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>3331 and id<3500;
SQL error: database disk image is malformed
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>3332 and id<3500;
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3332;
SQL error: database disk image is malformed
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3333;以上试验更进一步的证实了，即便出现database disk image is malformed错误，可能只是个别条目问题，本数据库的问题目前看来仅存在与id=3331与id=3332两个条目。
继续试验如下：
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3332;
SQL error: database disk image is malformed
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id=3331;
SQL error: database disk image is malformed
sqlite> select * from DayFreezeTable where id>3330 limit 5;
SQL error: database disk image is malformed
sqlite> select * from DayFreezeTable where id=3331;
SQL error: database disk image is malformed
sqlite> select * from DayFreezeTable where id=3332;
SQL error: database disk image is malformed
sqlite> select * from DayFreezeTable where id=3333;
3333|201-01-17||||||||||||||||||
sqlite> select count(*) from DayFreezeTable;
63270
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>3500;
sqlite> select * from DayFreezeTable where id>63266;
63267|2013-03-02|||||||||||||||||||0
63268|2013-03-02|||||||||||||||||||0
63269|2013-03-02|||||||||||||||||||0
63270|2013-03-02|||||||||||||||||||0
sqlite> select count(*) from MonthFreezeTable;
24491
sqlite> UPDATE MonthFreezeTable SET TimeScale='2013-03-02' where id>0;
sqlite> UPDATE CurveFreezeTable SET TimeScale='2013-03-02' where id>0;
SQL error: no such table: CurveFreezeTable
sqlite> UPDATE CurveDataTable SET TimeScale='2013-03-02' where id>0;
sqlite> alter table DayFreezeTable add column cmdbitmap integer default(0);
sqlite> .schema DayFreezeTable
CREATE TABLE DayFreezeTable(
id INTEGER PRIMARY KEY,
TimeScale date,
ForwardPowerInd blob,
ReactivePowerInd_1 blob,
ReversePowerInd blob,
ReactivePowerInd_2 blob,
OneQuadrantReactivePower blob,
TwoQuadrantReactivePower blob,
ThreeQuadrantReactivePower blob,
FourQuadrantReactivePower blob,
ForwardPower blob,
ReactivePower_1 blob,
ReversePower blob,
ReactivePower_2 blob,
ForwardActiveDemand blob,
ReverseActiveDemand blob,
ForwardReactiveDemand blob,
ReverseReactiveDemand blob,
PhasePowerTime blob,
PhaseDemandTime blob , cmdbitmap integer default(0));如上所示，证实本数据库确实只有DayFreezeTable 表中的两个条目出现问题！而其他常用的数据表并不存在问题。同时，上述实验还揭示了一个重要问题，即出现“database disk image is malformed”类型错误的数据条目查询也会报错！但执行select count(*)种查询表整体信息或alter table这种改变表整体结构的操作没有问题。只有操作问题条目时才出错，如下： 
对本部分测试再进行细化，试验如下：
测试id=3332项：
sqlite> select id,TimeScale,ForwardPowerInd from DayFreezeTable where id=3332;
3332|2013-01-16|
sqlite> select id,TimeScale,ForwardPowerInd,ReactivePowerInd_1 from DayFreezeTable where id=3332;
3332|2013-01-16||0
。。。。。。
sqlite> select id,TimeScale,PhasePowerTime from DayFreezeTable where id=3332;
3332|2013-01-16|
sqlite> select id,TimeScale,PhaseDemandTime from DayFreezeTable where id=3332;
SQL error: database disk image is malformed如上所示，出错的仅仅是最后一列，对前面的列进行查询是不会报错的。那么id=3331是什么情况呢？试验如下：
sqlite> select id,TimeScale,PhasePowerTime from DayFreezeTable where id=3331;
3331|201-01-16|
sqlite> select id,TimeScale,PhaseDemandTime from DayFreezeTable where id=3331;
SQL error: database disk image is malformed出错的同样是最后一列！但是不是所有出错的都只可能是从后面开始呢？在《sqlite3数据库SQL error: database disk image is malformed问题探究<2>》中，有一个损坏更严重的数据库，此处先将《2》中数据库本部分相关的试验总结如下：
sqlite> select id,TimeScale,PhasePowerTime from DayFreezeTable where id=19164;
SQL error: database disk image is malformed
。。。。
sqlite> select id,TimeScale from DayFreezeTable where id=19164;
SQL error: database disk image is malformed
sqlite> select id from DayFreezeTable where id=19164;
SQL error: database disk image is malformed以上是试验摘要，结果是id=19164行的所有数据均出错！并且，经测试发现同一区段的其他坏行现象一致。对其他损坏段的坏行进行测试，发现也是全部损坏。并且，测试中发现《2》中数据库的损坏行多达连续数百行！
修复方法如下所示：
sqlite> delete from DayFreezeTable where id=3331;
sqlite> delete from DayFreezeTable where id=3332;
sqlite> insert into DayFreezeTable values(3331,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,>
sqlite> insert into DayFreezeTable values(3332,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,>
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>3330 and id<3333;
sqlite> select * from DayFreezeTable where id > 3330 limit 5;
3331|2013-03-02|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0
3332|2013-03-02|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0
3333|2013-03-02|||||||||||||||||||0
3334|2013-03-02|||||||||||||||||||0
3335|2013-03-02|||||||||||||||||||0如上所示，修复方法就是将出错的条码删除，然后重建即可！
总结，通过上面的实验证实如下结论：
即便出现“database disk image is malformed”错误，也并不意味着整个数据库或数据表都有问题，有问题的可能仅仅是个别条目！
如果某些数据库条目出现“database disk image is malformed”错误，那么这些条目不论执行update动作或“select ”（注意：一定要是“select ”，因为如果select colname这种只查询一列的情况可能正确执行！）动作都会失败！但执行“select count(*)”这种查询表整体信息或“PRAGMA integrity_check;”这种检测整个数据库以及alter table这种改变表整体结构的操作没有问题。
出现“database disk image is malformed”错误的处理方法如下：
sqlite> delete from DayFreezeTable where id=3331;
sqlite> delete from DayFreezeTable where id=3332;
sqlite> insert into DayFreezeTable values(3331,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,>
sqlite> insert into DayFreezeTable values(3332,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,>
sqlite> UPDATE DayFreezeTable SET TimeScale='2013-03-02' where id>3330 and id<3333;
sqlite> select * from DayFreezeTable where id > 3330 limit 5;
3331|2013-03-02|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0
3332|2013-03-02|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0
3333|2013-03-02|||||||||||||||||||0
3334|2013-03-02|||||||||||||||||||0
3335|2013-03-02|||||||||||||||||||0如上所示，修复方法就是将出错的条目删除，然后重建即可！
待决问题：
- 嵌入式文件系统如jffs2，yaffs2的负载均衡策略是什么？
- 与sqlite3数据库读写频繁出现的文件错误有什么联系？