oracle 10gORA-07445 RAC环境下一个案例
oracle 10gORA-07445
1. 数据库使用的软件及其版本
| 序号 |
名称 |
内容 |
| 1 |
操作系统 |
CentOS release 5.6 (Final) x86-64 |
| 2 |
数据库版本 |
Oracle Database 10g Release 2 |
| 3 |
数据库安装包： 补丁包： |
10201_database_linux_x86_64.cpio.gz p6810189_10204_Linux-x86-64.zip |
| 4 |
OEMbug解决补丁 |
p8350262_10204_Generic.zip |
2. 数据库结构
3. 现象
现在出现的报表系统goldengate同步进程异常中断，还有修改部分product表内容rollback。数据库后台报严重的错误。
主要排查以下错误日志：tomcat日志、数据库alter日志、数据表日志错误、数据库转储文件、goldengate进程、goldengate同步日志。
3.1 tomcat日志错误
jdbc版本：ojdbc14-10.2.0.4.0.jar
| 2012-05-14 16:15:36,755 [http-8083-3] ERROR net.sf.hibernate.transaction.JDBCTransaction - Rollback failed java.sql.SQLException: No more data to read from socket at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14.jar:Oracle JDBC Driver version - "10.2.0.4.0"] at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) ~[ojdbc14.jar:Oracle JDBC Driver version - "10.2.0.4.0"] |
3.2 数据库alter日志
| Mon May 14 14:37:33 2012 Errors in file /oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc: ORA-07445: exception encountered: core dump [_intel_fast_memcpy.A()+10] [SIGSEGV] [Address not mapped to object] [0x2B09D44BE000] [] [] Mon May 14 14:48:14 2012 Process 0x0x2d8367dc0 appears to be hung while dumping Current time = 485605312, process death time = 485541208 interval = 60000 |
从上面的日志可以看出进程hung住，让我查看/oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc转存文件
3.3 查看转存文件
/oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc转存文件比较长，大概有5M，我们就看关键部分
| Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x2b09d44be000, PC: [0x2b09d26ee608, _intel_fast_memcpy.A()+10] *** 2012-05-14 14:37:33.145 ksedmp: internal or fatal error ORA-07445: exception encountered: core dump [_intel_fast_memcpy.A()+10] [SIGSEGV] [Address not mapped to object] [0x2B09D44BE000] [] [] Current SQL statement for this session: update product set type=:1, ship_type=:2, stock_warning2=:3, stock_warning=:4, store_id=:5, PRODUCT_NAME=:6, UNIT_PRICE=:7, LIST_PRICE=:8, QUANTITY=:9, BRAND=:10, MODEL=:11, Spec=:12, UNIT=:13, MANUFACTORY=:14, PRODUCINGAREA=:15, RESELLER=:16, supplier_id=:17, CREATE_TIME=:18, UPDATE_TIME=:19, DESCRIPTION=:20, BIG_PICTURE=:21, SMALL_PICTURE=:22, CENTER_PICTURE=:23, POP_PICTURE=:24, POP_PICTURE1=:25, POP_PICTURE2=:26, POP_PICTURE3=:27, POP_PICTURE4=:28, POP_PICTURE5=:29, note=:30, TEMPLATE_ID=:31, CHECKSTATUS=:32, check_man=:33, check_time=:34, STATUS=:35, SEARCHKEY=:36, MAIN_CATEGORY=:37, PRODUCT_NO=:38, sid=:39, displayId=:40, SORTINDEX=:41, corpid=:42, LINKURL=:43, spec_name1=:44, spec_name2=:45, spec_value1=:46, spec_value2=:47, POP_TYPE=:48, tag=:49, visitCount=:50, startTime=:51, endTime=:52, refer_price=:53, base_price=:54, parent_id=:55, isconfig=:56, short_name=:57, custom_image=:58, app_usercount=:59, app_totalCount=:60, buy_totalCount=:61, weight=:62, integral=:63, integral_Count=:64, is_userPrice=:65, short_name_color=:66, short_name_bold=:67, p_color=:68, p_color_pic=:69, charges_flag=:70, EXT_FIELD1=:71, EXT_FIELD2=:72, extFileld3=:73, extFileld4=:74, ext_fileld5=:75, ext_fileld6=:76, ext_fileld7=:77, ext_fileld8=:78, ext_fileld9=:79, ext_fileld10=:80, ext_fileld11=:81, ext_fileld12=:82, ext_fileld13=:83, ext_fileld14=:84, ext_fileld15=:85, ext_fileld16=:86, ext_fileld17=:87, ext_fileld18=:88, ext_fileld19=:89, ext_fileld20=:90, esalesno=:91, classtype=:92, game_type=:93, each_buy_count=:94, limit_buy_count=:95, start_limit_time=:96, end_limit_time=:97, channel_flag=:98, YDPRICE=:99, TQYD_DAYS=:100, SCOPE=:101, VD_ID1=:102, VD_ID2=:103, VD_ID3=:104, VD_ID4=:105, VD_ID5=:106, AUTO_GTCARD=:107, TV_NOTE=:108, MOBILE_NOTE=:109, IS_YYPRODUCT=:110, EXT_FILELD41=:111, WARRANTY=:112, AGENT_ID=:113, min_buynum=:114, MODEL_NO=:115, tp_spec_value1=:116, tp_spec_value2=:117 where PRODUCT_ID=:118 ----- Call Stack Trace ----- |
ORA-07445这个是个内部错误，查看oracle官方文档解释
ORA-07445:exception encountered: core dump [string] [string] [string] [string]
[string][string]
Cause: An OS exception occurred which should resultin the creation of a core file.
This is aninternal error.
Action: Contact your customer support representative.
从这个解释让我们联系oracle的技术支持。
3.4 报表数据库goldengate同步日志
查看oracle goldengate进程
| GGSCI (DB-87) 1> info all
Program Status Group Lag Time Since Chkpt
MANAGER RUNNING REPLICAT ABORT REPL 10:43:32 00:00:01
|
查看goldengate日志
| 2012-05-24 18:03:19 WARNING OGG-00869 Oracle GoldenGate Delivery for Oracle, repl.prm: OCI Error calling OCITransCommit (status = 3114-ORA-03114: not connected to ORACLE), SQL<UPDATE "JSCNBI"."PRODUCT" SET "PRODUCT_NAME" = :a1,"UNIT_PRICE" = :a2,"LIST_PRICE" = :a3,"QUANTITY" = :a4,"BRAND" = :a5,"MODEL" = :a6,"SPEC" = :a7,"UNIT" = :a8,"MANUFACTORY" = :a9,"RESELLER" = :a10,"P>. 2012-05-24 18:03:19 WARNING OGG-01003 Oracle GoldenGate Delivery for Oracle, repl.prm: Repositioning to rba 7146675 in seqno 1671. 2012-05-24 18:03:19 ERROR OGG-00665 Oracle GoldenGate Delivery for Oracle, repl.prm: OCI Error calling OCITransCommit (status = 3114-ORA-03114: not connected to ORACLE), SQL<UPDATE "JSCNBI"."PRODUCT" SET "PRODUCT_NAME" = :a1,"UNIT_PRICE" = :a2,"LIST_PRICE" = :a3,"QUANTITY" = :a4,"BRAND" = :a5,"MODEL" = :a6,"SPEC" = :a7,"UNIT" = :a8,"MANUFACTORY" = :a9,"RESELLER" = :a10,"P>. 2012-05-24 18:03:19 ERROR OGG-01668 Oracle GoldenGate Delivery for Oracle, repl.prm: PROCESS ABENDING. |
4. 解决问题
这个oracle 10.2.0.4 在rac环境下的一个bug，有以下两种方法来解决
l 升级到10.2.0.5版本
l 尽量避免行迁移
4.1尽量避免行迁移
4.1.1 避免新插入的数据行迁移
通过配置pctfree和pctused参数来使表减少行迁移
| SQL> alter table product pctfree 20;
Table altered.
SQL> alter table product pctfree 40;
Table altered. |
4.1.2 消除现有数据的行迁移
BEGIN
SYS.DBMS_STATS.GATHER_TABLE_STATS (
OwnName => 'JSCNBI'
,TabName => 'PRODUCT'
,Estimate_Percent => 0
,Method_Opt => 'FOR ALLINDEXED COLUMNS SIZE 1 '
,Degree => 4
,Cascade => TRUE
,No_Invalidate => FALSE);
END;
/
1) 重新move表
| SQL> alter table product move tablespace JSCNBIDATA01; |
|
|
2) 重新rebuild索引
| 执行sql语句查询出所有的索引 select 'alter index '|| I.INDEX_NAME || ' rebuild tablespace JSCNBIINDEX01;' from user_indexes i where I.TABLE_NAME='PRODUCT';
alter index VISICOUNT2 rebuild tablespace JSCNBIINDEX01; alter index PRODUCT_STATUS rebuild tablespace JSCNBIINDEX01; alter index SID_INDEX2 rebuild tablespace JSCNBIINDEX01; alter index PRODUCT_NAME2 rebuild tablespace JSCNBIINDEX01; alter index CREATE_TIME2 rebuild tablespace JSCNBIINDEX01; alter index CREATE_SID2 rebuild tablespace JSCNBIINDEX01; alter index PRODUCT_ID rebuild tablespace JSCNBIINDEX01;
|
3) 重新制定clob存储空间
| alter table PRODUCT move tablespace jscndata lob (EXT_FILELD41) store as(tablespace JSCNBIINDEX01); alter table PRODUCT move tablespace jscndata lob (DESCRIPTION) store as(tablespace JSCNBIINDEX01);
|
4) 结束以后收集统计信息
| BEGIN SYS.DBMS_STATS.GATHER_TABLE_STATS ( OwnName => 'JSCNBI' ,TabName => 'PRODUCT' ,Estimate_Percent => 0 ,Method_Opt => 'FOR ALL INDEXED COLUMNS SIZE 1 ' ,Degree => 4 ,Cascade => TRUE ,No_Invalidate => FALSE); END; / |
此方法，以及在测试环境验证通过。