Oracle RAC 19.3打补丁到19.6 Datapatch ORA-29516 ORA-00604 and ORA-29532 java.lang.ArrayIndexOutOfBoundsE
先安装19.3.0.0.0的GI和DB软件，然后安装19.6.0.0.0的GI和DB补丁，最后DBCA建库，建库耗时很长，用了3个多小时，然而数据库没有成功apply RU，并且数据库组件SDO INVALID。
SQL> select status,description from dba_registry_sqlpatch;
STATUS DESCRIPTION
------------------------- --------------------------------------------------------------------------------
WITH ERRORS Database Release Update : 19.6.0.0.200114 (30557433)
SQL> select comp_id,version,status from dba_registry;
COMP_ID VERSION STATUS
------------------------------ ------------------------------ ----------------------
CATALOG 19.0.0.0.0 VALID
CATPROC 19.0.0.0.0 VALID
RAC 19.0.0.0.0 VALID
JAVAVM 19.0.0.0.0 VALID
XML 19.0.0.0.0 VALID
CATJAVA 19.0.0.0.0 VALID
APS 19.0.0.0.0 VALID
XDB 19.0.0.0.0 VALID
OWM 19.0.0.0.0 VALID
CONTEXT 19.0.0.0.0 VALID
ORDIM 19.0.0.0.0 VALID
SDO 19.0.0.0.0 INVALID
XOQ 19.0.0.0.0 VALID
OLS 19.0.0.0.0 VALID
DV 19.0.0.0.0 VALID以下是部分日志
cfgtoollogs1\dbca\orcl\trace.log_2020-04-08_09-38-26AM
[main] [ 2020-04-08 09:38:27.599 CST ] [AssistantLogHandler.configureLogging:358] Begin tracing..
[progressPage.flowWorker] [ 2020-04-08 09:59:08.941 CST ] [PostDBCreationStep.executeImpl:717] executing datapatch /u02/app/oracle/product/19.0.0/db_1/OPatch/datapatch
[progressPage.flowWorker] [ 2020-04-08 11:07:46.738 CST ] [PostDBCreationStep.executeImpl:720] datapatch returned with exit status 2
INFO: Apr 08, 2020 11:07:46 AM oracle.assistants.dbca.driver.StepDBCAJob$1 update
INFO: Percentage Progress got for job:Completing Database Creation progress:50.0
[progressPage.flowWorker] [ 2020-04-08 11:07:48.519 CST ] [PerlEngine.execute:220] /u02/app/oracle/product/19.0.0/db_1/rdbms/admin/utlrp.sql
[progressPage.flowWorker] [ 2020-04-08 11:07:48.519 CST ] [PerlEngine.execute:232] Execing PERL process...
[progressPage.flowWorker] [ 2020-04-08 11:07:48.520 CST ] [PerlEngine.execute:274] m_bReaderStarted: false
[progressPage.flowWorker] [ 2020-04-08 11:07:48.520 CST ] [PerlEngine.execute:278] Starting Reader Thread...
[progressPage.flowWorker] [ 2020-04-08 11:07:48.521 CST ] [PerlEngine.execute:332] Inputing secret arg: 0
[progressPage.flowWorker] [ 2020-04-08 11:07:48.521 CST ] [PerlEngine.execute:332] Inputing secret arg: 1
[Thread-1824] [ 2020-04-08 11:07:48.592 CST ] [PerlEngine.readPerlOutput:597] perl engine output:Enter Password:
[Thread-1824] [ 2020-04-08 11:07:48.592 CST ] [PerlEngine.readPerlOutput:597] perl engine output:Enter Password:
[progressPage.flowWorker] [ 2020-04-08 12:39:20.693 CST ] [PerlEngine.execute:375] Exit value = 0
[progressPage.flowWorker] [ 2020-04-08 12:39:20.693 CST ] [PerlEngine.execute:376] status is = 0
[progressPage.flowWorker] [ 2020-04-08 12:41:29.351 CST ] [PostPDBCreationStep.recompilePDBObjects:919] Invalid objects : 256 and invalid components : 1
[progressPage.flowWorker] [ 2020-04-08 12:41:29.352 CST ] [PostPDBCreationStep.recompilePDBObjects:922] Recompile objects in PDB as number of invalid objects : 256 and invalid components : 1
[progressPage.flowWorker] [ 2020-04-08 12:59:29.791 CST ] [PostPDBCreationStep.recompilePDBObjects:927] after utlrp
[progressPage.flowWorker] [ 2020-04-08 12:59:29.819 CST ] [PostPDBCreationStep.recompilePDBObjects:930] Invalid objects 256
[progressPage.flowWorker] [ 2020-04-08 12:59:45.311 CST ] [PostPDBCreationStep.recompilePDBObjects:919] Invalid objects : 256 and invalid components : 1
[progressPage.flowWorker] [ 2020-04-08 12:59:45.311 CST ] [PostPDBCreationStep.recompilePDBObjects:922] Recompile objects in PDB as number of invalid objects : 256 and invalid components : 1
[progressPage.flowWorker] [ 2020-04-08 13:15:25.768 CST ] [PostPDBCreationStep.recompilePDBObjects:927] after utlrp
[progressPage.flowWorker] [ 2020-04-08 13:15:25.796 CST ] [PostPDBCreationStep.recompilePDBObjects:930] Invalid objects 256
cfgtoollogs1\sqlpatch\sqlpatch_182452_2020_04_08_09_59_09\sqlpatch_catcon_0.log
SQL> call dbms_java.loadjava('-resolve -synonym -schema MDSYS -grant PUBLIC
2 md/jlib/sdoapi.jar md/jlib/sdoutl.jar md/jlib/sdotopo.jar
3 md/jlib/sdotype.jar md/jlib/sdonm.jar');
call dbms_java.loadjava('-resolve -synonym -schema MDSYS -grant PUBLIC
*
ERROR at line 1:
ORA-29516: Aurora assertion failure: Bogus uncaught exception
ORA-00604: error occurred at recursive SQL level 1
ORA-29532: Java call terminated by uncaught Java exception:
java.lang.ArrayIndexOutOfBoundsException
cfgtoollogs1\sqlpatch\sqlpatch_182452_2020_04_08_09_59_09\sqlpatch_catcon_0.log
'PDB$SEED' => [
{
'errors' => [
{
'error_message' => [
'ORA-29532: Java call terminated by uncaught Java exception:',
'oracle.aurora.server.tools.loadjava.ToolsError:',
'Error during loadjava: Failures occurred during processing.',
'Check trace file for details'
],
'error_stack' => 'ORA-29532',
'line_num' => 212673,
'script' => 'md/admin/catmdloc.sql'
},
{
'error_message' => [
'ORA-29532: Java call terminated by uncaught Java exception:',
'oracle.aurora.server.tools.loadjava.ToolsError:',
'Error during loadjava: Failures occurred during processing.',
'Check trace file for details'
],
'error_stack' => 'ORA-29532',
'line_num' => 212692,
'script' => 'md/admin/catmdloc.sql'
},
{
'error_message' => [
'ORA-29532: Java call terminated by uncaught Java exception:',
'oracle.aurora.server.tools.loadjava.ToolsError:',
'Error during loadjava: Failures occurred during processing.',
'Check trace file for details'
],
'error_stack' => 'ORA-29532',
'line_num' => 212735,
'script' => 'md/admin/catmdloc.sql'
}
],
'logfile' => '/u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_ORCL_PDBSEED_2020Apr08_10_28_34.log',
'mode' => 'apply',
'patchID' => '30557433',
'patchUID' => '23305305',
'ru_errors' => undef,
'ru_logfile' => '/u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_ru_apply_ORCL_PDBSEED_2020Apr08_10_28_33.log',
'status' => 'WITH ERRORS'
}
]尝试手动执行datapatch -verbose
[oracle@db1:/u02/app/oracle/product/19.0.0/db_1/OPatch]$ ./datapatch -verbose
SQL Patching tool version 19.6.0.0.0 Production on Wed Apr 8 16:44:06 2020
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u02/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_182098_2020_04_08_16_44_06/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
No interim patches found
Current state of release update SQL patches:
Binary registry:
19.6.0.0.0 Release_Update 191217155004: Installed
PDB CDB$ROOT:
Applied 19.6.0.0.0 Release_Update 191217155004 with errors on 08-APR-20 11.07.37.653363 AM
PDB PDB$SEED:
Applied 19.6.0.0.0 Release_Update 191217155004 with errors on 08-APR-20 11.07.42.331994 AM
PDB PDB1:
Applied 19.6.0.0.0 Release_Update 191217155004 with errors on 08-APR-20 11.07.42.331994 AM
PDB PDB2:
Applied 19.6.0.0.0 Release_Update 191217155004 with errors on 08-APR-20 11.07.42.331994 AM
PDB PDB3:
Applied 19.6.0.0.0 Release_Update 191217155004 with errors on 08-APR-20 11.07.42.331994 AM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED PDB1 PDB2 PDB3
No interim patches need to be rolled back
Patch 30557433 (Database Release Update : 19.6.0.0.200114 (30557433)):
Apply from 19.1.0.0.0 Feature Release to 19.6.0.0.0 Release_Update 191217155004
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 5
Validating logfiles...done
Patch 30557433 apply (pdb CDB$ROOT): WITH ERRORS
logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_ORCL_CDBROOT_2020Apr08_16_45_15.log (errors)
-> Error at line 137597: script md/admin/catmdloc.sql
- ORA-29516: Aurora assertion failure: Bogus uncaught exception
- ORA-00604: error occurred at recursive SQL level 1
- ORA-29532: Java call terminated by uncaught Java exception:
- java.lang.ArrayIndexOutOfBoundsException
-> Error at line 137653: script md/admin/catmdloc.sql
- ORA-29516: Aurora assertion failure: Bogus uncaught exception
- ORA-00604: error occurred at recursive SQL level 1
- ORA-29532: Java call terminated by uncaught Java exception:
- java.lang.ArrayIndexOutOfBoundsException
Patch 30557433 apply (pdb PDB$SEED): WITH ERRORS
logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_ORCL_PDBSEED_2020Apr08_16_51_07.log (errors)
-> Error at line 211544: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 211563: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 211606: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
Patch 30557433 apply (pdb PDB1): WITH ERRORS
logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_ORCL_PDB1_2020Apr08_16_51_07.log (errors)
-> Error at line 137597: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 137616: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 137659: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
Patch 30557433 apply (pdb PDB2): WITH ERRORS
logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_ORCL_PDB2_2020Apr08_16_51_07.log (errors)
-> Error at line 137597: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 137616: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 137659: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
Patch 30557433 apply (pdb PDB3): WITH ERRORS
logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_ORCL_PDB3_2020Apr08_16_51_07.log (errors)
-> Error at line 137597: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 137616: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
-> Error at line 137659: script md/admin/catmdloc.sql
- ORA-29532: Java call terminated by uncaught Java exception:
- oracle.aurora.server.tools.loadjava.ToolsError:
- Error during loadjava: Failures occurred during processing.
- Check trace file for details
Please refer to MOS Note 1609718.1 and/or the invocation log
/u02/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_182098_2020_04_08_16_44_06/sqlpatch_invocation.log
for information on how to resolve the above errors.
SQL Patching tool complete on Wed Apr 8 17:09:42 2020
[oracle@db1:/u02/app/oracle/product/19.0.0/db_1/OPatch]$ MOS上没有搜索到有用的信息，Google看看有没有类似情况，
和这位国内网友 http://www.itpub.net/thread-2122131-1-1.html 的情况很类似，但是他打的补丁时19.5。
和这位好像是国外友人 https://www.dbarj.com.br/2019/10/datapatch-fails-with-ora-29516-ora-00604-and-ora-29532-java-lang-arrayindexoutofboundsexception/#comments 的情况也类似，但是他的解决方法是：
The only way I could workaround this error was applying 19.4 RU before applying 19.5 RU. This solved my issue.
最后得益于国外友人Raja的评论：
Similar issue I have faced when I applied Jan20 RU (19.6.0.0) on 19.3.0.0 base installation.
Downloaded Jan20 OJVM, applied the OJVM patch and ran Datapatch again.
It fixed the issue including the invalid Spacial component.
所以，此问题的解决方法：apply 19.6.0.0.0 OJVM patch，再DBCA建库，建库(包含1个PDB)用时30分钟。
其实之前也有遇到过有些漏洞扫描也会扫描OJVM相关的漏洞，所有以后打补丁建议带上OJVM。