Oracle 10.2.0.4中 OEM bug 8350262

1、发现问题

今天在OEL 4环境，从10.2.0.1升级至10.2.0.4，然后dbca建库，进度在91%的时候就不动了一直卡在那等了很久，最后报出来下面的错误~让我们来瞧瞧！

通过google搜索了下原来这是ORACLE在10.2.0.4和10.2.0.5的BUG,运气这么好，被我给碰上了~

根据提示打开emConfig.log日志文件
Vi /u01/app/oracle/product/10.2/db/cfgtoollogs/dbca/prod/emConfig.log

Aug 17, 2011 10:12:30 AM oracle.sysman.emcp.util.OUIInventoryUtil setOUILoc

CONFIG: Setting oracle.installer.oui_loc to /u01/app/oracle/product/10.2/db/oui

Aug 17, 2011 10:12:30 AM oracle.sysman.emcp.util.OUIInventoryUtil setOUILoc

CONFIG: Setting oracle.installer.oui_loc to /u01/app/oracle/product/10.2/db/oui

Aug 17, 2011 10:12:33 AM oracle.sysman.emcp.util.CentralAgentUtil getCentralAgentHomeAndURL

CONFIG: Central Agent home and URL: {}

Aug 17, 2011 10:12:33 AM oracle.sysman.emcp.EMConfig restoreOuiLoc

CONFIG: Restoring oracle.installer.oui_loc to /u01/app/oracle/product/10.2/db/oui

Aug 17, 2011 11:45:57 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: LOG_FILE value: /u01/app/oracle/product/10.2/db/cfgtoollogs/dbca/prod/emConfig.log

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: HOST value: oel1

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: PORT value: 1521

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: SID value: prod

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: LISTENER value: LISTENER

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: LISTENER_OH value: /u01/app/oracle/product/10.2/db

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setFlag

CONFIG: Flag ‘-config’ set to true

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setFlag

CONFIG: Flag ‘db’ set to true

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: EM_HOME value: /u01/app/oracle/product/10.2/db

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: DB_UNIQUE_NAME value: prod

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: SERVICE_NAME value: prod

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

<product/10.2/db/cfgtoollogs/dbca/prod/emConfig.log” 1500L, 138732C 1,1           Top

Aug 17, 2011 10:12:30 AM oracle.sysman.emcp.util.OUIInventoryUtil setOUILoc

CONFIG: Setting oracle.installer.oui_loc to /u01/app/oracle/product/10.2/db/oui

Aug 17, 2011 10:12:30 AM oracle.sysman.emcp.util.OUIInventoryUtil setOUILoc

CONFIG: Setting oracle.installer.oui_loc to /u01/app/oracle/product/10.2/db/oui

Aug 17, 2011 10:12:33 AM oracle.sysman.emcp.util.CentralAgentUtil getCentralAgentHomeAndURL

CONFIG: Central Agent home and URL: {}

Aug 17, 2011 10:12:33 AM oracle.sysman.emcp.EMConfig restoreOuiLoc

CONFIG: Restoring oracle.installer.oui_loc to /u01/app/oracle/product/10.2/db/oui

Aug 17, 2011 11:45:57 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: LOG_FILE value: /u01/app/oracle/product/10.2/db/cfgtoollogs/dbca/prod/emConfig.log

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: HOST value: oel1

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: PORT value: 1521

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: SID value: prod

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: LISTENER value: LISTENER

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: LISTENER_OH value: /u01/app/oracle/product/10.2/db

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setFlag

CONFIG: Flag ‘-config’ set to true

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setFlag

CONFIG: Flag ‘db’ set to true

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: EM_HOME value: /u01/app/oracle/product/10.2/db

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: DB_UNIQUE_NAME value: prod

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: SERVICE_NAME value: prod

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setParam

CONFIG: Setting param: ORACLE_HOME value: /u01/app/oracle/product/10.2/db

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.ParamsManager setFlag

CONFIG: Flag ‘-cluster’ set to true

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.util.ClusterUtil areNodesAlive

CONFIG: Alive node cache contents: []

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.util.ClusterUtil areNodesAlive

CONFIG: oel1 not found in alive node cache

Aug 17, 2011 11:46:27 AM oracle.sysman.emcp.util.ClusterUtil areNodesAlive

CONFIG: oel2 not found in alive node cache

Aug 17, 2011 11:46:29 AM oracle.sysman.emcp.util.ClusterUtil areNodesAlive

CONFIG: Node: oel1 is accessible.

Aug 17, 2011 11:46:29 AM oracle.sysman.emcp.util.ClusterUtil areNodesAlive

CONFIG: Node: oel2 is accessible.

Aug 17, 2011 11:46:29 AM oracle.sysman.emcp.ParamsManager getInaccessibleNodeList

CONFIG: All nodes are up. Nodes: [oel1, oel2]

Aug 17, 2011 11:46:29 AM oracle.sysman.emcp.ParamsManager getInaccessibleSidList

CONFIG: All nodes are up. Nodes: [oel1, oel2] Sids: [prod1, prod2]

2、Oracle对这个bug原因的说明：

      In Enterprise Manager Database Control with Oracle Database 10.2.0.4 and 10.2.0.5, the root certificate used to secure communications via the Secure Socket Layer (SSL) protocol will expire on 31-Dec-2010 00:00:00. The certificate expiration will cause errors if you attempt to configure Database Control on or after 31-Dec-2010. Existing Database Control configurations are not impacted by this issue.

      If you plan to configure Database Control with either of these Oracle Database releases, Oracle strongly recommends that you apply Patch 8350262 to your Oracle Home installations before you configure Database Control. Configuration of Database Control is typically done when you create or upgrade Oracle Database, or if you run Enterprise Manager Configuration Assistant (EMCA) in standalone mode.

Note the following:

      （1）The issue impacts configuration of Database Control with Oracle Database 10.2.0.4 and 10.2.0.5 only. It does not impact database creation or upgrade.

      （2）The issue does not impact existing Database Control configurations.

      （3）Application of Patch 8350262 does not require any database downtime

3、解决问题

ORACLE 给出的建议操作步骤如下：

Recovering from Configuration Errors on a Single Instance Database
单实例

Recovering from Configuration Errors on a Single Instance Database
1. Ignore any errors and continue with the installation or upgrade. The database will be created without errors.

2. Apply Patch 8350262 to your Oracle Home installation using OPatch.

opatch apply

Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.

Oracle Home : /myhost
Central Inventory : /scratch/pchebrol/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /myhost/oui
Log file location : /myhost/cfgtoollogs/opatch/opatch2011-01-02_11-00-00AM.log

ApplySession applying interim patch ’8350262′ to OH ‘/myhost’

Running prerequisite checks…

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch ’8350262′ for restore. This might take a while…
Backing up files affected by the patch ’8350262′ for rollback. This might take a while…

Patching component oracle.sysman.agent.core, 10.2.0.4.0a…
Updating jar file “/myhost/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class”
Updating jar file “/myhost/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class”
Updating jar file “/myhost/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class”
Updating jar file “/myhost/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class”
Updating jar file “/myhost/sysman/jlib/emd_java.jar” with “/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class”
Updating jar file “/myhost/sysman/jlib/emd_java.jar” with “/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class”
Updating jar file “/myhost/sysman/jlib/emd_java.jar” with “/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/util/RootCert.class”
Updating jar file “/myhost/sysman/jlib/emd_java.jar” with “/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/util/SecConstants.class”
ApplySession adding interim patch ’8350262′ to inventory

Verifying the update…
Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8350262 are present in Oracle Home.

OPatch succeeded.

3. After applying the patch, force stop the Database Control (dbconsole) process using the killDBConsole script bundled with the patch. Note that the dbconsole process cannot be stopped using the emctl stop dbconsole command, as EMCTL is unable to connect to the process.

To execute the killDBConsole script:

Set the ORACLE_HOME and ORACLE_SID environment variables.
Execute <PATCH_HOME>/killDBConsole.
Note for Windows Platform Only:

It is not necessary to force stop the dbconsole process on the Windows platform, because the process will
already be in a stopped state at the end of the failed configuration attempt.

The killDBConsole script output is shown below:

$ <PATCH_HOME>/killDBConsole
ORACLE_HOME=/myhost/db_1
ORACLE_SID=caem31
State directory = /myhost/db_1/staxd10_caem31
WatchDog PID = 802932
DBconsole PID = 577716
EMAgent PID = 512156
Killing WatchDog (pid=802932) …
Successfully killed process 802932
Killing DBConsole (pid=577716) …
Successfully killed process 577716
Killing EMAgent (pid=512156) …
Successfully killed process 512156

4. Re-secure Database Control with the following command:

<ORACLE_HOME>/bin/emctl secure dbconsole -reset

You will be prompted twice to confirm that the Root key must be overwritten. In both cases, enter upper-case “Y” as the response. Any other response (including lower-case “y”) will cause the command to terminate without completing. If this happens, the command can be re-invoked.

$ ./emctl secure dbconsole -reset
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

https://myhost:5501/em/console/aboutApplication

Enter Enterprise Manager Root Password :
DBCONSOLE already stopped… Done.
Agent is already stopped… Done.
Securing dbconsole… Started.
Checking Repository… Done.
Checking Em Key… Done.
Checking Repository for an existing Enterprise Manager Root Key…
WARNING! An Enterprise Manager Root Key already exists in
the Repository. This operation will replace your Enterprise
Manager Root Key.
All existing Agents that use HTTPS will need to be
reconfigured if you proceed. Do you wish to continue and
overwrite your Root Key
(Y/N) ?
Y
Are you sure ? Reset of the Enterprise Manager Root Key
will mean that you will need to reconfigure each Agent
that is associated with this OMS before they will be
able to upload any data to it. Monitoring of Targets
associated with these Agents will be unavailable until
after they are reconfigured.
(Y/N) ?
Y
Generating Enterprise Manager Root Key (this takes a minute)… Done.Fetching Root Certificate from
the Repository… Done.
Updating HTTPS port in emoms.properties file… Done.
Generating Java Keystore… Done.
Securing OMS … Done.
Generating Oracle Wallet Password for Agent…. Done.
Generating wallet for Agent … Done.
Copying the wallet for agent use… Done.
Storing agent key in repository… Done.
Storing agent key for agent … Done.
Configuring Agent…
Configuring Agent for HTTPS in DBCONSOLE mode… Done.
EMD_URL set in /myhost/myhost/sysman/config/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole… Sucessful.

5. Re-start Database Control with the following command:

<ORACLE_HOME>/bin/emctl start dbconsole

RAC环境

Recovering from Configuration Errors in an Oracle Real Application Clusters (RAC) Environment
1. Ignore any errors and continue with the upgrade, so that the database is upgraded without errors.

2. Apply Patch 8350262 to your Oracle Home installation. Note that the OPatch utility will apply the patch to all nodes in the cluster, as shown below:

../OPatch/opatch apply
Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.

Oracle Home : /myhost/oracle/product/10.2.0/db_1
Central Inventory : /myhost/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /myhost/oracle/product/10.2.0/db_1/oui
Log file location : /myhost/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2011-01-01_
21-30-27PM.log

ApplySession applying interim patch ’8350262′ to OH ‘/myhost/oracle/product/10.2.0/db_1′

Running prerequisite checks…

OPatch detected the node list and the local node from the inventory. OPatch will patch the local
system then propagate the patch to the remote nodes.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch ’8350262′ for restore. This might take a while…
Backing up files affected by the patch ’8350262′ for rollback. This might take a while…

Patching component oracle.sysman.agent.core, 10.2.0.4.0a…
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar” with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar” with
“/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar” with
“/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar” with
“/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/RootCert.class”
Updating jar file “/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar” with
“/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/SecConstants.class”
ApplySession adding interim patch ’8350262′ to inventory

Verifying the update…
Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8350262 are present in Oracle Home.

Patching in rolling mode.

Updating nodes ‘myhost’
Apply-related files are:
FP = :/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_
files.txt”
DP = “/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_
dirs.txt”
MP = “/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/make_
cmds.txt”
RC = “/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/remote_
cmds.txt”

Instantiating the file “/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_
44/rac/copy_files.txt.instantiated” by replacing $ORACLE_HOME in “/myhost/oracle/product/10.2.0/db_
1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt” with actual path.
Propagating files to remote nodes…
Instantiating the file “/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_
44/rac/copy_dirs.txt.instantiated” by replacing $ORACLE_HOME in “/myhost/oracle/product/10.2.0/db_
1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt” with actual path.
Propagating directories to remote nodes…

OPatch succeeded.

3. After applying the patch, force stop the Database Control (dbconsole) process by executing the
killDBConsole script bundled with the patch on each node in the cluster. Note that the dbconsole
process cannot be stopped using the emctl stop dbconsole command, as EMCTL is unable to connect
to the process.

To execute the killDBConsole script:

Set the ORACLE_HOME and ORACLE_SID environment variables.
Execute <PATCH_HOME>/killDBConsole
Note for Windows Platform Only:
It is not necessary to force stop the dbconsole process on the Windows platform, because the process will
already be in a stopped state at the end of the failed configuration attempt.

The killDBConsole script output is shown below:

$ <PATCH_HOME>/killDBConsole
ORACLE_HOME=/myhost/catest/db_1
ORACLE_SID=caem31
State directory = /myhost/catest/db_1/staxd10_caem31
WatchDog PID = 802932
DBconsole PID = 577716
EMAgent PID = 512156
Killing WatchDog (pid=802932) …
Successfully killed process 802932
Killing DBConsole (pid=577716) …
Successfully killed process 577716
Killing EMAgent (pid=512156) …
Successfully killed process 512156

4. Re-secure Database Control on the first cluster node with the following command:

<ORACLE_HOME>/bin/emctl secure dbconsole -reset

You will be prompted twice to confirm that the Root key must be overwritten. In both cases, enter upper-case “Y” as the response. Any other response (including lower-case “y”) will cause the command to terminate without completing. If this happens, the command can be re-invoked.

$ ./emctl secure dbconsole -reset
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

https://myhost:5501/em/console/aboutApplication

Enter Enterprise Manager Root Password :
DBCONSOLE already stopped… Done.
Agent is already stopped… Done.
Securing dbconsole… Started.
Checking Repository… Done.
Checking Em Key… Done.
Checking Repository for an existing Enterprise Manager Root Key…
WARNING! An Enterprise Manager Root Key already exists in
the Repository. This operation will replace your Enterprise
Manager Root Key.
All existing Agents that use HTTPS will need to be
reconfigured if you proceed. Do you wish to continue and
overwrite your Root Key
(Y/N) ?
Y
Are you sure ? Reset of the Enterprise Manager Root Key
will mean that you will need to reconfigure each Agent
that is associated with this OMS before they will be
able to upload any data to it. Monitoring of Targets
associated with these Agents will be unavailable until
after they are reconfigured.
(Y/N) ?
Y
Generating Enterprise Manager Root Key (this takes a minute)… Done.Fetching Root Certificate from
the Repository… Done.
Updating HTTPS port in emoms.properties file… Done.
Generating Java Keystore… Done.
Securing OMS … Done.
Generating Oracle Wallet Password for Agent…. Done.
Generating wallet for Agent … Done.
Copying the wallet for agent use… Done.
Storing agent key in repository… Done.
Storing agent key for agent … Done.
Configuring Agent…
Configuring Agent for HTTPS in DBCONSOLE mode… Done.
EMD_URL set in /myhost/sysman/config/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole… Sucessful.

5. Re-secure Database Control on the remaining cluster nodes with the following command. Note that the -reset switch is not included with this command:

<ORACLE_HOME>/bin/emctl secure dbconsole

[myhost bin]$ ./emctl secure dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

https://myhost:1158/em/console/aboutApplication

Enter Enterprise Manager Root password :
Enter a Hostname for this OMS : myhost
DBCONSOLE already stopped… Done.
Agent is already stopped… Done.
Securing dbconsole… Started.
Checking Repository… Done.
Checking Em Key… Done.
Checking Repository for an existing Enterprise Manager Root Key… Done.
Fetching Root Certificate from the Repository… Done.
Updating HTTPS port in emoms.properties file… Done.
Generating Java Keystore… Done.
Securing OMS … Done.
Generating Oracle Wallet Password for Agent…. Done.
Generating wallet for Agent … Done.
Copying the wallet for agent use… Done.
Storing agent key in repository… Done.
Storing agent key for agent … Done.
Configuring Agent…
Configuring Agent for HTTPS in DBCONSOLE mode… Done.
EMD_URL set in /myhost/oracle/product/10.2.0/db_1/myhost/sysman/c
onfig/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole… Sucessful.

6. Re-start Database Control by executing the following command on each node in the cluster:

<ORACLE_HOME>/bin/emctl start dbconsole