In this Document
Purpose
What is the Issue?
Scope and Application
Who is Affected?
Enterprise Manager Database Control Configuration - Recovering From Errors Due to CA Expiry on Oracle Database 10.2.0.4 or 10.2.0.5 [Video]
What Happens During Database Control Configuration Failure?
Recovering from Configuration Errors on a Single Instance Database
Recovering from Configuration Errors in an Oracle Real Application Clusters (RAC) Environment
References
Could not complete the Enterprise Manager configuration.
Enterprise manager configuration failed due to the following error:
Error starting Database Control
CONFIG: Securing Database Control completed successfully .
Jan 2, 2011 7:22:47 PM oracle.sysman.emcp.ParamsManager getParam
CONFIG: No value was set for the parameter ORACLE_HOSTNAME.
Jan 2, 2011 7:22:47 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Jan 2, 2011 7:22:47 PM oracle.sysman.emcp.util.PlatformInterface addEnvVarToList
CONFIG: Value for env var 'ORACLE_HOSTNAME' is '', discarding the same
CONFIG: Returning env array from cache
Jan 2, 2011 7:22:47 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
CONFIG: Starting execution: /myhost/bin/emctl start dbconsole
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
CONFIG: Exit value of 1
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
CONFIG: Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:5501/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control
............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /myhost/sysman/log
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /myhost/bin/emctl start dbconsole
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Error starting Database Control
Refer to the log file at /myhost/dbua/d4/upgrade/emConfig.log for more details.
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.EMConfig perform
CONFIG: Stack Trace:
oracle.sysman.emcp.exception.EMConfigException: Error starting Database Control
at oracle.sysman.emcp.EMDBPostConfig.performUpgrade(EMDBPostConfig.java:763)
at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:232)
at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:193)
at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:184)
at oracle.sysman.assistants.util.em.EMConfiguration.run(EMConfiguration.java:436)
at oracle.sysman.assistants.util.em.EMConfigStep.executeImpl(EMConfigStep.java:140)
at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
at oracle.sysman.assistants.util.step.BasicStep.callStep(BasicStep.java:251)
at oracle.sysman.assistants.dbma.backend.EMConfigStep.executeStepImpl(EMConfigStep.java:104)
at oracle.sysman.assistants.dbma.backend.SummarizableStep.executeImpl(SummarizableStep.java:175)
at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
at oracle.sysman.assistants.util.step.Step.execute(Step.java:140)
at oracle.sysman.assistants.util.step.StepContext$ModeRunner.run(StepContext.java:2488)
at java.lang.Thread.run(Thread.java:534)
aime@myhost09 db_1]$ bin/emca -config dbcontrol db -repos recreate -cluster
STARTED EMCA at Jan 11, 2011 4:11:01 PM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database unique name: catest
Database Control is already configured for the database catest
You have chosen to configure Database Control for managing the database catest
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: Y
Listener port number: 1521
Cluster name: mycluster
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
........
Jan 11, 2011 4:18:05 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
Jan 11, 2011 4:19:31 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Jan 11, 2011 4:28:38 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Error starting Database Control
Refer to the log file at /myhost/oracle/product/10.2.0/db_1/cfgtoollogs/emca/catest/emca_2011-01-11_
04-11-01-PM.log for more details.
Could not complete the configuration. Refer to the log file at /myhost/oracle/product/10.2.0/db_
1/cfgtoollogs/emca/catest/emca_2011-01-11_04-11-01-PM.log for more details.
2011-01-09 09:36:56 Thread-51125136 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://myhost:1158/em/upload/: retStatus=-1
2011-01-09 09:36:56 Thread-51125136 ERROR ssl: Open wallet failed, ret = 28750
2011-01-09 09:36:56 Thread-51125136 ERROR ssl: nmehlenv_openWallet failed
2011-01-09 09:36:56 Thread-51125136 ERROR http: 15: Unable to initialize ssl connection with server, aborting connection attempt
2011-01-06 18:50:54 Thread-3393 ERROR ssl: nzos_Initialize failed, ret = 43061
2011-01-06 18:50:54 Thread-3393 ERROR http: 14: Unable to initialize ssl connection with server, aborting connection attempt
2011-01-06 18:50:54 Thread-3393 ERROR pingManager: nmepm_pingReposURL: Cannot connect tohttps://myhost:1158/em/upload/:retStatus=-1
$ ./emctl status dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
https://myhost:1158/em/console/aboutApplication
Oracle Enterprise Manager 10g is not running.
$ ./emctl status agent
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent is Not Running
opatch apply
Invoking OPatch 10.2.0.4.2
Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : /myhost
Central Inventory : /oracle/inv/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /myhost/oui
Log file location : /myhost/cfgtoollogs/opatch/opatch2011-01-02_11-00-00AM.log
ApplySession applying interim patch '8350262' to OH '/myhost'
Running prerequisite checks...
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '8350262' for restore. This might take a while...
Backing up files affected by the patch '8350262' for rollback. This might take a while...
Patching component oracle.sysman.agent.core, 10.2.0.4.0a...
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/util/SecConstants.class"
ApplySession adding interim patch '8350262' to inventory
Verifying the update...
Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8350262 are present in Oracle Home.
OPatch succeeded.
$ <PATCH_HOME>/killDBConsole
ORACLE_HOME=/oracle/db_1
ORACLE_SID=mysid
State directory = /oracle/db_1/myhost_mysid
WatchDog PID = 802932
DBconsole PID = 577716
EMAgent PID = 512156
Killing WatchDog (pid=802932) ...
Successfully killed process 802932
Killing DBConsole (pid=577716) ...
Successfully killed process 577716
Killing EMAgent (pid=512156) ...
Successfully killed process 512156
You will be prompted twice to confirm that the Root key must be overwritten. In both cases, enter upper-case "Y" as the response. Any other response (including lower-case "y") will cause the command to terminate without completing. If this happens, the command can be re-invoked.
$ ./emctl secure dbconsole -reset
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:5501/em/console/aboutApplication
Enter Enterprise Manager Root Password :
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Checking Repository... Done.
Checking Em Key... Done.
Checking Repository for an existing Enterprise Manager Root Key...
WARNING! An Enterprise Manager Root Key already exists in
the Repository. This operation will replace your Enterprise
Manager Root Key.
All existing Agents that use HTTPS will need to be
reconfigured if you proceed. Do you wish to continue and
overwrite your Root Key
(Y/N) ?
Y
Are you sure ? Reset of the Enterprise Manager Root Key
will mean that you will need to reconfigure each Agent
that is associated with this OMS before they will be
able to upload any data to it. Monitoring of Targets
associated with these Agents will be unavailable until
after they are reconfigured.
(Y/N) ?
Y
Generating Enterprise Manager Root Key (this takes a minute)... Done.Fetching Root Certificate from
the Repository... Done.
Updating HTTPS port in emoms.properties file... Done.
Generating Java Keystore... Done.
Securing OMS ... Done.
Generating Oracle Wallet Password for Agent.... Done.
Generating wallet for Agent ... Done.
Copying the wallet for agent use... Done.
Storing agent key in repository... Done.
Storing agent key for agent ... Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode... Done.
EMD_URL set in /oracle/db_1/sysman/config/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole... Sucessful.
../OPatch/opatch apply
Invoking OPatch 10.2.0.4.2
Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : /myhost/oracle/product/10.2.0/db_1
Central Inventory : /myhost/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /myhost/oracle/product/10.2.0/db_1/oui
Log file location : /myhost/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2011-01-01_
21-30-27PM.log
ApplySession applying interim patch '8350262' to OH '/myhost/oracle/product/10.2.0/db_1'
Running prerequisite checks...
OPatch detected the node list and the local node from the inventory. OPatch will patch the local
system then propagate the patch to the remote nodes.
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '8350262' for restore. This might take a while...
Backing up files affected by the patch '8350262' for rollback. This might take a while...
Patching component oracle.sysman.agent.core, 10.2.0.4.0a...
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with
"/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/SecConstants.class"
ApplySession adding interim patch '8350262' to inventory
Verifying the update...
Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8350262 are present in Oracle Home.
Patching in rolling mode.
Updating nodes 'myhost'
Apply-related files are:
FP = :/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt"
DP = "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt"
MP = "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/make_cmds.txt"
RC = "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/remote_cmds.txt"
Instantiating the file "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_
44/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/myhost/oracle/product/10.2.0/db_
1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_
44/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/myhost/oracle/product/10.2.0/db_
1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
OPatch succeeded.
$ <PATCH_HOME>/killDBConsole
ORACLE_HOME=/u01/oracle/db_1
ORACLE_SID=mysid
State directory = /u01/oracle/db_1/myhost_mysid
WatchDog PID = 802932
DBconsole PID = 577716
EMAgent PID = 512156
Killing WatchDog (pid=802932) ...
Successfully killed process 802932
Killing DBConsole (pid=577716) ...
Successfully killed process 577716
Killing EMAgent (pid=512156) ...
Successfully killed process 512156
$ ./emctl secure dbconsole -reset
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:5501/em/console/aboutApplication
Enter Enterprise Manager Root Password :
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Checking Repository... Done.
Checking Em Key... Done.
Checking Repository for an existing Enterprise Manager Root Key...
WARNING! An Enterprise Manager Root Key already exists in
the Repository. This operation will replace your Enterprise
Manager Root Key.
All existing Agents that use HTTPS will need to be
reconfigured if you proceed. Do you wish to continue and
overwrite your Root Key
(Y/N) ?
Y
Are you sure ? Reset of the Enterprise Manager Root Key
will mean that you will need to reconfigure each Agent
that is associated with this OMS before they will be
able to upload any data to it. Monitoring of Targets
associated with these Agents will be unavailable until
after they are reconfigured.
(Y/N) ?
Y
Generating Enterprise Manager Root Key (this takes a minute)... Done.Fetching Root Certificate from
the Repository... Done.
Updating HTTPS port in emoms.properties file... Done.
Generating Java Keystore... Done.
Securing OMS ... Done.
Generating Oracle Wallet Password for Agent.... Done.
Generating wallet for Agent ... Done.
Copying the wallet for agent use... Done.
Storing agent key in repository... Done.
Storing agent key for agent ... Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode... Done.
EMD_URL set in /myhost/sysman/config/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole... Sucessful.
[myhost bin]$ ./emctl secure dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:1158/em/console/aboutApplication
Enter Enterprise Manager Root password :
Enter a Hostname for this OMS : myhost
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Checking Repository... Done.
Checking Em Key... Done.
Checking Repository for an existing Enterprise Manager Root Key... Done.
Fetching Root Certificate from the Repository... Done.
Updating HTTPS port in emoms.properties file... Done.
Generating Java Keystore... Done.
Securing OMS ... Done.
Generating Oracle Wallet Password for Agent.... Done.
Generating wallet for Agent ... Done.
Copying the wallet for agent use... Done.
Storing agent key in repository... Done.
Storing agent key for agent ... Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode... Done.
EMD_URL set in /myhost/oracle/product/10.2.0/db_1/myhost/sysman/c
onfig/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole... Sucessful.