DBCA建库问题处理之-无法创建RAC实例

在安装完RAC11.2.0.4,并打上PSU后，在通过DBCA创建数据库实例是出现诡异现象：

弹出来的框为单实例创建，二并非RAC实例。反复试了几次都是同样的问题，检查安装无问题。
在metalink上查看了下，问题原因很多，同事通过sqlplus连接空实例测试出现如下报错：
[oracle@zderp2vprd01 cfgtoollogs]$ sqlplus / as sysdba


SQL*Plus: Release 11.2.0.4.0 Production on Thu Nov 17 09:59:10 2016


Copyright (c) 1982, 2013, Oracle.  All rights reserved.


ERROR:
ORA-12547: TNS:lost contact




Enter user-name: ^C

上述报错通常出现在打完PSU后，看到这个错误，怀疑是一个经常遇到的老问题了。目录权限问题，检查如下：
[oracle@zderp2vprd01 11.2.0]$ ls -lrt
total 8
drwxr-xr-x 73 root oinstall 4096 Nov 16 21:29 grid_1
drwxrwxr-x 78 grid oinstall 4096 Nov 17 09:40 db_1
[oracle@zderp2vprd01 11.2.0]$ chown oracle:oinstall db_1 
[root@zderp2vprd01 11.2.0]# ls -lrt
total 8
drwxr-xr-x 73 root oinstall 4096 Nov 16 21:29 grid_1
drwxrwxr-x 78 grid oinstall 4096 Nov 17 09:40 db_1
[root@zderp2vprd01 11.2.0]# chown oracle:oinstall db_1

修改后终于可以正常创建RAC实例了。

此处将metalink上的信息也贴出来，供大家参考学习。
Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.2.0.4 [Release 10.2 to 11.2]
Information in this document applies to any platform.

SYMPTOMS

After CRS or Grid Infrastructure installation and RDBMS installation, using dbca to create a RAC database,  it gives a welcome screen without the 'Oracle Real Applications Clusters' and 'Oracle single instance' database options. (same for netca).



While it supposes to be:



This can happen for the following cases.

Case I

With 11.2 Grid Infrastructure, using dbca to create a RAC database from a 10.x or 11.1.x ORACLE_HOME.

$ORACLE_HOME/cfgtoologs/dbca/trace.log shows:

[main] [10:36:4:10] [OCR.:259] Primary Group of Current user: oracle
[main] [10:36:4:11] [HASContext.getInstance:200] Module init : 24
[main] [10:36:4:11] [HASContext.getInstance:223] Local Module init : 24
[main] [10:36:4:13] [ClusterLock.:56] ClusterLock Instance Allocation Failed: oracle.ops.mgmt.has.HASContextException: OCR Error(Native: prsr_initCLSS:[21])
[main] [10:36:4:20] [Cluster.isCluster:206] oracle.ops.mgmt.cluster.Cluster.isCluster(Cluster.java:260)
oracle.ops.mgmt.cluster.Cluster.isCluster(Cluster.java:197)
oracle.sysman.assistants.util.ClusterUtils.(ClusterUtils.java:185)
oracle.sysman.assistants.util.ClusterUtils.getInstance(ClusterUtils.java:236)
oracle.sysman.assistants.dbca.backend.Host.checkOPS(Host.java:2070)
oracle.sysman.assistants.dbca.backend.Host.startOperation(Host.java:2303)
oracle.sysman.assistants.dbca.Dbca.execute(Dbca.java:115)
oracle.sysman.assistants.dbca.Dbca.main(Dbca.java:180)
[main] [10:36:4:20] [Host.checkOPS:2073] cluster existence:false
[main] [10:36:4:20] [Host.checkOPS:2111] Cluster installed=false
[Finalizer] [10:36:4:25] [ClusterLock.finalize:88] ClusterLock: finalized called for oracle.ops.mgmt.has.ClusterLock@5c7734

Case II

RDBMS is at the same release as CRS or Grid Infrastructure, RDBMS is installed with Real Application Cluster option

Case III

RDBMS is at the same release as CRS or Grid Infrastructure, RDBMS is installed on each individual node without selecting Real Application Cluster option.

Case IV,  V,  VI & VII

Grid Infrastructure and RDBMS are on the same release, they are installed as different user: grid and oracle. Grid Infrastructure is running on all nodes, RDBMS is installed with RAC option.

CAUSE

Case I

  This is caused by the cluster nodes not being pinned in 11.2 Grid Infrastructure.

See Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux Chapter 5.3.3 Pinning Cluster Nodes for Oracle Database Release 10.x or 11.x for more details.

Case II

This is caused by the problem inventory, eg: oraInventory/ContentsXML/inventory.xml does not list RAC nodes for the corresponding RDBMS ORACLE_HOME. For example:

/HOME>

While it should be similar to:

/HOME>

  
    
    
  

 

Case III

If the RDBMS software is not installed with RAC option, then dbca will not show RAC option either. To confirm if RDBMS software is installed with RAC option, please refer Note 284785.1 How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC.

Case IV

The problem is caused by RDBMS ORACLE_HOME wrongly owned by grid user. It should be owned by oracle user.
dbca trace shows:

[main] [ 2012-08-09 03:46:46.745 GST ] [OracleHome.isDatabaseOptionOn:1133]  Checking Database Option: Real Application Clusters
[main] [ 2012-08-09 03:46:46.745 GST ] [OracleHome.isDatabaseOptionOn:1190]  Database Option Real Application Clusters is false

Despite RDBMS ORACLE_HOME has RAC option installed correctly.

Further strace against dbca reveals there are ORA-600 reported before dbca display the option page:

21999 02:45:37.053231 open("/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/log/dbua4535717_ora_21999.trc", O_WRONLY|O_CREAT|O_TRUNC, 0660) = 3
...
21999 02:45:37.058573 write(3, "ORA-00600: internal error code, "..., 139) = 139
21999 02:45:37.214275 --- SIGSEGV (Segmentation fault) @ 0 (0) ---

checking /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/log/dbua4535717_ora_21999.trc, shows:
ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], []

While:
# id grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper),1300(dba)
# id oracle
uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)

$ ls -l /u01/app/oracle/product/11.2.0
drwxrwxr-x 75 grid oinstall 4096 Aug  8 14:21 dbhome_1  << this is wrongly owned by grid user

Case V

The problem is caused by an prior abnormal termination of dbca which left a test instance DBUA0 running on 1 of the cluster nodes, this prevents a new test instance to be created before the RAC option is displayed. dbca trace shows:

[main] [ 2011-03-11 21:21:08.473 GMT ] [OracleHome.initOptions:1226] Initializing Database Options with  for dummy sid=DBUA0 using initfile=/data/oracle/product/11.2.0.2/dbs/initDBUA0.ora using pwdfile=/data/oracle/product/11.2.0.2/dbs/orapwDBUA0
[main] [ 2011-03-11 21:21:08.899 GMT ] [OracleHome.initOptions:1240]  executing: startup nomount pfile='/data/oracle/product/11.2.0.2/dbs/initDBUA0.ora'
[main] [ 2011-03-11 21:21:32.871 GMT ] [SQLEngine.done:2167]  Done called
[main] [ 2011-03-11 21:21:32.872 GMT ] [OracleHome.initOptions:1247]  ORA-304: requested INSTANCE_NUMBER is busy <<<<
oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-304: requested INSTANCE_NUMBER is busy 
        at 
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1
...
[main] [ 2011-03-11 21:21:33.499 GMT ] [OracleHome.isDatabaseOptionOn:1129] Checking Database Option: Real Application Clusters
[main] [ 2011-03-11 21:21:33.500 GMT ] [OracleHome.isDatabaseOptionOn:1186] Database Option Real Application Clusters is false <<<<

Case VI

The problem is caused by $ORACLE_HOME/rdbms/audit directory missing or not writable.

$ORACLE_BASE/cfgtoollogs/dbca/trace.log__ shows:

[main] [ 2014-06-27 00:15:15.835 EST ] [OracleHome.initOptions:1236]  Initializing Database Options with  for dummy sid=DBUA1500252 using initfile=/u01/app/oradb1/product/11.2.0/dbhome_1/dbs/initDBUA1500252.ora using pwdfile=/u01/app/oradb1/product/11.2.0/dbhome_1/dbs/orapwDBUA1500252
...
[main] [ 2014-06-27 00:15:38.793 EST ] [OracleHome.initOptions:1285]  ORA-09925: Unable to create audit trail file

 

Case VII

The problem is caused by insufficient semaphores configed which leads to test instance startup failure.
dbca trace.log shows:

[main] [ 2015-08-06 09:28:58.544 EDT ] [OracleHome.initOptions:1253]  executing: startup nomount pfile='/oracle/product/11.2.0.4.5/db11g/dbs/initDBUA2857847.ora'
[main] [ 2015-08-06 09:29:00.158 EDT ] [SQLEngine.done:2189]  Done called
[main] [ 2015-08-06 09:29:00.159 EDT ] [OracleHome.initOptions:1262]  ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpbitsper

 

 

Other Possible causes
1. RDBMS software corrupted, oracle binary shows different size for different nodes

2. Grid_home is not relinked properly

 

SOLUTION

Case I

1.  To fix the issue, pin all nodes within the cluster using:

as root user:
crsctl pin css -n

2. To list all pinned nodes:

olsnodes -t -n

After this, dbca shows Oracle Real Application Clusters database option correctly.

For 10.x and 11.1.x dbca to work with 11.2 listener, patch 8288940 needs to be applied. Please refer to Note 948456.1 for other potential issues in such mixed environments.

Case II

   Fix the inventory.xml via detaching the problem ORACLE_HOME, then re-attaching with correct information.

1. To detach ORACLE_HOME, on each cluster node, issue:

$ORACLE_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=<$ORACLE_HOME path>

eg:
$ORACLE_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=/opt/oracle/product/11.2.0/racdb

2. To attach ORACLE_HOME, on each cluster node, issue:

$ORACLE_HOME/oui/bin/runInstaller -silent -local -ignoreSysPrereqs -attachHome ORACLE_HOME=<$ORACLE_HOME path> ORACLE_HOME_NAME=<$ORACLE_HOME_NAME> CLUSTER_NODES=, LOCAL_NODE=

For example, on node 1, issue:

$ORACLE_HOME/oui/bin/runInstaller -silent -attachHome -local ORACLE_HOME="/opt/oracle/product/11.2.0/racdb" ORACLE_HOME_NAME="OraDb11g_home" "CLUSTER_NODES={racnode1,racnode2}" LOCAL_NODE='racnode1'

 on node 2, issue:

$ORACLE_HOME/oui/bin/runInstaller -silent -attachHome -local ORACLE_HOME="/opt/oracle/product/11.2.0/racdb" ORACLE_HOME_NAME="OraDb11g_home" "CLUSTER_NODES={racnode1,racnode2}" LOCAL_NODE='racnode2'

Case III

1. Relink Oracle binary to include RAC option.

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk rac_on ioracle

2. Fix the inventory using the same procedure as case II.

Case IV

Change RDBMS ORACLE_HOME to be owned by oracle user:

As root user:
cd /u01/app/oracle/product/11.2.0
chown oracle dbhome_1

Case V

Manually shutdown the instance DBUA0.
Please ensure there is no instance with name DBUA0 running on any node of the cluster before start dbca. This is not required if there is no earlier abnormal termination of dbca happened.

Case VI

Create the default audit directory under $ORACLE_HOME/rdbms/audit if it is missing, otherwise correct the permission and ownership:

$ mkdir $ORACLE_HOME/rdbms/audit
It should have permission 755 (drwxr-xr-x) and ownership :oinstall

 

Case VII

Either increase the semaphores setting or clear up left over semaphores. Please refer to Note 825001.1 for solution.

Others

1. Reinstall RDBMS oracle software

2. Relinking Grid Infrastructure binaries, refer to
Oracle? Grid Infrastructure Installation Guide
11g Release 2 (11.2) for Linux
6.3 Relinking Oracle Grid Infrastructure for a Cluster Binaries 

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/28612416/viewspace-2128595/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/28612416/viewspace-2128595/