Redhat 5.4 RAC multipath 配置raw，运行root.sh 时报错Failed to upgrade Oracle Cluster Registry configuration

之前打算装Oracle 11gR2的RAC的。 后来因为资源不够，就改装oracle 10g了。 

       raw 是用openfiler 配置 加 Multipath多路径配置的。

      

       raw的配置参考我的Blog：

              Linux 5.4 LVM RAW 设备 配置的深入研究

              http://blog.csdn.net/xujinyang/article/details/6834895

 

这种情况下配置有些注意事项。 否则会在执行root.sh命令的时候，报错：

 

[root@rac1 init.d]# /u01/app/oracle/product/crs/root.sh

WARNING: directory '/u01/app/oracle/product' is not owned by root

WARNING: directory '/u01/app/oracle' is not owned by root

WARNING: directory '/u01/app' is not owned by root

WARNING: directory '/u01' is not owned by root

Checking to see if Oracle CRS stack is already configured

 

Setting the permissions on OCR backup directory

Setting up NS directories

PROT-1: Failed to initialize ocrconfig

Failed to upgrade Oracle Cluster Registry configuration

 

 

一般出现这个问题的原因有几种：

 

（1） Bug

       这种情况参考：

       Executing root.sh errors with "Failed To Upgrade Oracle Cluster Registry Configuration" [ID 466673.1]

       http://blog.csdn.net/xujinyang/article/details/6933166

 

（2） raw 设备的权限问题

              [root@rac1 raw]# pwd

              /dev/raw

              [root@rac1 raw]# ll

              total 0

              crw-r----- 1 root   oinstall 162, 1 Nov 25 15:34 raw1

              crw-r----- 1 root   oinstall 162, 2 Nov 25 15:34 raw2

              crw-r--r-- 1 oracle oinstall 162, 3 Nov 25 13:48 raw3

              crw-r--r-- 1 oracle oinstall 162, 4 Nov 25 13:48 raw4

              crw-r--r-- 1 oracle oinstall 162, 5 Nov 25 13:48 raw5

      

权限要设置成如下：

chown root.oinstall /dev/raw/raw[1-2]
chown oracle.oinstall /dev/raw/raw[3-5]


chmod 775 /dev/raw/raw[1-2] 
chmod 775 /dev/raw/raw[3-5]

 

可以在udev的规则里设置， 也可以把权限添加到/etc/rc.d/rc.local  中。

 

（3） raw 设备读写问题

       这种情况参考：

       Placement of Voting disk and OCR Files in Oracle RAC 10g and 11gR1 [ID 293819.1]             

       http://blog.csdn.net/xujinyang/article/details/6834893

 

清空一下RAW:

[root@rac1 raw]# dd if=/dev/zero of=/dev/mapper/crsp6 bs=1M count=200

dd: writing `/dev/mapper/crsp6': No space left on device

197+0 records in

196+0 records out

205599744 bytes (206 MB) copied, 21.1817 seconds, 9.7 MB/s

说明：

       这里有个提示信息，磁盘空间不足: No space left on device。 这里的情况是， 我的每个raw是200M。 但是实际容量并没有这么多。 所以我清空写200M的时候，会报空间不足。 这种情况会按照实际大小进行清空。 从下面我们可以该raw设备实际大小是197M。

 

[root@rac1 raw]# dd if=/dev/zero of=/dev/mapper/crsp6 bs=1M count=190

190+0 records in

190+0 records out

199229440 bytes (199 MB) copied, 21.9521 seconds, 9.1 MB/s

[root@rac1 raw]# dd if=/dev/zero of=/dev/mapper/crsp5 bs=1M count=195

195+0 records in

195+0 records out

204472320 bytes (204 MB) copied, 22.0362 seconds, 9.3 MB/s

[root@rac1 raw]# dd if=/dev/zero of=/dev/mapper/crsp7 bs=1M count=195

195+0 records in

195+0 records out

204472320 bytes (204 MB) copied, 19.9692 seconds, 10.2 MB/s

[root@rac1 raw]# dd if=/dev/zero of=/dev/mapper/crsp8 bs=1M count=195

195+0 records in

195+0 records out

204472320 bytes (204 MB) copied, 18.5526 seconds, 11.0 MB/s

[root@rac1 raw]# dd if=/dev/zero of=/dev/mapper/crsp9 bs=1M count=195

195+0 records in

195+0 records out

204472320 bytes (204 MB) copied, 21.9439 seconds, 9.3 MB/s

 

 

 

我这次遇到问题的Log如下：

[root@rac1 client]# pwd

/u01/app/oracle/product/crs/log/rac1/client

[root@rac1 client]# ls

ocrconfig_7527.log

[root@rac1 client]# cat ocrconfig_7527.log

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.

2010-11-25 19:54:14.694: [ OCRCONF][1189568]ocrconfig starts...

2010-11-25 19:54:14.696: [ OCRCONF][1189568]Upgrading OCR data

2010-11-25 19:54:15.434: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:15.464: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:15.464: [  OCRRAW][1189568]proprioini: both disks are not OCR formatted

2010-11-25 19:54:15.464: [  OCRRAW][1189568]proprinit: Could not open raw device

2010-11-25 19:54:15.465: [ default][1189568]a_init:7!: Backend init unsuccessful : [26]

2010-11-25 19:54:15.465: [ OCRCONF][1189568]Exporting OCR data to [OCRUPGRADEFILE]

2010-11-25 19:54:15.465: [  OCRAPI][1189568]a_init:7!: Backend init unsuccessful : [33]

2010-11-25 19:54:15.465: [ OCRCONF][1189568]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]

2010-11-25 19:54:16.072: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:16.075: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:16.075: [  OCRRAW][1189568]proprioini: both disks are not OCR formatted

2010-11-25 19:54:16.075: [  OCRRAW][1189568]proprinit: Could not open raw device

2010-11-25 19:54:16.076: [ default][1189568]a_init:7!: Backend init unsuccessful : [26]

2010-11-25 19:54:16.231: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:16.232: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:16.265: [  OCRRAW][1189568]ibctx:1:ERROR: INVALID FORMAT

2010-11-25 19:54:16.265: [  OCRRAW][1189568]proprinit:problem reading the bootblock or superbloc 22

 

2010-11-25 19:54:16.646: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:16.647: [  OCRRAW][1189568]propriogid:1: INVALID FORMAT

2010-11-25 19:54:16.757: [  OCRRAW][1189568]propriowv: Vote information on disk 0 [/dev/raw/raw1] is adjusted from [0/0] to [1/2]

2010-11-25 19:54:16.762: [  OCRRAW][1189568]propriowv: Vote information on disk 1 [/dev/raw/raw2] is adjusted from [0/0] to [1/2]

2010-11-25 19:54:17.193: [  OCRRAW][1189568]propriniconfig:No 92 configuration

2010-11-25 19:54:17.193: [  OCRAPI][1189568]a_init:6a: Backend init successful

2010-11-25 19:54:18.263: [ OCRCONF][1189568]Initialized DATABASE keys in OCR

2010-11-25 19:54:18.478: [ OCRCONF][1189568]csetskgfrblock0: clsfmt returned with error [4].

2010-11-25 19:54:18.478: [ OCRCONF][1189568]Failure in setting block0 [-1]

2010-11-25 19:54:18.478: [ OCRCONF][1189568]OCR block 0 is not set !

2010-11-25 19:54:18.479: [ OCRCONF][1189568]Exiting [status=failed]...

 

 

这个是Multipath 的bug 问题，就是我们遇到的第一种情况：

 

（1） 用clsfmt检查raw 设备

[root@rac1 bin]# ./clsfmt.bin ocr /dev/raw/raw1

clsfmt: Failure trying to resize OCR file from 205567488 bytes to 205570048 bytes

clsfmt: Received unexpected error 4 from skgfifi

skgfifi: Additional information: -2

Additional information: 205567488

 

这里没有成功， 就是bug。

 

（2）用Pathch中的新文件替换clsfmt.bin 文件。

       Patch 是：p4679769。 这个需要Oracle 帐号才能下载。 所以我将Linux 平台下32位和64位的path都上传到csdn的下载里了。 有遇到相同问题的朋友自己去下载。 下载地址：

       http://download.csdn.net/source/2862150

 

 

 

（3）再次测试：

[root@rac1 bin]# ./clsfmt ocr /dev/raw/raw1

clsfmt: Failure trying to resize OCR file from 205567488 bytes to 205570048 bytes

clsfmt: successfully initialized file /dev/raw/raw1

[root@rac1 bin]# ./clsfmt ocr /dev/raw/raw2

clsfmt: Failure trying to resize OCR file from 205599744 bytes to 205602816 bytes

clsfmt: successfully initialized file /dev/raw/raw2

[root@rac1 bin]# ./clsfmt ocr /dev/raw/raw3

clsfmt: Failure trying to resize OCR file from 205599744 bytes to 205602816 bytes

clsfmt: successfully initialized file /dev/raw/raw3

[root@rac1 bin]# ./clsfmt ocr /dev/raw/raw4

clsfmt: Failure trying to resize OCR file from 205599744 bytes to 205602816 bytes

clsfmt: successfully initialized file /dev/raw/raw4

[root@rac1 bin]# ./clsfmt ocr /dev/raw/raw5

clsfmt: Failure trying to resize OCR file from 205599744 bytes to 205602816 bytes

clsfmt: successfully initialized file /dev/raw/raw5

 

 

bug 修复成功了。

 

（4） 清空raw设备

 

因为我们已经执行过root.sh 命令，会有相关信息写到raw设备里。 用dd命令清空这些信息。 最好留一点空间。 因为在Solaris平台上有预留 1M 空间的做法，不然会出现问题。

 

[root@rac1 rac1]# dd if=/dev/zero of=/dev/raw/raw1 bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 17.4534 seconds, 6.0 MB/s

[root@rac1 rac1]# dd if=/dev/zero of=/dev/raw/raw2 bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 11.4307 seconds, 9.2 MB/s

[root@rac1 rac1]# dd if=/dev/zero of=/dev/raw/raw3 bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 18.4136 seconds, 5.7 MB/s

[root@rac1 rac1]# dd if=/dev/zero of=/dev/raw/raw4 bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 17.821 seconds, 5.9 MB/s

[root@rac1 rac1]# dd if=/dev/zero of=/dev/raw/raw5 bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 17.3289 seconds, 6.1 MB/s

 

 

如果不清空raw 设备，直接执行root.sh 命令，还是会报错误：

[root@rac1 bin]# /u01/app/oracle/product/crs/root.sh

WARNING: directory '/u01/app/oracle/product' is not owned by root

WARNING: directory '/u01/app/oracle' is not owned by root

WARNING: directory '/u01/app' is not owned by root

WARNING: directory '/u01' is not owned by root

Checking to see if Oracle CRS stack is already configured

 

Setting the permissions on OCR backup directory

Setting up NS directories

PROT-1: Failed to initialize ocrconfig

Failed to upgrade Oracle Cluster Registry configuration

 

错误的log 日志如下：

#cat /u01/app/oracle/product/crs/log/rac1/client/ocrconfig_16159.log

 

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.

2010-11-25 23:27:25.468: [ OCRCONF][2118112]ocrconfig starts...

2010-11-25 23:27:25.469: [ OCRCONF][2118112]Upgrading OCR data

2010-11-25 23:27:25.558: [  OCRRAW][2118112]propriogid:1: INVALID FORMAT

2010-11-25 23:27:25.559: [  OCRRAW][2118112]proprioini: disk 0 (/dev/raw/raw1) doesn't have enough votes (1,2)

2010-11-25 23:27:25.559: [  OCRRAW][2118112]proprinit: Could not open raw device

2010-11-25 23:27:25.560: [ default][2118112]a_init:7!: Backend init unsuccessful : [26]

2010-11-25 23:27:25.560: [ OCRCONF][2118112]Exporting OCR data to [OCRUPGRADEFILE]

2010-11-25 23:27:25.560: [  OCRAPI][2118112]a_init:7!: Backend init unsuccessful : [33]

2010-11-25 23:27:25.560: [ OCRCONF][2118112]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]

2010-11-25 23:27:25.746: [  OCRRAW][2118112]propriogid:1: INVALID FORMAT

2010-11-25 23:27:25.746: [  OCRRAW][2118112]proprioini: disk 0 (/dev/raw/raw1) doesn't have enough votes (1,2)

2010-11-25 23:27:25.746: [  OCRRAW][2118112]proprinit: Could not open raw device

2010-11-25 23:27:25.747: [ default][2118112]a_init:7!: Backend init unsuccessful : [26]

2010-11-25 23:27:26.112: [  OCRRAW][2118112]propriogid:1: INVALID FORMAT

2010-11-25 23:27:26.112: [  OCRRAW][2118112]proprioini: disk 0 (/dev/raw/raw1) doesn't have enough votes (1,2)

2010-11-25 23:27:26.112: [  OCRRAW][2118112]proprinit: Could not open raw device

2010-11-25 23:27:26.120: [  OCRAPI][2118112]a_init:6b!: Backend init unsuccessful : [26]

2010-11-25 23:27:26.120: [ OCRCONF][2118112]Failed to initialized OCR context. error:[PROC-26: Error while accessing the physical storage]

2010-11-25 23:27:26.120: [ OCRCONF][2118112]Exiting [status=failed]...

 

 

（5）清空raw 设备后，在执行/u01/app/oracle/product/crs/root.sh 命令

       这次成功执行。

 

 

------------------------------------------------------------------------------