Oracle RAC集群19.3升级19.7

1.Opatch升级

Opatch版本需求12.2.0.1.19或更高版本
下载Opatch升级包 到/oracle/soft目录下权限775

节点一：

GI Opatch升级：
//root用户执行，grid用户权限不足
//移动旧版本OPatch
[root@rac1 crs_1]# mv \$ORACLE_HOME/OPatch /tmp/OPatch_grid

//解压也需要用root解压，grid用户权限不足
[root@rac1 soft]# unzip p6880880_190000_Linux-x86-64.zip -d /oracle/grid/crs_1/

//修改权限用户属组为grid.oinstall
[root@rac1 crs_1]# chown grid.oinstall OPatch/ -R 

//版本查看：
[grid@rac1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.


数据库 Opatch升级：
//用oracle用户即可
//移动旧版本OPatch
[oracle@rac1 db_1]$ mv \$ORACLE_HOME/OPatch/ /tmp/OPatch_oracle

//解压新版本OPatch
[oracle@rac1 soft]$unzip p6880880_190000_Linux-x86-64.zip -d $ORACLE_HOME

//版本查看：
[oracle@rac1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.

节点二：

GI Opatch升级：
//root用户执行，grid用户权限不足
//移动旧版本OPatch
[root@rac2 crs_1]# mv \$ORACLE_HOME/OPatch /tmp/OPatch_grid

//解压也需要用root解压，grid用户权限不足
[root@rac2 soft]# unzip p6880880_190000_Linux-x86-64.zip -d /oracle/grid/crs_1/

修改权限用户属组为grid.oinstall
[root@rac2 crs_1]# chown grid.oinstall OPatch/ -R 

版本查看：
[grid@rac2 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.


数据库 Opatch升级：
//用oracle用户即可
//移动旧版本OPatch
[oracle@rac2 db_1]$ mv \$ORACLE_HOME/OPatch/ /tmp/OPatch_oracle

//解压新版本OPatch
[oracle@rac2 soft]$unzip p6880880_190000_Linux-x86-64.zip -d $ORACLE_HOME

版本查看：
[oracle@rac2 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.

2.检查GI主页和数据库主页的清单信息的一致性

以各自的Oracle Home所有者身份运行以下命令以检查一致性。
节点一：

[grid@rac1 OPatch]$ ./opatch lsinventory -detail -oh $ORACLE_HOME

[Oracle@rac1 OPatch]$ ./opatch lsinventory -detail -oh $ORACLE_HOME

节点二：

[grid@rac2 OPatch]$ ./opatch lsinventory -detail -oh $ORACLE_HOME

[Oracle@rac2 OPatch]$ ./opatch lsinventory -detail -oh $ORACLE_HOME

3.下载并解压缩补丁

p30899722_190000_Linux-x86-64.zip

节点一：
[grid@rac1 soft]$ unzip p30899722_190000_Linux-x86-64.zip 

节点二：
[grid@rac2 soft]$ unzip p30899722_190000_Linux-x86-64.zip

4.进行OPatch冲突检测

GI：

[grid@rac1 soft]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/30869156/

[grid@rac1 soft]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/30894985/

[grid@rac1 soft]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/30869304/

[grid@rac1 soft]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/30898856/

[grid@rac1 soft]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/

Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home    : /oracle/grid/crs_1
Central Inventory : /oracle/oraInventory
  from      : /oracle/grid/crs_1/oraInst.loc
OPatch version  : 12.2.0.1.21
OUI version    : 12.2.0.7.0
Log file location : /oracle/grid/crs_1/cfgtoollogs/opatch/opatch2020-06-27_15-33-38PM_1.log
This command doesn't support System Patch.
OPatch failed with error code 21

 

根据opatch CheckSystemSpace Command For Grid Infrastructure RU Fails With: "This command doesn't support System Patch"
 (Doc ID 2634165.1) //不需要检测

数据库：

[oracle@rac1 30899722]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/30869156/
[oracle@rac1 30899722]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 30899722/30894985/

5.运行OPatch SystemSpace检查

节点一：

对于GI主目录：

1.创建/tmp/patch_list_gihome.txt 具有以下内容的文件：
/oracle/soft/30899722/30869156
/oracle/soft/30899722/30894985
/oracle/soft/30899722/30869304
/oracle/soft/30899722/30898856
/oracle/soft/30899722/

 

2.运行opatch命令以检查Grid Infrastructure Home中是否有足够的可用空间：

[grid@rac1 ~]$ $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
/tmp/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home    : /oracle/grid/crs_1
Central Inventory : /oracle/oraInventory
  from      : /oracle/grid/crs_1/oraInst.loc
OPatch version  : 12.2.0.1.21
OUI version    : 12.2.0.7.0
Log file location : /oracle/grid/crs_1/cfgtoollogs/opatch/opatch2020-06-27_23-32-24PM_1.log
This command doesn't support System Patch.
OPatch failed with error code 21


解决：根据Mos2634165.1 修改/tmp/patch_list_gihome.txt文件

/oracle/soft/30899722/30869156
/oracle/soft/30899722/30894985
/oracle/soft/30899722/30869304
/oracle/soft/30899722/30898856

再次执行即可。
[grid@rac1 ~]$ $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt

对于数据库主目录：

1.创建/tmp/patch_list_dbhome.txt 具有以下内容的文件：
/oracle/soft/30899722/30869156
/oracle/soft/30899722/30894985

 

2.运行opatch命令以检查数据库主页中是否有足够的可用空间：
[oracle@rac1 ~]$$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt


如果OPatch报告Prereq "checkSystemSpace" failed.，则清理系统空间，因为所需的空间量不可用。
如果OPatch报告Prereq "checkSystemSpace" passed.，则无需采取任何措施。进行补丁安装。

节点二同节点一做相同操作。

6.一次性补丁冲突检测与解决

使用root用户执行：

[root@rac1 OPatch]# ./opatchauto apply /oracle/soft/30899722/ -analyze
OPatchauto session is initiated at Sat Jun 27 23:49:53 2020
System initialization log file is /oracle/grid/crs_1/cfgtoollogs/opatchautodb/systemconfig2020-06-27_11-50-00PM.log.
Session log file is /oracle/grid/crs_1/cfgtoollogs/opatchauto/opatchauto2020-06-27_11-52-11PM.log
The id for this session is 9MKG
Executing OPatch prereq operations to verify patch applicability on home /oracle/grid/crs_1
Executing OPatch prereq operations to verify patch applicability on home /oracle/app/product/19.3.0/db_1
Patch applicability verified successfully on home /oracle/app/product/19.3.0/db_1
Patch applicability verified successfully on home /oracle/grid/crs_1
Verifying SQL patch applicability on home /oracle/app/product/19.3.0/db_1
SQL patch applicability verified successfully on home /oracle/app/product/19.3.0/db_1
OPatchAuto successful.
--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:
Host:rac1
RAC Home:/oracle/app/product/19.3.0/db_1
Version:19.0.0.0.0
==Following patches were SKIPPED:
Patch: /oracle/soft/30899722/30869304
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /oracle/soft/30899722/30898856
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /oracle/soft/30899722/30894985
Log: /oracle/app/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-27_23-53-28PM_1.log
Patch: /oracle/soft/30899722/30869156
Log: /oracle/app/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-27_23-53-28PM_1.log
Host:rac1
CRS Home:/oracle/grid/crs_1
Version:19.0.0.0.0
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /oracle/soft/30899722/30894985
Log: /oracle/grid/crs_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-27_23-53-27PM_1.log
Patch: /oracle/soft/30899722/30869304
Log: /oracle/grid/crs_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-27_23-53-27PM_1.log
Patch: /oracle/soft/30899722/30898856
Log: /oracle/grid/crs_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-27_23-53-27PM_1.log
Patch: /oracle/soft/30899722/30869156
Log: /oracle/grid/crs_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-27_23-53-27PM_1.log
OPatchauto session completed at Sat Jun 27 23:57:26 2020
Time taken to complete the session 7 minutes, 34 seconds

节点二同节点一做相同操作。

7.自动补丁安装

[root@rac1 OPatch]# ./opatchauto apply /oracle/soft/30899722

[root@rac2 OPatch]# ./opatchauto apply /oracle/soft/30899722

//使用自动打补丁方式，不需要手动将修改后的SQL文件加载到数据库。

//补丁安装完成后，查看补丁信息。
[grid@rac1 OPatch]$ ./opatch lsinv
Patch 30894985   : applied on Sun Jun 28 00:50:04 CST 2020
Unique Patch ID: 23451591
Patch description: "OCW RELEASE UPDATE 19.7.0.0.0 (30894985)"
  Created on 10 Apr 2020, 05:35:01 hrs PST8PDT
Patch 30869304   : applied on Sun Jun 28 00:48:22 CST 2020
Unique Patch ID: 23391523
Patch description: "ACFS RELEASE UPDATE 19.7.0.0.0 (30869304)"
  Created on 16 Feb 2020, 07:11:33 hrs PST8PDT
Patch 30869156   : applied on Sun Jun 28 00:39:18 CST 2020
Unique Patch ID: 23493838
Patch description: "Database Release Update : 19.7.0.0.200414 (30869156)"
  Created on 6 Apr 2020, 23:20:53 hrs PST8PDT

8.补丁安装时需要的问题

问题一：

节点一成功

节点二失败：

Start applying binary patch on home /oracle/grid/crs_1
Failed while applying binary patches on home /oracle/grid/crs_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : rac2->/oracle/grid/crs_1 Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /oracle/grid/crs_1, host: rac2.
Command failed: /oracle/grid/crs_1/OPatch/opatchauto apply /oracle/soft/30899722 -oh /oracle/grid/crs_1 -target_type cluster -binary -invPtrLoc /oracle/grid/crs_1/oraInst.loc -jre /oracle/grid/crs_1/OPatch/jre -persistresult /oracle/grid/crs_1/opatchautocfg/db/sessioninfo/sessionresult_rac2_crs_1.ser -analyzedresult /oracle/grid/crs_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_rac2_crs_1.ser
Command failure output:

==Following patches FAILED in apply:
Patch: /oracle/soft/30899722/30869156
Log: /oracle/grid/crs_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-06-28_01-27-06AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 
After fixing the cause of failure Run opatchauto resume
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Sun Jun 28 01:36:27 2020
Time taken to complete the session 31 minutes, 32 seconds
opatchauto failed with error code 42

故障解决solution：
观察节点1与节点2 /oracle/oraInventory/ContentsXML/oui-patch.xml权限差异，并将节点二修改成与节点一一致。

问题二：

./opatchauto resume 继续执行

[root@rac2 OPatch]# ./opatchauto resume
Start applying binary patch on home /oracle/grid/crs_1
Failed while applying binary patches on home /oracle/grid/crs_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : rac2->/oracle/grid/crs_1 Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /oracle/grid/crs_1, host: rac2.
Command failed: /oracle/grid/crs_1/OPatch/opatchauto apply /oracle/soft/30899722 -oh /oracle/grid/crs_1 -targerid/crs_1/OPatch/jre -persistresult /oracle/grid/crs_1/opatchautocfg/db/sessioninfo/sessionresult_rac2_crs_1.seranalyze_rac2_crs_1.ser
Command failure output:
==Following patches FAILED in apply:
Patch: /oracle/soft/30899722/30869156
Log: 
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
  ORACLE_HOME/inventory/oneoffs/30869156 is corrupted. PatchObject constructor: Input file "/oracle/grid/crs_1//oneoffs/30869156/etc/config/inventory" does not exist. 
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Sun Jun 28 10:44:27 2020
Time taken to complete the session 0 minute, 50 seconds
 opatchauto failed with error code 42

查看补丁信息：(也出现错误)

[grid@rac2 OPatch]$ ./opatch lsinventory 
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home    : /oracle/grid/crs_1
Central Inventory : /oracle/oraInventory
  from      : /oracle/grid/crs_1/oraInst.loc
OPatch version  : 12.2.0.1.21
OUI version    : 12.2.0.7.0
Log file location : /oracle/grid/crs_1/cfgtoollogs/opatch/opatch2020-06-28_10-54-39AM_1.log
Lsinventory Output file location : /oracle/grid/crs_1/cfgtoollogs/opatch/lsinv/lsinventory2020-06-28_10-54-39AM.txt
\--------------------------------------------------------------------------------
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
LsInventorySession failed: Unable to create patchObject
Possible causes are:
  ORACLE_HOME/inventory/oneoffs/30869156 is corrupted. PatchObject constructor: Input file "/oracle/grid/crs_1/inventory/oneoffs/30869156/etc/config/actions" or "/oracle/grid/crs_1/inventory/oneoffs/30869156/etc/config/inventory" does not exist.


故障解决solution：
根据MOS2286025.1 solution：
从节点1拷贝
[grid@rac1 oneoffs]$ scp -r 30869156/  rac2:/oracle/grid/crs_1/inventory/oneoffs/
在节点2再次resume

问题三:

[grid@rac2 OPatch]$ ./opatchauto resume

OPatchauto session is initiated at Sun Jun 28 11:10:27 2020
Session log file is /oracle/grid/crs_1/cfgtoollogs/opatchauto/opatchauto2020-06-28_11-10-28AM.log
Resuming existing session with id V8YC
OPATCHAUTO-68116: Failed to save patch session.
OPATCHAUTO-68116: Failed to save patch session file /oracle/grid/crs_1/opatchautocfg/db/sessioninfo/V8YC.json
OPATCHAUTO-68116: Check log for more information.
OPatchAuto failed.
OPatchauto session completed at Sun Jun 28 11:10:37 2020
Time taken to complete the session 0 minute, 11 seconds
 opatchauto failed with error code 42

 

再次执行一次错误消失：
[root@rac2 OPatch]# ./opatchauto resume
OPatchauto session is initiated at Sun Jun 28 17:28:56 2020
Session log file is /oracle/grid/crs_1/cfgtoollogs/opatchauto/opatchauto2020-06-28_05-28-56PM.log
Resuming existing session with id V8YC
Start applying binary patch on home /oracle/grid/crs_1
Binary patch applied successfully on home /oracle/grid/crs_1
Checking shared status of home.....
Starting CRS service on home /oracle/grid/crs_1
CRS service started successfully on home /oracle/grid/crs_1
Preparing home /oracle/app/product/19.3.0/db_1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /oracle/app/product/19.3.0/db_1
SQL patch applied successfully on home /oracle/app/product/19.3.0/db_1
OPatchAuto successful.