升级Oracle 12C+ PSU补丁

升级Oracle 12C+ PSU补丁

1. 补丁和版本变化信息

Release Schedule of Current Database Releases (Doc ID 742060.1)

1.1 版本编号的变化

从2018年开始，版本号重新的模式，使用3个字段格式( Year.Update.Revision)

• the feature release designation of the database software (the first field) 特性产品版本
• the quarterly Update (the second field) 季度更新版本号
• the quarterly Revision (the third field) 季度修复版本号

1.2 补丁信息变化

1.2.1 补丁种类

名称	说明
Update (Release Update)	2017年7月引入补丁修复策略
Revision (Release Update Revision)	补丁集修订
BP(Bundle Patch) 捆绑补丁	Windows平台修复补丁包
Final CPU
On-Request	通过SR发起请求提供补丁
PSU (Patch Set Update) 补丁集更新	季度补丁，包含功能更新，安全修复累积补丁包
SPU (Security Patch Update) 安全修补程序更新
Overlay SPU
NA (Not Applicable)
CPU (Critical Patch Update)	关键安全更新，用以修复安全漏洞
DBBP (Database Bundle Patch)	(Database Proactive Bundle Patch)是多个PSU的超集， 包含了GI PSU、DB PSU及EXPDATA和DBIM的相关修复。 dba_registry_sqlpatch视图查看版本信息

1.2.2 补丁更新发布及常见问题(2285040.1)

• Oracle DB和GI从12.2开始（2017年7月）Oracle已针对数据库软件发布过程过渡到更灵活，响应速度更快的策略（RU/RUR）
• 数据库11.2和12.1继续使用旧补丁集更新和捆绑补丁（PSU/BP）的更新修复策略。

1.2.3 Release Updates（RU） and Release Update Revisions（RUR）

从计划于2018年发布的下一个数据库版本（最初指定为12.2.0.2）开始，每年将提供数据库产品的新功能版本，并且不再发布补丁集。

1）对比说明

• RU： 为了同时支持与安全相关的修补程序和对每个功能版本的高优先级非安全修补程序，分别在每年的1月，4月，7月和10月提供季度发布更新。季度补丁包，它包含查询优化器修复、功能修复、安全修复和回退修复。
• RUR：除每季度更新（RU）外，还将在同一季度发布发行版本更新修复（即RUR），用以修复上一个RU已知问题，并包括最新的安全漏洞修复。Revision 1滞后3个月，而Revision 2滞后6个月。季度补丁包的修复，包含安全修复和回退修复。整合最近的两个 RU，将稳定安全可靠的修正整合发布给用户
• 在确保新的补丁程序级别必须是先前补丁程序的超级集（新补丁包包含之前的所有修复）的情况下，RU和RUR可以相互切换。建议使用一致的升级策略。
• 查询优化器修复默认是禁用的，它可能会改变执行计划。（Doc ID 2147007.1）说明如何开启查询优化器修复。

总结：RU注重安全和功能修补，而RUR主安全漏洞更新。

2）命名格式

• Release Update - Database Release Update 12.2.0.1.
• Release Update Revision - Database Release Update Revision 12.2.0.1.

3) Release Updates（RU）

Oracle的季度更新包含针对客户最有可能遇到的错误的修复程序：

• 默认情况下，更新中默认禁用了查询优化器错误修复程序，这些错误修复程序是早期版本中PSU和BP不允许的。
• 更新包括针对安全漏洞的修复程序。
• 更新经过Oracle的广泛测试，涵盖功能，压力，性能和破坏性测试方案。
• 及时应用更新会减少重新发现已知问题的可能性。
• 可以通过RAC滚动以零停机时间安装更新。

4）Release Update Revisions（RUR）

除每季度更新外，还将在同一季度发布发行版本更新修复（修复），以利用已知的修复来扩充更新，并包括最新的安全漏洞修复。

• 在发布此更新后的六个月中，每个更新最多包含两个单独的修复版。例如，Release.Update .1和Release.Update .2，其中“ 1”和“ 2”代表修复版本。
• 保守软件维护方法，可能会遇到最新更新（RU）中已解决的已知问题的风险。

5）补丁更新线路

• 12.2+：求稳派可以选择RUR升级线路，通常情况可以选取走RU升级更新路线
• 11.2.0.4：继续沿用旧的升级策略，即Patch Set Updates (PSU)
• 其它老版本：可以选择计划升级到原厂支持的版本（很多老版本基本都不提供新补丁）

1.2.4 查看版本信息

-- on windows 用 findstr /i 替换 grep -i

opatch lsinventory

opatch lsinventory -bugs_fixed | egrep -i 'PSU|DATABASE PATCH SET UPDATE'

-- For CRS (Cluster Ready Services) PSUs, enter the following command
opatch lsinventory -bugs_fixed | grep -i 'TRACKING BUG' | grep -i 'PSU'

-- For GI (Grid Infrastructure) PSUs, enter the following command
opatch lsinventory -bugs_fixed | grep -i 'GRID INFRASTRUCTURE PATCH SET UPDATE'

-- For Enterprise Manager OMS PSUs, enter the following command:
opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER' | grep -i 'OMS'

-- For Enterprise Manager Agent PSUs, enter the following command:
opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER' | grep -i 'AGENT'

1.3 补丁下载地址

# 1. 可以依据需求选择下载补丁版本
Assistant: Download Reference for Oracle Database/GI PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (文档 ID 2118136.2) 

#  最新发布补丁（good）
Master Note for Database Proactive Patch Program (Doc ID 756671.1)

# OPatch工具
https://updates.oracle.com/download/6880880.html

# 关键补丁更新和安全公告
https://www.oracle.com/security-alerts/#CriticalPatchUpdates
# 最近一次
Critical Patch Update (CPU) Program Apr 2020 Patch Availability Document (PAD) (Doc ID 2633852.1)

# PSU 补丁下载地址文档
Patch Set Updates for Oracle Products (Doc ID 854428.1)

1.4 OPatch工具

OPatch

1.4.1 改变

OCM（Oracle Configuration Manager）：响应文件

从OPatch 12.2.0.1.5和11.2.0.3.14版本开始之后，已经不需要-ocmrf选项，即（不再需要OCM）Document 2161861.1。

在历史版本，需要使用emocmrsp工具创建响应文件

su - grid
$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /tmp/ocm.rsp

su - grid
[[ -f "${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp" ]] && (sleep 3;echo -e '\n';sleep 3;echo -e 'Y\n')|${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -no_banner -output /tmp/ocm.rsp

1.4.2 增强

在12.2.0.1.5 和 11.2.0.3.14 版本及以后版本增强了补丁处理的效率

• 在过去，当 OPatch 识别到正在安装的补丁是一个已经存在的补丁的超集时，它会在安装新补丁前回滚子集补丁，然后安装新补丁。
• 现在，当 Opatch 识别到被安装的补丁是一个已经存在的补丁的超集，Opatch 不会在安装新超集补丁前，对已经安装的补丁执行回滚。
  • 安装新的超集补丁，在应用过程中禁用子集补丁
  • 回滚过程重新激活子集补丁
  • Running the Post De-install (datapatch) the July OJVM PSU is rolled back and the April OJVM PSU is applied (12.1.x only)

1.4.3 存在疑问

1）在Opatch 11.2.3.21版本升级过程需要OCM

当前版本

升级错误

处理

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /tmp/ocm.rsp
opatch apply -local -ocmrf /tmp/ocm.rsp

1.5 OJVM PSU

Oracle OJVM组件作为数据库PSU补丁更新的一部分。它包含JDBC（客户端）修补和JavaVM组件修复。缓解补丁（Mitigation Patch）可防止数据库中所有当前已知的Oracle JavaVM安全漏洞。OJVM PSU 依赖DB PSU

2217053.1

• 在2017年1月发布的OJVM PSU补丁集之前，先将OJVM PSU补丁集安装到$ORACLE_HOME二进制文件，然后startup upgrade这种受限模式下应用OJVM补丁到各个数据库中。【在RAC环境中，cluster_database应将参数设置为FALSE，以便STARTUP UPGRADE】
• 从11.2.0.4和12.1.0.2的2017年1月OJVM PSU补丁集开始，可以有条件"滚动安装"方式
• OJVM PSU独立的安装

1.5.1 OJVM补丁安装步骤

1）常规安装步骤（首选）

• 关闭所有节点上的数据库和服务
• 如果在Windows平台上，请回滚旧的OJVM补丁
• 应用DB PSU（不要安装OJVM后再运行等效版本的DB PSU）
• 应用OJVM PSU补丁
• 应用JDBC补丁
• 更新数据字典
• 重启数据库服务

2）OJVM的缓解补丁安装（通过视图JAVA_DEV_STATUS查看是否安装)

• 关闭本地节点上的实例和服务

• 应用DB PSU

• 应用缓解补丁

• （可选）应用JDBC补丁

• 更新数据字典

• 在修补后的主目录中的所有数据库上执行dbms_java_dev.disable，禁用JAVA

• 重新启动所有实例和服务

  下载地址: Patch 19721304

3）滚动安装(2217053.1)

• 适用于2017年1月或更晚的所有OJVM PSU
• 先决条件：只有在Oracle RAC一节点配置或具有至少两个实例的Oracle RAC数据库中。
• 判断是否安装OJVM
  • SELECT version, status FROM dba_registry WHERE comp_id='JAVAVM';
• 判断是否使用OJVM
  • select count(*) from x$kglob where KGLOBTYP = 29 OR KGLOBTYP = 56;
  • 通过v$session视图查看

col service_name format a20
col username format a20
col program format a20
set num 8

select sess.service_name, sess.username,sess.program, count(*)
from
v$session sess,
dba_users usr,
x$kgllk lk,
x$kglob
where kgllkuse=saddr
and kgllkhdl=kglhdadr
and kglobtyp in (29,56)
and sess.user# = usr.user_id
and usr.oracle_maintained = 'N'      #### omit this line on 11.2.0.4
group by sess.service_name, sess.username, sess.program
order by sess.service_name, sess.username, sess.program;

1.6 关键补丁和安全补丁更新确认及其修复补丁

1. 打开网址查看安全公告（https://www.oracle.com/security-alerts/）

   

2. 找到对应产品CPU建议

3. 从目录跳到对应章节

4. OJVM Release Update 19.7.0.0.200414 Patch 30805684 for all platforms修复CVE-2020-2735l漏洞

2. PSU补丁升级步骤

2.1 环境信息

操作系统版本	rhel7.6 x86_64
数据库版本	12.2.0.1+
节点数量	2
待应用补丁	RU: 26610291 (GRID INFRASTRUCTURE RELEASE UPDATE 12.2.0.1.170814)

2.2 实施流程

• 检查当前数据库服务
• 上传相关补丁文件
• 解压补丁文件到工作区
• 冲突检查
• 调研并编写实施方案
• 申请实施作业窗口
• 数据备份
• 升级补丁
• 检查确认

2.3 调研

2.4 补丁升级详情

2.4.1 前期准备工作

1) CVU工具检查

Cluster Verify Utility(CVU): Oracle 提供了一个校验工具,主要功能是检查系统的硬件和软件环境是否满足安装的要求。CVU主要包括2个脚本（${GI_HOME}/runcluvfy.sh和${GI_HOME}/bin/cluvfy）

su - grid
export REMOTE_NODE_NAME='o19c1,o19c2,o19c3'
./runcluvfy.sh stage -pre crsinst -n ${REMOTE_NODE_NAME} -verbose >/tmp/cvu_chk.log
or
./runcluvfy.sh stage -pre crsinst -n ${REMOTE_NODE_NAME} -fixup -verbose 
# -fixup: 新增的参数，产生一个名为runfixup.sh的修复脚本


# 语法
./runcluvfy.sh -help
USAGE:
runcluvfy.sh [-help|-version]
runcluvfy.sh stage {-list|-help}
runcluvfy.sh stage {-pre|-post}    [-verbose]
runcluvfy.sh comp  {-list|-help}
runcluvfy.sh comp     [-verbose]

2）数据备份（若接入备份）

提前发起全备，升级前增备

3）解压软件包

# 1. opatch工具
# 1.1 GI OPatch
su - root
.  /home/grid/.bash_profile
export GI_HOME=${ORACLE_HOME}
export UNZIPPED_PATCH_LOCATION=/ups/soft

mv ${GI_HOME}/OPatch ${GI_HOME}/OPatch_$(date +%Y%m%d)
cd ${UNZIPPED_PATCH_LOCATION}
unzip -qo p6880880_*_Linux-x86-64.zip -d ${GI_HOME}
chown -R grid:oinstall ${GI_HOME}/OPatch
chmod -R +x ${GI_HOME}/OPatch
ls -ld ${GI_HOME}/OPatch*
su - grid -c "${GI_HOME}/OPatch/opatch version"

# 1.2 DB OPatch
su - oracle
export DB_HOME=${ORACLE_HOME}
mv ${DB_HOME}/OPatch ${DB_HOME}/OPatch_$(date +%Y%m%d)
export UNZIPPED_PATCH_LOCATION=/ups/soft
cd ${UNZIPPED_PATCH_LOCATION}
unzip -qo p6880880_*_Linux-x86-64.zip -d ${DB_HOME}
ls -ld ${DB_HOME}/OPatch*
${ORACLE_HOME}/OPatch/opatch version

# 2. PSU
unzip -qo p26610291_122010_Linux-x86-64.zip -d /ups/soft/

3）冲突检查

# 1. 
su - grid
export UNZIPPED_PATCH_LOCATION=/ups/soft
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/28822515
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/28870605
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/28864846
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/26839277
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/28566910 

cat >> ${UNZIPPED_PATCH_LOCATION}/patch_list_gihome.txt << EOF
${UNZIPPED_PATCH_LOCATION}/28828733/28822515
${UNZIPPED_PATCH_LOCATION}/28828733/28870605
${UNZIPPED_PATCH_LOCATION}/28828733/28864846
${UNZIPPED_PATCH_LOCATION}/28828733/26839277
${UNZIPPED_PATCH_LOCATION}/28828733/28566910
EOF

# 检查文件系统空间(可选)
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile ${UNZIPPED_PATCH_LOCATION}/patch_list_gihome.txt
$ORACLE_HOME/OPatch/opatch prereq CheckSystemCommandAvailable -phBaseFile ${UNZIPPED_PATCH_LOCATION}/patch_list_gihome.txt

# 2. 
su - oracle
export UNZIPPED_PATCH_LOCATION=/ups/soft
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/28822515
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ${UNZIPPED_PATCH_LOCATION}/28828733/28870605 
cat >> ${UNZIPPED_PATCH_LOCATION}/patch_list_dbhome.txt << EOF
${UNZIPPED_PATCH_LOCATION}/28828733/28822515
${UNZIPPED_PATCH_LOCATION}/28828733/28870605
EOF

# 检查文件系统空间(可选)
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile ${UNZIPPED_PATCH_LOCATION}/patch_list_dbhome.txt
$ORACLE_HOME/OPatch/opatch prereq CheckSystemCommandAvailable -phBaseFile ${UNZIPPED_PATCH_LOCATION}/patch_list_dbhome.txt

4）补丁应用分析（仅分析不应用更新）

su - root
. /home/grid/.bash_profile && export GI_HOME=${ORACLE_HOME} && export UNZIPPED_PATCH_LOCATION=/ups/soft
${GI_HOME}/OPatch/opatchauto apply ${UNZIPPED_PATCH_LOCATION}/28828733 -analyze
OR
${GI_HOME}/OPatch/opatchauto rollback ${UNZIPPED_PATCH_LOCATION}/28828733 -analyze

2.4.2 GI PSU补丁

1) 处理Bug(19.X)

During 19.x GI installation, the file 'oui-patch.xml' will be created under the central inventory directory on the OUI node (node where gridSetup.sh was invoked) but not on the other nodes

除节点外的其它节点因缺少oui-patch.xml文件引起PSU升级失败。

# 解决方案
export REMOTE_NODE_NAME='o19c2,o19c3'
export INV_DIR="$(grep 'inventory_loc' /etc/oraInst.loc|awk -F= '{print $NF}')/ContentsXML"
for _host in $(echo ${REMOTE_NODE_NAME}|sed 's/,/ /g');do
	scp ${INV_DIR}/oui-patch.xml  ${_host}:${INV_DIR}/oui-patch.xml
	ssh ${_host} "chmod 660 ${INV_DIR}/oui-patch.xml && chown grid:oinstall ${INV_DIR}/oui-patch.xml"
done

2) 备份软件

-- 备份oracle_home和inventory目录文件
su - root
# . /home/grid/.bash_profile && export GI_HOME=${ORACLE_HOME}
# . /home/oracle/.bash_profile && export DB_HOME=${ORACLE_HOME}
# tar -zcpf ora_app_$(date +%Y%m%d).tgz ${GI_HOME} ${DB_HOME} &

cd /oracle
tar -czpf app.tgz app --exclude=*.trc --exclude=*.trm 

3) 增备或全备（未接入备份作业）

# 未接入备份
expdp \'/ as sysdba\' 

4）补丁更新应用（GI HOME+ DB HOME）（非DG等存在备库的情况）

所有节点依次顺序进行补丁应用

4.1）opatchauto方式

# 节点上的集群服务正常情况下
# 1. 配置环境变量
su - root
.  /home/grid/.bash_profile
export GI_HOME=${ORACLE_HOME}
export UNZIPPED_PATCH_LOCATION=/ups/soft
export PATH=${ORACLE_HOME}/OPatch:$PATH

# 2. 应用补丁（GI补丁已包含DB）
opatchauto apply ${UNZIPPED_PATCH_LOCATION}/28828733

4.2）手动应用补丁文件

# 1. 关闭DB服务(oracle)
# 1.1 语法：
$ /bin/srvctl stop home -o  -s  -n 
# 1.2 执行命令关闭节点上的数据库实例
su - oracle
$ORACLE_HOME/bin/srvctl stop home -o $ORACLE_HOME -s /tmp/orc1.stats -n orcl

# 2. 执行`rootcrs.pl -prepatch`命令关闭集群并解锁GI HOME
su - root
/ups/oracle/12.2/grid/perl/bin/perl /ups/oracle/12.2/grid/crs/install/rootcrs.pl -prepatch


# 3. 应用补丁包内所有子集(OCW, ACFS, DBWLM and DB, ...)到GI HOME，详见readme中Table2-1
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%OCW TRACKING BUG%
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%ACFS TRACKING BUG%
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%DB WLM TRACKING BUG%
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%DB RU TRACKING BUG%
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%TOMCAT RU TRACKING BUG%  

su - grid
# 3.1 应用db psu子包
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28822515 -silent

# 3.2 应用 OCW PSU 
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28870605 -silent

# 3.3 ACFS PSU
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28864846 -silent

# 3.4 DBWLM PSU
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/26839277 -silent

# 3.5 tomcat PSU
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28566910 -silent

# 4. 检查补丁更新详细日志
tail -100f $(ls -lrt /ups/oracle/12.2/grid/cfgtoollogs/opatch/*.log|tail -1|awk '{print $NF}')

# 5. 更新除 ACFS PSU到 DB HOME，详见readme Table2-2
$ /%BUGNO%/%OCW TRACKING BUG%/custom/scripts/prepatch.sh -dbhome 
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%OCW TRACKING BUG%
$ /OPatch/opatch apply -oh  -local /%BUGNO%/%DB RU TRACKING BUG%
$ /%BUGNO%/%OCW TRACKING BUG%/custom/scripts/postpatch.sh -dbhome   

# 5.1 执行prepatch.sh脚本保存db home的配置信息
su - oracle
/ups/soft/28828733/28870605/custom/scripts/prepatch.sh -dbhome /ups/oracle/database/product/12.2/db_1

# 5.2  Apply DB PSU sub-patches
/ups/oracle/database/product/12.2/db_1/OPatch/opatch apply -oh /ups/oracle/database/product/12.2/db_1 -local /ups/soft/28828733/28822515 -silent
# 5.3  Apply OCW PSU sub-patches
/ups/oracle/database/product/12.2/db_1/OPatch/opatch apply -oh /ups/oracle/database/product/12.2/db_1 -local /ups/soft/28828733/28870605 -silent

# 5.4 补丁更新详细日志
tail -100f $(ls -lrt /ups/oracle/database/product/12.2/db_1/cfgtoollogs/opatch/*.log|tail -1|awk '{print $NF}')

# 5.5 执行 postpatch.sh 脚本还原db home配置信息
/ups/soft/28828733/28870605/custom/scripts/postpatch.sh -dbhome /ups/oracle/database/product/12.2/db_1


# 6. root启动集群服务
su - root
/ups/oracle/12.2/grid/rdbms/install/rootadd_rdbms.sh
/ups/oracle/12.2/grid/perl/bin/perl /ups/oracle/12.2/grid/crs/install/rootcrs.pl -postpatch

# 6.1 查看详细日志
tail -100f /ups/oracle/grid/crsdata/$(hostname)/crsconfig/crspatch_*.log

4.3) non-rolling方式应用补丁

待验证

# 1. 关闭非本地外所有节点GI服务（即远端节点）
su - root
. /home/grid/.bash_profile
crsctl stop crs

# 2. （本地节点GI要启动）应用补丁GI HOME(依次顺序更新所有节点)
su - root
opatchauto apply /ups/soft/28828733 -oh /oracle/app/12.2/grid -nonrolling

5) 补丁应用（仅GI HOME）

5.1) opatchauto

opatchauto apply ${UNZIPPED_PATCH_LOCATION}/28828733 -oh 

5.2) 手动应用

# 1. 关闭集群服务
su - root
/ups/oracle/12.2/grid/perl/bin/perl /ups/oracle/12.2/grid/crs/install/rootcrs.pl -prepatch


# 2. 应用补丁包内所有子集(OCW, ACFS, DBWLM and DB, ...)到GI HOME，详见readme中Table2-1
su - grid
# 2.1 应用db psu子包
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28822515 -silent

# 2.2 应用 OCW PSU 
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28870605 -silent

# 2.3 ACFS PSU
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28864846 -silent

# 2.4 DBWLM PSU
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/26839277 -silent

# 2.5 tomcat PSU
/ups/oracle/12.2/grid/OPatch/opatch apply -oh /ups/oracle/12.2/grid -local /ups/soft/28828733/28566910 -silent

# 3. 启动集群root
su - root
/ups/oracle/12.2/grid/rdbms/install/rootadd_rdbms.sh
/ups/oracle/12.2/grid/perl/bin/perl /ups/oracle/12.2/grid/crs/install/rootcrs.pl -postpatch

6）补丁应用（仅DB HOME)

6.1) opatchauto

opatchauto apply ${UNZIPPED_PATCH_LOCATION}/28828733 -oh ,

6.2) 手动应用

# 1. 关闭db服务并保存状态
su - oracle
$ORACLE_HOME/bin/srvctl stop home -o $ORACLE_HOME -s /tmp/orc1.stats -n orc1

# 2. 更新除 ACFS PSU到 DB HOME，详见readme Table2-2
# 2.1 执行prepatch.sh脚本保存db home的配置信息
su - oracle
/ups/soft/28828733/28870605/custom/scripts/prepatch.sh -dbhome /ups/oracle/database/product/12.2/db_1

# 2.2  Apply DB PSU sub-patches
/ups/oracle/database/product/12.2/db_1/OPatch/opatch apply -oh /ups/oracle/database/product/12.2/db_1 -local /ups/soft/28828733/28822515 -silent
# 2.3  Apply OCW PSU sub-patches
/ups/oracle/database/product/12.2/db_1/OPatch/opatch apply -oh /ups/oracle/database/product/12.2/db_1 -local /ups/soft/28828733/28870605 -silent

# 2.4 补丁更新详细日志
tail -100f $(ls -lrt /ups/oracle/database/product/12.2/db_1/cfgtoollogs/opatch/*.log|tail -1|awk '{print $NF}')

# 3. 还原Db 配置
su - oracle
/ups/soft/28828733/28870605/custom/scripts/postpatch.sh -dbhome /ups/oracle/database/product/12.2/db_1

# 4. 启动DB 服务
su - oracle
/ups/oracle/database/product/12.2/db_1/bin/srvctl start home -o /ups/oracle/database/product/12.2/db_1 -n orc1 -s /tmp/orc1.stats

7) 更新数据字典

待所有节点都成功应用补丁软件后，选一DB实例执行数据字典更新操作。

# 启动orcl1服务
su - oracle
srvctl start instance -d orcl -i orcl1
# 连接并启动所有PDB
sqlplus /nolog
connect / as sysdba
alter pluggable database all open instances=all; 
quit;
# 更新字典
cd $ORACLE_HOME/OPatch
./datapatch -verbose

# 若存在失效对象，执行脚本修复
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
CONNECT / AS SYSDBA
@?/rdbms/admin/utlrp.sql

-- # PDB 中执行
alter session set container=PDB1;
@?/rdbms/admin/utlrp.sql

单独对指定的容器更新数据字典

cd ${ORACLE_HOME}/OPatch
sqlplus /  as sysdba
alter pluggable database all close instances = all;
alter pluggable database all open read write instances= all;

Execute datapatch individually in each of the containers, starting with CDB$ROOT :

1. Apply datapatch is CDB$ROOT :
  ===================================
  ./datapatch -verbose -pdbs CDB\$ROOT

2. Apply datapatch is PDB :
  ===================================
  $ ./datapatch -verbose -pdbs 

3. Apply datapatch is PDB$SEED :
  ==================================
  SQL> alter session set container=PDB$SEED;
  SQL> alter session set "_oracle_script"=TRUE;
  SQL> alter pluggable database pdb$seed close immediate instances=all;
  SQL> alter pluggable database pdb$seed OPEN READ WRITE;
  SQL> select open_mode from v$database;
  SQL> exit

  Execute: ./datapatch -verbose -pdbs PDB\$SEED

  SQL> alter session set "_oracle_script"=FALSE;
  
  SQL> alter pluggable database pdb$seed close immediate instances=all;
  SQL> alter pluggable database pdb$seed OPEN READ ONLY instances=all;
  
4. show pdbs -> If this shows the PDB$SEED in restricted mode, bounce the database once.

5. Check for the relevant patch existence in the registry$sqlpatch fopr all the containers (CDB$ROOT, PDB$SEED and RCPDB) :

  set linesize 150
  col logfile for a90
  select patch_id, action, logfile, status from registry$sqlpatch;

8）检查确认SQL

-- 检查版本
sqlplus "/ as sysdba"
set lines 168 pages 999
col DESCRIPTION for a40
col status for a10
col version for a10
select patch_id,PATCH_UID,STATUS,VERSION,to_char(ACTION_TIME,'yyyy-mm-dd hh24:mi:ss') ACTION_TIME,DESCRIPTION from dba_registry_sqlpatch;
  PATCH_ID  PATCH_UID STATUS     VERSION       BUNDLE_ID ACTION_TIME         DESCRIPTION
---------- ---------- ---------- ------------ ---------- ------------------- ----------------------------------------
  26609817   21483023 SUCCESS    12.2.0.1         170814 2017-10-11 09:37:18 DATABASE RELEASE UPDATE 12.2.0.1.170814
 
set lines 168 pages 999
col DESCRIPTION for a60
col status for a10
select INSTALL_ID,PATCH_ID,PATCH_UID,STATUS,TO_CHAR(ACTION_TIME,'YYYY-MM-DD HH24:MI:SS') ACTION_TIME,DESCRIPTION from dba_registry_sqlpatch;
INSTALL_ID   PATCH_ID  PATCH_UID STATUS     ACTION_TIME 	DESCRIPTION
---------- ---------- ---------- ---------- ------------------- ------------------------------------------------------------
	 1   29517242	22862832 SUCCESS    2020-01-07 09:25:57 Database Release Update : 19.3.0.0.190416 (29517242)
	 2   30125133	23151502 SUCCESS    2020-01-07 15:13:04 Database Release Update : 19.5.0.0.191015 (30125133)
	 3   30128191	23093535 SUCCESS    2020-01-07 15:44:41 OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)

SQL>

-- 检查失效对象情况
$ sqlplus "/ as sysdba"
set lines 168 pages 99
col owner for a18
col name for a18
SELECT o.owner, c.name, o.con_id, COUNT(*)
  FROM cdb_objects o, v$containers c
 WHERE o.con_id = c.con_id
   AND o.status = 'INVALID'
 GROUP BY o.owner, c.name, o.con_id
 ORDER BY 2, 1;
 
col COMP_NAME for a40
col schema for a18
select r.con_id,r.comp_name,r.version,r.status,c.name from cdb_registry r, v$containers c 
where r.con_id = c.con_id 
and r.status != 'VALID';

select r.con_id,r.comp_name,r.version,r.status,c.name from cdb_registry r, v$containers c 
where r.con_id = c.con_id;

-- 检查pdb是否存在异常错误信息
set lines 200 pages 99
col name for a12
col time for a22
col cause for a10
col type for a8
col message for a48
col status for a8
col action for a32
col con_id for 9999
col ERROR_NUMBER for 9999
SELECT to_char(v.time,'yyyy-mm-dd hh24:mi:ss') time,
       v.name,
       v.cause,
       v.type,
       v.error_number errcode,
       v.message,
       v.status,
       v.action,
       v.con_id
  FROM pdb_plug_in_violations v
  where v.type='ERROR';

2.4.3 DB PSU补丁

1）备份软件

su - oracle
cd /oracle
tar -xf app.tgz app

2) 更新软件

# 1. 冲突检查
cd 
opatch prereq CheckConflictAgainstOHWithDetail -ph ./

# 2. 关闭DB服务
sqlplus / as sysdba
shutdown immediate;

# 3. 应用补丁
cd 
opatc apply

# 4. 更新数据字典
cd $ORACLE_HOME/OPatch
./datapatch -verbose


# 5. 检查确认
opatch lsinv

2.4.4 OJVM PSU补丁

注意：需要全部停止数据库服务

1) 解压补丁文件

su - oracle
unzip -qo p25811364_122010_Linux-x86-64.zip -d /oracle/soft

2) 冲突检测

su - oracle
export PATH=${ORACLE_HOME}/OPatch:${PATH}
cd 25811364
opatch prereq CheckConflictAgainstOHWithDetail -ph ./

3) 停止服务(所有节点)

su - oracle
srvctl stop database -d orcl 
su - root
. /home/grid/.bash_profile
crsctl stop crs

4）应用补丁

su - oracle
cd 25811364
opatch apply

5) 检查确认

opatch lsinventory

6）更新数据字典

6.1) RAC

待所有节点软件更完成后，选一DB实例完成数据字典更新

-- Single/Multitenant (CDB/PDB) DB
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> alter system set cluster_database=false scope=spfile;
 
$ORACLE_HOME/bin/srvctl stop database -d orcl
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP UPGRADE
SQL> alter pluggable database all open upgrade;
SQL> quit

cd $ORACLE_HOME/OPatch
./datapatch -verbose

sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> alter system set cluster_database=true scope=spfile;
SQL> SHUTDOWN
SQL> quit
$ORACLE_HOME/bin/srvctl start database -d orcl

6.2) SI单实例

sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP UPGRADE
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> shutdown
SQL> startup

7) 检查失效对象

# 若存在失效对象，重新编译
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql

8) 检查视图确认

sqlplus "/ as sysdba"
set lines 168 pages 999
REM 'invalid pdb'
col name for a12
col time for a22
col cause for a10
col type for a8
col message for a48
col status for a8
col action for a32
col con_id for 9999
col ERROR_NUMBER for 9999
SELECT to_char(v.time,'yyyy-mm-dd hh24:mi:ss') time,
       v.name,
       v.cause,
       v.type,
       v.error_number errcode,
       v.message,
       v.status,
       v.action,
       v.con_id
  FROM pdb_plug_in_violations v
  where v.type='ERROR';

REM 'invalid objs'
col owner for a18
select owner,con_id,count(*) cnt from cdb_objects where status = 'INVALID' group by owner,con_id, status order by 2,1;

col COMP_NAME for a40
col schema for a18
REM 'invalid comps'
select con_id,comp_name,version,status,schema from cdb_registry where status = 'INVALID';

-- 12.2
col DESCRIPTION for a40
col status for a10
col version for a10
REM 'patch version'
SELECT patch_id
      ,patch_uid
      ,status
      ,version
      ,bundle_id
      ,to_char(action_time, 'yyyy-mm-dd hh24:mi:ss') action_time
      ,description
  FROM dba_registry_sqlpatch;

-- 19
col ACTION for a8
col DESCRIPTION for a32
col SOURCE_BUILD_DESCRIPTION for a32
col SOURCE_BUILD_TIMESTAMP for a24
col TARGET_BUILD_DESCRIPTION for a24
col TARGET_BUILD_TIMESTAMP for a24
SELECT rsp.install_id
      ,rsp.patch_id
      ,rsp.patch_uid
      ,rsp.patch_type
      ,rsp.action
      ,rsp.status
      ,to_char(rsp.action_time, 'yyyy-mm-dd hh24:mi:ss') action_time
      ,rsp.description
      ,rsp.source_version
      ,rsp.source_build_description
     -- ,rsp.source_build_timestamp
      ,rsp.target_version
      ,rsp.target_build_description
     -- ,rsp.target_build_timestamp
  FROM dba_registry_sqlpatch rsp;

2.5 含有Standby环境升级

Oracle Patch Assurance - Data Guard Standby-First Patch Apply (Doc ID 1265700.1)

3. 回退步骤

3.1 GI 补丁回滚

3.1.1 opatchauto方式

# 1. 回滚GI 补丁
# GI HOME + DB HOME
su - root
/OPatch/opatchauto rollback /26610291

#  only GI HOME
opatchauto rollback ${UNZIPPED_PATCH_LOCATION}/28828733 -oh 

# only DB HOME
opatchauto rollback /26610291 -oh ,

# 回滚数据字典
su - oracle
sqlplus /nolog
connect / as sysdba
startup
alter pluggable database all open; 
quit;

cd $(orabasehome)/OPatch
./datapatch -verbose

# 若存在失效对象，执行脚本修复

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
CONNECT / AS SYSDBA
@?/rdbms/admin/utlrp.sql

3.1.2 手工回滚

GI HOME+DB HOME

# 1. 停DB服务
su - oracle
/bin/srvctl stop home -o  -s  -n 

# 2. 关闭集群
su - root
/crs/install/rootcrs.sh -prepatch -rollback


# 3. 回滚GI HOME 补丁
su - grid
$ /OPatch/opatch nrollback -local -id %OCW TRACKING BUG%,%ACFS TRACKING BUG%,%DB RU TRACKING BUG%, %DB WLM TRACKING BUG%, %TOMCAT TRACKING BUG -oh 

# 4. 回滚DB HOME 补丁
su - oracle
$ /%BUGNO%/%OCW TRACKING BUG%/custom/scripts/prepatch.sh -dbhome 

$ /OPatch/opatch nrollback -local -id %OCW TRACKING BUG%,%DBRU TRACKING BUG% -oh  

$ /%BUGNO%/%OCW TRACKING BUG%/custom/scripts/postpatch.sh -dbhome  

# 5. 启动集群服务
su - root
/rdbms/install/rootadd_rdbms.sh
/crs/install/rootcrs.sh -postpatch -rollback

# 6. 启动DB服务
su - oracle
/bin/srvctl start home -o  -s  -n 

# 7. 回滚数据字典

3.2 DB 补丁回滚

# oracle用户执行
$ /OPatch/opatch nrollback -local -id %OCW TRACKING BUG%,%DB RU TRACKING BUG% -oh  

3.3 OJVM补丁回滚

# 停止所有服务，同2.4.4.4节内容
# 软件回退
su - oracle
opatch rollback -id 25811364

# 2. 回退数据字典，同2.4.4.6节
#

# 3. 检查日志
cd $ORACLE_HOME/sqlpatch/25811364/
tail -100f _rollback___.log

# 4. 启动服务

4. 常见问题与AFQ

4.1 软件版本不一致问题

1) 问题信息

[grid@o19c1 ~]$ asmca -silent -addDisk -diskGroupName DATA -disk '/dev/asm-data' 

[FATAL] [DBT-30007] Addition of disks to disk group DATA failed.
ORA-15032: not all alterations performed
ORA-15137: The ASM cluster is in rolling patch state.

/*

*/

2) 处理方案

可能原因是当前实际的 active version 不是一致的。
检查所有节点：

1. /bin/kfod op=PATCHES
2. /bin/kfod op=patchlvl
3. crsctl query crs activeversion -f
4. cd /OPatch
   ./opatch lspatches

# 确认所有节点补丁相同 crsctl query crs softwarepatch
[grid@o19c1 ~]$ crsctl query crs softwarepatch o19c1
Oracle Clusterware patch level on node o19c1 is [724960844].
[grid@o19c1 ~]$ crsctl query crs softwarepatch o19c2
Oracle Clusterware patch level on node o19c2 is [724960844].
[grid@o19c1 ~]$ 

# 以root在任一节点执行
su - root
. /home/grid/.bash_profile
crsctl stop rollingpatch
/crs/install/rootcrs.sh -unlock
/crs/install/rootcrs.sh -patch

4.2 public权限问题引起组件失效

-- 检查public默认权限
set lines 168 pages 99
select table_name 
from dba_tab_privs 
where grantee='PUBLIC' 
and privilege='EXECUTE' 
and table_name in ('UTL_FILE', 'UTL_TCP', 'UTL_HTTP', 'UTL_SMTP', 'DBMS_RANDOM','DBMS_SQL','DBMS_JOB','DBMS_LOB');

-- 重新授权
grant execute on dbms_sql to PUBLIC;
grant execute on dbms_lob to PUBLIC;
grant execute on utl_file to PUBLIC;
grant execute on dbms_job to PUBLIC;
grant execute on dbms_scheduler to PUBLIC;
grant execute on dbms_random to public;
grant execute on utl_http to public;
grant execute on utl_smtp to public;
grant execute on UTL_FILE to MDSYS;

4.3 软件版本不一致引起CRS不能启动问题

1) 问题信息

[root@o19c1 ~]# crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('74762968') does not match Software patch level ('724960844'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
[root@o19c1 ~]# crsctl query crs releasepatch

2) 处理方案

Patching 12.2.0.1 Grid Infrastructure gives error CRS-6706: Oracle Clusterware Release Patch Level ('748994161') Does Not Match Software Patch Level (文档 ID 2348013.1)

su - root
# 1. Run the following command as the root user to complete the patching set up behind the scenes:
cd /ups/oracle/19.2/grid/bin
./clscfg -localpatch

# 2. Run the following command as the root user to lock the GI home
cd /ups/oracle/19.2/grid/crs/install/
./rootcrs.sh -lock

# 3. Run the following command as the root user to start the GI
crsctl start crs