Oracle 12C补丁升级步骤及异常原因记录（CRS-6760）
最近需要配合某厂商开展12C 的补丁升级工作，升级过程中由于操作人员的误操作导致了节点宕机。特在此进行记录。
主机名 |
IP地址 |
DB&GI版本 |
OS版本 |
补丁集 |
Oracle |
192.168.56.101 |
12.2.0.1.0 |
Oracle Linux 7.3 |
27674384 |
Hadoop1 |
192.168.56.102 |
12.2.0.1.0 |
Oracle Linux 7.3 |
27674384 |
Hadoop2 |
192.168.56.103 |
12.2.0.1.0 |
Oracle Linux 7.3 |
27674384 |
首先我们看下我们的节点按照官方的补丁安装文档，按照后的情况：
12C的补丁升级很简单，主要步骤如下：
1、下载补丁，需要metalink账号
https://updates.oracle.com/Orion/Services/download/p27468969_122010_Linux-x86-64.zip?aru=22096835&patch_file=p27468969_122010_Linux-x86-64.zip
2、使用grid用户解压缩补丁，解压完以后权限如下，这里千万注意解压后文件夹的权限，要注意解压的文件夹属主必须是grid。
[grid@hadoop1 patch]$ ls -l
total 1288616
drwxr-x---. 8 grid oinstall 4096 Apr 9 03:25 27468969
drwxrwxrwx. 8 grid oinstall 4096 Apr 9 03:25 27468969_bak
-rw-r--r--. 1 grid oinstall 1222109132 Jun 5 01:51 p27468969_122010_Linux-x86-64.zip
-rw-r--r--. 1 grid oinstall 97422646 Jun 5 02:59 p6880880_122010_Linux-x86-64.zip3、检查是否存在冲突
---grid用户下
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/27674384
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/27464465
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/27458609
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/26839277
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/27144050
---oracle 用户下
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/27674384
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/patch/27468969/27464465以上全部PASS后可以进行补丁更新
4、使用root在线更新补丁
[root@hadoop1 patch]# echo $ORACLE_HOME
/oracle/app/grid/grid_home
[root@hadoop1 patch]# opatchauto apply /u01/patch/27468969
bash: opatchauto: command not found...
[root@hadoop1 patch]# $ORACLE_HOME/OPatch/opatchauto apply /u01/patch/27468969
OPatchauto session is initiated at Wed Jun 13 00:25:51 2018
System initialization log file is /oracle/app/grid/grid_home/cfgtoollogs/opatchautodb/systemconfig2018-06-13_12-25-55AM.log.
Session log file is /oracle/app/grid/grid_home/cfgtoollogs/opatchauto/opatchauto2018-06-13_12-26-34AM.log
The id for this session is GLQ4
Executing OPatch prereq operations to verify patch applicability on home /oracle/app/grid/grid_home
Executing OPatch prereq operations to verify patch applicability on home /oracle/app/oracle/oracle_home
Patch applicability verified successfully on home /oracle/app/grid/grid_home
Patch applicability verified successfully on home /oracle/app/oracle/oracle_home
Verifying SQL patch applicability on home /oracle/app/oracle/oracle_home
"/bin/sh -c 'cd /oracle/app/oracle/oracle_home; ORACLE_HOME=/oracle/app/oracle/oracle_home ORACLE_SID=greatdata1 /oracle/app/oracle/oracle_home/OPatch/datapatch -prereq -verbose'" command failed with errors. Please refer to logs for more details. SQL changes, if any, can be analyzed by manually retrying the same command.
SQL patch applicability verified successfully on home /oracle/app/oracle/oracle_home
Preparing to bring down database service on home /oracle/app/oracle/oracle_home
Successfully prepared home /oracle/app/oracle/oracle_home to bring down database service
Bringing down CRS service on home /oracle/app/grid/grid_home
Prepatch operation log file location: /oracle/app/grid/grid_base/crsdata/hadoop1/crsconfig/crspatch_hadoop1_2018-06-13_00-28-17AM.log
CRS service brought down successfully on home /oracle/app/grid/grid_home
Performing prepatch operation on home /oracle/app/oracle/oracle_home
Perpatch operation completed successfully on home /oracle/app/oracle/oracle_home
Start applying binary patch on home /oracle/app/oracle/oracle_home
Binary patch applied successfully on home /oracle/app/oracle/oracle_home
Performing postpatch operation on home /oracle/app/oracle/oracle_home
Postpatch operation completed successfully on home /oracle/app/oracle/oracle_home
Start applying binary patch on home /oracle/app/grid/grid_home
Binary patch applied successfully on home /oracle/app/grid/grid_home
Starting CRS service on home /oracle/app/grid/grid_home
Postpatch operation log file location: /oracle/app/grid/grid_base/crsdata/hadoop1/crsconfig/crspatch_hadoop1_2018-06-13_00-37-55AM.log
CRS service started successfully on home /oracle/app/grid/grid_home
Preparing home /oracle/app/oracle/oracle_home after database service restarted
No step execution required.........
Trying to apply SQL patch on home /oracle/app/oracle/oracle_home
SQL patch applied successfully on home /oracle/app/oracle/oracle_home
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:hadoop1
RAC Home:/oracle/app/oracle/oracle_home
Version:12.2.0.1.0
Summary:
==Following patches were SKIPPED:
Patch: /u01/patch/27468969/27458609
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /u01/patch/27468969/26839277
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /u01/patch/27468969/27144050
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /u01/patch/27468969/27464465
Log: /oracle/app/oracle/oracle_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-31-16AM_1.log
Patch: /u01/patch/27468969/27674384
Log: /oracle/app/oracle/oracle_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-31-16AM_1.log
Host:hadoop1
CRS Home:/oracle/app/grid/grid_home
Version:12.2.0.1.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /u01/patch/27468969/26839277
Log: /oracle/app/grid/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-33-33AM_1.log
Patch: /u01/patch/27468969/27144050
Log: /oracle/app/grid/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-33-33AM_1.log
Patch: /u01/patch/27468969/27458609
Log: /oracle/app/grid/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-33-33AM_1.log
Patch: /u01/patch/27468969/27464465
Log: /oracle/app/grid/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-33-33AM_1.log
Patch: /u01/patch/27468969/27674384
Log: /oracle/app/grid/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-13_00-33-33AM_1.log
OPatchauto session completed at Wed Jun 13 00:42:53 2018
Time taken to complete the session 17 minutes, 2 seconds查看查看补丁更新情况
[grid@hadoop1 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.13
Copyright (c) 2018, Oracle Corporation. All rights reserved.
Oracle Home : /oracle/app/grid/grid_home
Central Inventory : /oracle/app/grid/oraInventory
from : /oracle/app/grid/grid_home/oraInst.loc
OPatch version : 12.2.0.1.13
OUI version : 12.2.0.1.4
Log file location : /oracle/app/grid/grid_home/cfgtoollogs/opatch/opatch2018-06-13_01-08-18AM_1.log
Lsinventory Output file location : /oracle/app/grid/grid_home/cfgtoollogs/opatch/lsinv/lsinventory2018-06-13_01-08-18AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: hadoop1
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.2.0.1.0
There are 1 products installed in this Oracle Home.
Interim patches (5) :
Patch 27674384 : applied on Wed Jun 13 00:37:28 CST 2018
Unique Patch ID: 22098633
Patch description: "Database Apr 2018 Release Update : 12.2.0.1.180417 (27674384)"
Created on 9 Apr 2018, 00:43:55 hrs PST8PDT
Bugs fixed:
23026585, 24336249, 24929210, 24942749, 25036474, 25110233, 25410877
25417050, 25427662, 25459958, 25547901, 25569149, 25600342, 25600421
25606091, 25655390, 25662088, 24385983, 24923215, 25099758, 25429959
25662101, 25728085, 25823754, 22594071, 23665623, 23749454, 24326846
24334708, 24560906, 24573817, 24578797, 24609996, 24624166, 24668398
24674955, 24744686, 24811725, 24827228, 24831514, 24908321, 24976007
25184555, 25210499, 25211628, 25223839, 25262869, 25316758, 25337332
25455795, 25457409, 25539063, 25546608, 25612095, 25643931, 25410017
22729345, 24485174, 24509056, 24714096, 25329664, 25410180, 25607726
25957038, 25973152, 26024732, 24376878, 24589590, 24676172, 23548817
24796092, 24907917, 25044977, 25736747, 25766822, 25856821, 25051628
24534401, 24835919, 25050160, 25395696, 25430120, 25616359, 25715167
25967985, 26546754, 26546664, 26544823, 26575788, 26271001, 26198926
26025681, 26078437, 25789579, 25034396, 24655717, 25124363, 23527363
25530814, 21981529, 25062592, 25057811, 23179662, 25530080, 25528838
25417056, 25100579, 25739065, 25700654, 25655966, 25631933, 25616645
25616417, 25594901, 25825910, 25789041, 22568728, 21159907, 20736227
19327292, 14690846, 25099497, 24938784, 25257085, 25237577, 25230945
25425760, 25337640, 25263960, 22654475, 21186167, 23491861, 23300142
23234232, 23151677, 22981722, 23733981, 24556967, 24485161, 24457597
24737064, 24578718, 24792678, 20324049, 17533661, 17027695, 22645009
22628825, 23521523, 23499160, 24289874, 24421668, 24332831, 24425998
24555417, 24737403, 25121089, 25192729, 25164293, 25150925, 25579761
25570929, 25980605, 25954022, 25797124, 26308650, 26261327, 26088836
25874050, 25178179, 25176408, 22072543, 23735292, 21985256, 22950945
23746128, 25175723, 25451531, 25575628, 25838361, 25932524, 26479173
26526726, 26609942, 24717183, 26362821, 25397936, 25554787, 26523432
26599395, 26331743, 26476244, 26388538, 25498930, 26421667, 26446098
26108080, 25648731, 25087436, 24812047, 25382812, 25695903, 25797092
22179537, 26111842, 19285025, 25558986, 25551676, 24642495, 25633101
25852885, 25897615, 26486365, 26333141, 26537307, 26444887, 26396790
26358670, 26244115, 25489607, 25947799, 26095405, 26492866, 25248384
26430737, 26187943, 26426526, 22898198, 23581777, 25694206, 26088426
26045732, 24693290, 26153977, 25654936, 25870579, 20003668, 24912588
25638456, 25320555, 25240590, 23599216, 25182817, 24735430, 25872127
25192044, 20620169, 26324769, 24960044, 24677696, 25995938, 23125560
23061453, 25166187, 25722608, 25662524, 25598473, 25774077, 19614243
24664211, 25145215, 25123585, 24968162, 25241625, 25472112, 25463844
25410802, 25383204, 22446455, 23481673, 23055900, 26249718, 25967544
22087683, 23730961, 24376875, 24368004, 25162645, 25029022, 25099339
25093872, 25433696, 25392535, 25357142, 25328518, 25553616, 25970731
25953857, 26169345, 26169341, 26091640, 26032573, 22503297, 24341675
24850622, 25186079, 25602488, 25861398, 25980770, 26096382, 26765212
23035249, 24797119, 25405813, 24443539, 25098160, 25384462, 24593740
25241448, 26751171, 26751106, 26545688, 26367012, 25871639, 25597525
25546580, 25063971, 24654629, 23105538, 24401351, 25764020, 25259611
25244807, 25265499, 26650226, 26330994, 26658759, 26351334, 22347493
25140197, 26659182, 25990907, 25660847, 25411036, 24845157, 25888073
20591151, 25954054, 25822410, 25788879, 25335790, 25199585, 24717859
22564336, 24461826, 24337882, 24010030, 22826067, 26680105, 26626879
26641610, 26189861, 25826740, 25335249, 25335360, 25100063, 21216226
26327624, 25977302, 26597140, 26493289, 26435073, 25895224, 25634317
24617969, 19211433, 25911724, 22599050, 24843188, 24486059, 24922704
18594510, 19649997, 24680959, 25107662, 25964954, 23645516, 26237431
25754606, 23572982, 25045228, 25462714, 25722720, 24848746, 24760407
25203656, 25444961, 21520266, 23600861, 24600330, 24415926, 24423416
17947871, 23066146, 24326444, 24346821, 24623975, 24699619, 24801152
24786669, 24710696, 25065563, 25179774, 25129925, 25417958, 26237773
27087426, 27009164, 26969321, 26366517, 26898563, 26784509, 26522439
23271203, 22364044, 25307368, 26758193, 26542135, 15931756, 26548363
25710420, 26434999, 25207410, 26223039, 25524955, 25579458, 25489367
25313411, 24554533, 20756305, 25478885, 25858672, 26036748, 24966788
24766309, 21178363, 23715460, 23197730, 23602213, 25072986, 25729507
25890056, 25807997, 25206864, 23065002, 24345420, 24303148, 22864303
24435982, 24848923, 24978100, 24953434, 25210268, 25209912, 25981498
25919622, 25871177, 26569225, 26744595, 26908788, 25313154, 27135993
26635897, 26939314, 26983259, 26820076, 25785441, 27142373, 26729494
20532077, 23249829, 27073314, 26629381, 27079140, 26911000, 26944190
25305405, 23741944, 25348956, 25768681, 26794786, 25646373, 24570598
26399626, 25674386, 25225795, 25224242, 25536819, 25296876, 26375250
25963024, 26615690, 26442308, 25476149, 26170715, 25685152, 24307571
19526548, 24468470, 24744383, 25171084, 24713381, 24827654, 24343905
21433452, 24373756, 24923790, 24911709, 26019148, 24940060, 24595699
25691904, 25687460, 25672640, 25874678, 25785331, 22379010, 18521691
8480838, 25159176, 25227381, 25561296, 25440818, 23588722, 24470606
24692973, 24674197, 20118035, 24563422, 24759556, 25201454, 27165231
27052607, 22007324, 26037215, 25830492, 25086233, 26584641, 21147908
26110632, 24403922, 25386748, 24350620, 27305039, 22503283, 22581771
25945130, 26967713, 22367053, 26478970, 19647894, 26149904, 25437699
18774543, 26849779, 26121990, 26374791, 24374976, 25202355, 26608137
27034890, 22970869, 25489342, 25717371, 16727454, 27117822, 25557886
20087519, 25904273, 26637273, 25078611, 26006257, 25803545, 26038086
24689376, 25941836, 23709062, 19072655, 25634348, 25661819, 24486237
25388896, 22104866, 22070853, 25647325, 22504793, 26418088, 25287072
26633355, 26909100, 23080557, 25120284, 26245237, 23126545, 24616637
26256131, 23019710, 24308349, 22700845, 27169796, 25975723, 26797591
26080410, 24923338, 27213224, 27044297, 25813931, 22495673, 24802934
26263328, 25653109, 26840654, 25914276, 25492379, 26822620, 26556014
26198757, 25722055, 25516250, 21479706, 27124624, 26029780, 25760195
25790353, 22843979, 25616268, 26095327, 26775602, 20866970, 25353983
26981902, 25103996, 26740700, 26243698, 26259265, 25555252, 26875822
26452606, 26263721, 25639019, 25635149, 27060167, 24437162, 24589081
26546070, 25775213, 25865785, 24609592, 25943271, 26513709, 27274536
27508985, 27548131, 27611612, 26712331, 27338912, 26090767, 24473736
26444601, 27501373, 16942578, 21320338, 26907327, 26353617, 27329612
27394703, 25832935, 24737581, 25871753, 26091786, 24923080, 25680221
27207110, 24844549, 25784002, 25982666, 26412540, 26201113, 26729611
25226665, 26475419, 24757934, 25654459, 25614866, 27333106, 26637824
26498354, 20976443, 26138085, 27229389, 24793511, 26714910, 25058080
24719799, 23184263, 26802503, 24800423, 25540738, 27367194, 27292213
27138325, 27441326, 27502420
Patch 27464465 : applied on Wed Jun 13 00:36:35 CST 2018
Unique Patch ID: 22045693
Patch description: "OCW APR 2018 RELEASE UPDATE 12.2.0.1.0(180129) (27464465)"
Created on 31 Mar 2018, 05:33:42 hrs PST8PDT
Bugs fixed:
13250991, 20559126, 22322021, 22986384, 22999793, 23170674, 23297046
23301767, 23340259, 23346234, 23722215, 23760401, 23762756, 24330981
24372941, 24396050, 24403376, 24411578, 24422155, 24423011, 24445255
24451580, 24462646, 24464953, 24509984, 24546078, 24578464, 24583356
24584419, 24600431, 24657753, 24663993, 24664849, 24665035, 24670214
24673586, 24696235, 24732650, 24736511, 24792446, 24796183, 24797972
24801915, 24808260, 24820676, 24824518, 24831158, 24831217, 24902553
24908629, 24932026, 24953946, 25025157, 25026470, 25037011, 25037836
25049033, 25066549, 25089164, 25100966, 25131365, 25137752, 25164540
25197395, 25197670, 25216003, 25233797, 25245759, 25251382, 25291358
25301463, 25307145, 25325522, 25327670, 25354530, 25360754, 25369170
25371632, 25397620, 25408895, 25409838, 25448462, 25448476, 25485737
25493588, 25496699, 25499276, 25505841, 25518447, 25537905, 25540583
25541343, 25556203, 25564644, 25569634, 25576767, 25591658, 25644425
25646592, 25655495, 25662028, 25667162, 25685286, 25697364, 25698636
25715179, 25716769, 25717212, 25728787, 25742471, 25774482, 25784424
25785073, 25789051, 25790699, 25795447, 25801922, 25803563, 25825732
25825755, 25832312, 25832375, 25851874, 25856024, 25865563, 25890662
25893570, 25915863, 25927065, 25943805, 25947195, 25956724, 25987595
25996923, 25998093, 26002739, 26052877, 26088770, 26111524, 26115835
26121788, 26144044, 26198627, 26238801, 26248428, 26257547, 26264417
26282865, 26301456, 26316238, 26326556, 26357098, 26379711, 26387283
26391418, 26418606, 26419112, 26431982, 26433972, 26433979, 26439462
26439893, 26473232, 26477705, 26493466, 26494006, 26529142, 26546632
26547521, 26552807, 26567325, 26581118, 26586174, 26593581, 26624184
26633228, 26664476, 26710631, 26755613, 26791882, 26817081, 26817275
26817995, 26818064, 26818368, 26822082, 26860285, 26950548, 26979409
27006180, 27012234, 27047274, 27068526, 27070729, 27080761, 27114112
27119820, 27130514, 27133637, 27154002, 27174938, 27182006, 27190262
27213140, 27222423, 27222938, 27252023, 27309182, 27399762, 27554103
27581484, 27628221, 27753602, 27757979
Patch 27458609 : applied on Wed Jun 13 00:35:32 CST 2018
Unique Patch ID: 21914305
Patch description: "ACFS APR 2018 RELEASE UPDATE 12.2.0.1.0(180129) (27458609)"
Created on 23 Mar 2018, 00:25:23 hrs PST8PDT
Bugs fixed:
21129279, 22591010, 23152694, 23181299, 23625427, 24285969, 24346777
24652931, 24661214, 24674652, 24679041, 24690973, 24964969, 25078431
25098392, 25375360, 25381434, 25480028, 25491831, 25526314, 25549648
25560948, 25726952, 25764672, 25826440, 25966987, 26051087, 26085458
26247490, 26275740, 26396215, 26625494, 26667459, 26730740, 26740931
26759355, 26844019, 26871374, 26912733, 26987877, 26990202, 27012440
27027294, 27065091, 27081038, 27163313, 27223171, 27230645, 27333978
27339654, 27573409
Patch 27144050 : applied on Wed Jun 13 00:33:50 CST 2018
Unique Patch ID: 21771994
Patch description: "Tomcat Release Update 12.2.0.1.0(ID:171023.0830) (27144050)"
Created on 26 Dec 2017, 23:52:30 hrs PST8PDT
Bugs fixed:
25728967, 26934551
Patch 26839277 : applied on Wed Jun 13 00:33:43 CST 2018
Unique Patch ID: 21578760
Patch description: "DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)"
Created on 21 Sep 2017, 03:13:10 hrs PST8PDT
Bugs fixed:
26584906
--------------------------------------------------------------------------------
OPatch succeeded.就是这么简单，但是今天的问题是，如果在解压补丁的时候给了错误的文件夹权限，将会导致补丁升级失败节点宕机。
现场升级失败的原因总结如下：
1、补丁的目录放到了根目录下/opatch
2、解压时目录的权限为600，导致补丁部分文件夹的属主数组全部改变
3、进行补丁哦patchauto后，节点宕机，手工启动集群CRS-6706错误
CRS-6706: Oracle Clusterware Release patch level ('0') does not match Software patch level ('85922511'). Oracle Clusterware cannot be started4、处理方法如下在grid和oracle用户下分别执行执行
$ORACLE_HOME/OPatch/opatch rollback -id 276743845、解锁相关文件，使用root执行
$GI_HOME/bin/clscfg -localpatch
$GI_HOME/crs/install/rootcrs.sh -lock
$GI_HOME/bin/crsctl start crs6、集群恢复正常，节点恢复正常。
注：如果在补丁升级过程中，认为是HAS故障，重新初始化了该节点的HAS，那么解决方法需要增加一步。
在上面的第4步以后，使用root重新在宕机节点执行
$GI_HOME/root.sh