Oracle 11gR2 RAC搭建（虚拟机搭建）

安装环境说明与虚拟机规化：

安装环境
主机操作系统：windows 10
虚拟机VMware15：两台CentOS-7.6
Oracle Database software: Oracle11gR2 （11.2.0.1.0）
Cluster software: Oracle grid infrastructure 11gR2（11.2.0.1.0）
共享存储：ASM
oracle传送门 https://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html

虚拟机规划

1. 安装虚拟机时注意分配两个网卡，一个网卡为Host Only方式，用于两台虚拟机节点的通讯，另一个网卡为NAT方式，用于连接外网，这里由虚拟机自动分配IP（为了以后省事还是配静态好）。每台主机的内存和swap规划为至少2.5G。硬盘规划为：boot 500M，其他空间分配为LVM方式管理，LVM划分2.5G为swap，其他为/。 主机名设为rac1、rac2 （注意这里两个虚拟机最好在不同的硬盘中，否则I/O会很吃力，否则万分痛苦！）
2. 由于采用的是共享存储ASM，而且搭建集群需要共享空间作注册盘（OCR）和投票盘（votingdisk）。VMware创建共享存储方式：
   进入VMware安装目录，cmd命令下：
   E:\Program Files (x86)\VMware\VMware Workstation

   vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 F:\asm\ocr.vmdk
   vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 F:\asm\ocr2.vmdk
   vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 F:\asm\votingdisk.vmdk
   vmware-vdiskmanager.exe -c -s 20000Mb -a lsilogic -t 2 F:\asm\data.vmdk
   vmware-vdiskmanager.exe -c -s 10000Mb -a lsilogic -t 2 F:\asm\backup.vmdk

   创建了两个1G的ocr盘，一个1G的投票盘，一个20G的数据盘，一个10G的备份盘。
   3、虚拟机添加磁盘，选定虚拟机然后编辑配置，添加磁盘选择刚刚添加的虚拟磁盘，大概如下（2台都执行）
   

---

虚拟机操作系统配置（没有特别注明都是2台均执行）

1. 关闭防火墙
   查看防火墙状态
   #firewall-cmd --state
   停止firewall
   #systemctl stop firewalld.service
   禁止firewall开机启动
   #systemctl disable firewalld.service
2. 创建必要的用户、组和目录，并授权

   #groupadd -g 1000 oinstall
   #groupadd -g 1020 asmadmin
   #groupadd -g 1021 asmdba
   #groupadd -g 1022 asmoper
   #groupadd -g 1031 dba
   #groupadd -g 1032 oper
   #useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
   #useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle
   #mkdir -p /u01/app/11.2.0/grid
   #mkdir -p /u01/app/grid
   #mkdir /u01/app/oracle
   #chown -R grid:oinstall /u01
   #chown oracle:oinstall /u01/app/oracle
   #chmod -R 775 /u01/

3. 系统文件设置
   a.修改内核配置，在文件/etc/sysctl.conf 的末尾处添加如下信息
   #vi /etc/sysctl.conf
   kernel.msgmnb = 65536
   kernel.msgmax = 65536
   kernel.shmmax = 68719476736
   kernel.shmall = 4294967296
   fs.aio-max-nr = 1048576
   fs.file-max = 6815744
   kernel.shmall = 2097152
   kernel.shmmax = 1306910720
   kernel.shmmni = 4096
   kernel.sem = 250 32000 100 128
   net.ipv4.ip_local_port_range = 9000 65500
   net.core.rmem_default = 262144
   net.core.rmem_max = 4194304
   net.core.wmem_default = 262144
   net.core.wmem_max = 1048586
   net.ipv4.tcp_wmem = 262144 262144 262144
   net.ipv4.tcp_rmem = 4194304 4194304 4194304

使配置生效
#sysctl -p
b.配置oracle、grid用户的shell限制,在文件/etc/security/limits.conf 的末尾加入如下内容
#vi /etc/security/limits.conf
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
c.配置login,在文件/etc/pam.d/login的末尾加入如下内容
#vi /etc/pam.d/login
session required pam_limits.so
d.修改操作系统发行版本名，将文件/etc/centos-release的原有内容注释，添加redheat-7（因为oracle不支持centos7安装）

e.使用安装介质作为yum源安装依赖的包
配置本地源的方式，自己先进行配置:
#mount /dev/sr0 /mnt/cdrom/
#vi /etc/yum.repos.d/dvd.repo
[dvd]
name=dvd
baseurl=file:///mnt/cdrom
gpgcheck=0
enabled=1
#yum clean all
#yum makecache
#yum install gcc gcc-c++ glibc glibc-devel ksh libgcc libstdc++ libstdc++-devel* make sysstat
4.配置IP和hosts、hostname（这里的hostname在系统安装时已经设定为rac1和rac2，ip由虚拟机自动分配，因此只配hosts），在文件/etc/hosts 的末尾添加如下内容
#vi /etc/hosts
192.168.149.129 rac1
192.168.149.201 rac1-vip
192.168.96.128 rac1-priv

192.168.149.130 rac2
192.168.149.202 rac2-vip
192.168.96.129 rac2-priv

192.168.149.100 scan-ip

5.配置grid和oracle用户环境变量
Oracle_sid需要根据节点不同进行修改
#su - grid
$ vi .bash_profile
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=+ASM1 # RAC1使用这个配置
export ORACLE_SID=+ASM2 # RAC2使用这个配置
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
umask 022

需要注意的是ORACLE_UNQNAME是数据库名（这里用orcl），创建数据库时指定多个节点是会创建多个实例，ORACLE_SID指的是数据库实例名
#su - oracle
$vi .bash_profile
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=orcl1 # RAC1使用这个配置
export ORACLE_SID=orcl2 # RAC2使用这个配置
export ORACLE_UNQNAME=orcl
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
$ source .bash_profile使配置文件生效（修改环境变量都是在最末行追加，保存后都要source使配置生效）

7.配置oracle、grid用户ssh互信

#su – oracle
$mkdir ~/.ssh
$chmod 755 .ssh 
$/usr/bin/ssh-keygen -t rsa
$/usr/bin/ssh-keygen -t dsa
下面步骤只在节点1执行，将所有的key文件汇总到一个总的认证文件中
$ssh rac1 cat ~/.ssh/id_rsa.pub >> authorized_keys
$ssh rac2 cat ~/.ssh/id_rsa.pub >> authorized_keys
$ssh rac1 cat ~/.ssh/id_dsa.pub >> authorized_keys
$ssh rac2 cat ~/.ssh/id_dsa.pub >> authorized_keys
$ cd ~/.ssh/
$ scp authorized_keys rac2:~/.ssh/  拷贝完整的key到节点2
[oracle@rac2 .ssh]chmod 600 authorized_keys  此处注意是登陆到节点2修改权限
下面步骤2个节点都要执行，否则后续安装会报错，相当重要，得到的结果应该是一个时间
$ssh rac1 date
$ssh rac2 date
$ssh rac1-priv date
$ssh rac2-priv date

grid的互信参照oracle用户处理即可

8.配置裸盘
(1)fdisk -l 通过该命令查看磁盘，可看到我们后面添加上去的磁盘信息，磁盘从sdb到sdf
(2)分区、格式化磁盘。由于是共享磁盘，所以只需要在一个节点上执行即可
在节点1上格式化，以/dev/sdb为例：
#fdisk /dev/sdb
The number of cylinders for this disk is set to 3824.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3824, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-3824, default 3824):
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
格式化完毕后得到如下分区
#ls /dev/sd*
/dev/sda /dev/sda2 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sda1 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
(3)在节点上添加裸设备，注意2个节点都要执行

#vi /etc/udev/rules.d/99-oracle-asmdevices.rules
ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdf1", RUN+="/bin/raw /dev/raw/raw5 %N"
KERNEL=="raw[1]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[2]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[3]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[4]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[5]", MODE="0660", OWNER="grid", GROUP="asmadmin"

(4)启动裸设备，2个节点都执行
#start_udev
(5)查看裸设备，2个节点都要查看
#raw -qa
/dev/raw/raw1: bound to major 8, minor 17
/dev/raw/raw2: bound to major 8, minor 33
/dev/raw/raw3: bound to major 8, minor 49
/dev/raw/raw4: bound to major 8, minor 65
/dev/raw/raw5: bound to major 8, minor 81
9.挂载安装软件文件夹
这里是主机windows系统开启文件夹共享，将压缩包解压后供虚拟机挂载即可 ，这里的用户名密码目录仅供参考
mkdir -p /home/grid/db
mount -t cifs -o username=share,password=123456 //192.168.248.1/DB /home/grid/db
mkdir -p /home/oracle/db
mount -t cifs -o username=share,password=123456 //192.168.248.1/DB /home/oracle/db

10.安装用于环境检查的cvuqdisk（双机）

#cd /home/grid/db/rpm
#rpm -ivh cvuqdisk-1.0.7-1.rpm 

11.手动运行cvu使用验证程序验证Oracle集群要求

#su - grid
$cd /home/grid/db/
$./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -fixup -verbose

注：这里检查见过出现很多32位（i386或i686）的软件包校验不通过，我直接忽略不处理也没有影响系统使用，因此可以忽略，另外NTP、pdksh的报错也可以忽略，校验时2机报 glibc不存在，因此只能重新打包，从网上下载包
glibc-2.17-260.el7_6.4.i686.rpm ，直接安装存在依赖
error: Failed dependencies:
libfreebl3.so is needed by glibc-2.17-260.el7_6.4.i686
libfreebl3.so(N×××AWHASH_3.12.3) is needed by glibc-2.17-260.el7_6.4.i686
百度查询需要打包 nss-softokn-freebl-3.36.0-5.el7_5.i686.rpm ，直接打包报错：
error: Failed dependencies:
libc.so.6 is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.0) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.1) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.1.3) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.3) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.3.4) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.4) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libdl.so.2 is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libdl.so.2(GLIBC_2.0) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libdl.so.2(GLIBC_2.1) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
通过2个包一起安装解决：
#rpm -ivh glibc-2.17-260.el7_6.4.i686.rpm nss-softokn-freebl-3.36.0-5.el7_5.i686.rpm
Preparing... ################################# [100%]
Updating / installing...
1:nss-softokn-freebl-3.36.0-5.el7_5################################# [ 50%]
2:glibc-2.17-260.el7_6.4 ################################# [100%]

---

安装Grid，只需要在1个节点执行即可，另外一个节点自动同步
1.安装流程
（1）开启安装（这里为了更好的可用性用Xmanager来远程安装，配置参考https://www.cnblogs.com/songyuejie/p/6372534.html）

#xhost +
#su - grid
$export DISPLAY=Windows主机IP:0.0
$cd cd /home/grid/db/
$./runInstaller

图形化流程
跳过更新-安装集群

自定义安装

安装语音默认英文

定义集群名字，SCAN Name 为hosts中定义的scan-ip，GNS去勾选

界面只有第一个节点rac1，点击“Add”把第二个节点rac2加上，信息可以参照rac1填写


选择网卡，默认即可

配置ASM，这里选择前面配置的裸盘raw1,raw2,raw3，冗余方式为External即不冗余。因为是不用于，所以也可以只选一个设备。这里的设备是用来做OCR注册盘和votingdisk投票盘的。


配置ASM实例需要为具有sysasm权限的sys用户，具有sysdba权限的asmsnmp用户设置密码，这里设置统一密码，点击OK即可

不选择智能管理

检查ASM实例权限分组情况

选择grid软件安装路径和base目录 ，这里默认即可

选择grid安装清单目录

环境检测出现ntp等刚刚跑脚本检测出的错误，选择忽略全部

安装概要，点击安装后开始安装并且自动复制到rac2

安装grid完成，提示需要root用户依次执行脚本orainstRoot.sh ，root.sh （一定要先在rac1执行完脚本后，才能在其他节点执行）

rac1：
#/u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
#/u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]: （回车）
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
ASM created and started successfully.
Disk Group OCR created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 496abcfc4e214fc9bf85cf755e0cc8e2.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
STATE File Universal Id File Name Disk group

1. ONLINE 496abcfc4e214fc9bf85cf755e0cc8e2 (/dev/raw/raw1) [OCR]
   Located 1 voting disk(s).
   CRS-2672: Attempting to start 'ora.asm' on 'rac1'
   CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
   CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac1'
   CRS-2676: Start of 'ora.OCR.dg' on 'rac1' succeeded
   Configure Oracle Grid Infrastructure for a Cluster ... succeeded

在rac2用同样的方法跑2个脚本，完成后点击上图中的ok按钮再执行下一步，忽略报错完成安装
2.安装grid后的资源检查

#su - grid
$ crsctl check crs

CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online12345
检查Clusterware资源
$crs_stat -t -v

Name Type R/RA F/FT Target State Host
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
检查集群节点

$ olsnodes -n
rac1    1
rac2    2

检查两个节点上的Oracle TNS监听器进程

$ ps -ef|grep lsnr|grep -v 'grep'|grep -v 'ocfs'|awk '{print$9}'
LISTENER_SCAN1
LISTENER

确认针对Oracle Clusterware文件的Oracle ASM功能：
如果在 Oracle ASM 上安装过了OCR和表决磁盘文件，则以Grid Infrastructure 安装所有者的身份，使用给下面的命令语法来确认当前正在运行已安装的Oracle ASM:

$ srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.

3.为数据和快速恢复去创建ASM磁盘组（只在rac1执行即可）

#su - grid 
$ asmca

这里看到安装grid时配置的OCR盘已存在

点击create，添加DATA盘，使用裸盘raw4

同样创建FRA盘，使用裸盘raw5

ASM磁盘组的情况

ASM的实例

---

安装Oracle database软件（RAC）
1.安装流程（只在rac1执行即可，图形化设置参照asm的）
#su - oracle
$ cd db/database
$ ./runInstaller
进入图形化界面，跳过更新选项接着选择只安装数据库软件


选择Oracel Real Application Clusters database installation按钮（默认），点击select all 按钮 确保勾选所有的节点

默认英文安装

选择安装企业版软件

选择安装Oracle软件路径，其中ORACLE_BASE,ORACLE_HOME均选择之前配置好的，默认即可

oracle权限授予用户组，默认即可

安装前的预检查，忽略所有检查异常项

显示安装RAC的概要信息

开始安装，会自动复制到其他节点

安装过程的异常解决方案：
/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib//libnmectl.a(nmectlt.o): In function nmectlt_genSudoProps':
nmectlt.c:(.text+0x76): undefined reference toB_DestroyKeyObject'
nmectlt.c:(.text+0x7f): undefined reference to B_DestroyKeyObject'
nmectlt.c:(.text+0x88): undefined reference toB_DestroyKeyObject'
nmectlt.c:(.text+0x91): undefined reference to `B_DestroyKeyObject'
INFO: collect2: error: ld returned 1 exit status
INFO: /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk:176: recipe for target '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl' failed
make[1]: Leaving directory '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib'
INFO: /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk:52: recipe for target 'emdctl' failed
INFO: make[1]: [/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl] Error 1
INFO: make: [emdctl] Error 2
INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'agent nmhs' of makefile '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk'. See '/u01/app/oraInventory/logs/installActions2017-05-02_12-37-15PM.log' for details.
Exception Severity: 1
方案：
修改“emdctl”的编译参数，编辑“/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk”文件，将
171 #===========================
172 # emdctl
173 #===========================
174
175 $(SYSMANBIN)emdctl:
176 $(MK_EMAGENT_NMECTL)
改为
171 #===========================
172 # emdctl
173 #===========================
174
175 $(SYSMANBIN)emdctl:
176 $(MK_EMAGENT_NMECTL) -lnnz11
然后点击retry
NFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'irman ioracle' of makefile '/u01/app/oracle/product/11.2.0/db_1/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/installActions2019-04-30_03-12-13PM.log' for details.
解决方法如下：
cd $ORACLE_HOME/rdbms/admin
/usr/bin/make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk irman
然后点击retry

安装完，在每个节点用root用户执行脚本

#/u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]: （回车）
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
执行过程出错：
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443.
这是11.0.2.1的经典问题，一个bug，解决办法也很简单，开2个窗口一个跑root.sh在执行root.sh命令出现Adding daemon to inittab的时候在另外一个窗口反复执行/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1
直到没有出现/bin/dd: opening`/var/tmp/.oracle/npohasd': No such file or directory
重新执行root.sh之前别忘了删除配置：/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force-verbose
参考：
http://blog.csdn.net/tianlesoftware/article/details/8207629

到此完成数据库软件安装

2.创建集群数据库
在节点rac1上用oracle用户执行dbca创建RAC数据库（图形设置参照之前）
#su - oracle
$dbca

这里如果看到的部署RAC安装可能是RAC没有启动，grid用户检查RAC资源
$ crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services
$ crs_stat -t -v
CRS-0184: Cannot communicate with the CRS daemon.
看到没有起点，可以在root用户下手动拉起服务
#/etc/init.d/init.ohasd run >/dev/null 2>&1 选择创建数据库

选择自定义数据库（也可以是通用）

配置类型选择Admin-Managed，输入全局数据库名orcl，每个节点实例SID前缀为orcl，选择双节点

选择默认，配置OEM，启用数据库自动维护任务

统一设置sys,system,dbsnmp,sysman用户的密码为oracle

使用ASM存储，使用OMF（oracle的自动管理文件），数据区选择之前创建的DATA磁盘组 （
a.这里有可能出现没有磁盘的问题，参考解决方案https://www.cnblogs.com/cqubityj/p/6828946.html
本次是修改了grid下oracle的权限6751
b.还有可能出现磁盘没有挂载的情况,解决方案：
$ asmcmd
asmcmd>lsdg --查看当前挂载情况
asmcmd>mount DATA 挂载名为data的磁盘）

设置ASM密码为oracle_4U
（这里会报"ORA-01031: insufficient privileges"
先删除文件/u01/app/11.2.0/grid/dbs/orapw+ASM，然后新生成文件，此时的密码为oracle_4U
[grid@node1 bin]$ orapwd file=/u01/app/11.2.0/grid/dbs/orapw+ASM password=oracle_4U
[grid@node1 dbs]$ scp orapw+ASM Node2:/u01/app/11.2.0/grid/dbs/ --拷贝到另外一个节点
[grid@node1 bin]$ sqlplus / as sysasm
SQL>create user asmsnmp identified by oracle_4U;
SQL> grant sysdba to asmsnmp;
然后点击retry即可）

指定数据闪回区，选择之前创建好的FRA磁盘组，不开归档

组件选择，默认

选择字符集AL32UTF8

选择默认的数据存储信息

开始创建数据库，勾选生成数据库的脚本

数据库的概要信息

开始安装


安装过程可能出现的问题：ORA-12547: TNS:lost contact
检查$ORACLE_HOME/bin/oracle 和 $ORACLE_HOME/rdbms/lib/config.o 的文件大小是否为0，
如果大小为0，需要重新编译oracle软件：
[oracle@normal backup]$ relink all
writing relink log to: /u01/app/oracle/product/11.2.0/db_1/install/relink.log
完成数据库实例安装

---

RAC维护
1.查看服务状态（gsd 是为了兼容9i，10G以后不再启动，忽略offline）
#su - grid
$crs_stat -t
Name Type Target State Host
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora.FRA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac1
ora.OCR.dg ora....up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac1
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.orcl.db ora....se.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora.scan1.vip ora....ip.type ONLINE ONLINE rac1

2.检查集群状态

$srvctl status database -d orcl 
Instance orcl1 is running on node rac1 
Instance orcl2 is running on node rac2 

3.检查CRS状态
本地

$crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

集群

$ crsctl check cluster
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

4.查看集群中节点配置信息

$ olsnodes
rac1
rac2
$ olsnodes -n
rac1    1
rac2    2
$ olsnodes -n -i -s -t
rac1    1       rac1-vip        Active  Unpinned
rac2    2       rac2-vip        Active  Unpinned

5.查看集群件的表决磁盘信息

.$ crsctl query css votedisk
STATE    File Universal Id                File Name Disk group
 1. ONLINE   496abcfc4e214fc9bf85cf755e0cc8e2 (/dev/raw/raw1) [OCR]
Located 1 voting disk(s).

6.查看集群SCAN VIP信息（参考示例）

$ srvctl config scan
SCAN name: scan-ip, Network: 1/192.168.248.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /scan-ip/192.168.248.110

$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

7.启、停集群数据库
整个集群的数据库启停
进入grid用户

$ srvctl stop database -d orcl
$ srvctl status database -d orcl
$ srvctl start database -d orcl
关闭所有节点 
进入root用户 
关闭所有节点 
# /u01/app/11.2.0/grid/bin/crsctl stop crs 

实际只关闭了当前结点
启动过程出现的异常（由于内存不够后面降低了内存重新启动虚拟机）：
报错：ORA-00845: MEMORY_TARGET not supported on this system
$ srvctl start database -d orcl
PRCR-1079 : Failed to start resource ora.orcl.db
ORA-00845: MEMORY_TARGET not supported on this system
CRS-2674: Start of 'ora.orcl.db' on 'rac1' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-00845: MEMORY_TARGET not supported on this system
CRS-2674: Start of 'ora.orcl.db' on 'rac2' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
解决办法：
加大/dev/shm，使他大于系统参数，由于停机后无法获取系统参数，在此可以设置一个总内存大小：
#cat /etc/fstab | grep tmpfs
tmpfs /dev/shm tmpfs defaults,size=4G 0 0
#mount -o remount,size=4G /dev/shm
#df -h | grep shm
tmpfs 4.0G 0 4.0G 0% /dev/shm
接着重新启动即可，然后进入数据库修改内存
$ sqlplus / as sysdba
SQL> show parameter target; 其中/dev/shm必须大于MEMORY_TARGET，需要做修改下面2个值，改完需要重新启动数据库
sql> alter system set memory_max_target=12288M scope=spfile;
sql> alter system set memory_target=12288M scope=spfile;

---

EM管理（oracle用户，其中一个节点执行即可，访问地址https://节点IP:1158/em 用户名密码用上文设置的system之类的）

$ emctl status dbconsole
$ emctl start dbconsole
$ emctl stop dbconsole

本地sqlplus连接
安装oracle客户端版修改tsnames.ora
D:\develop\app\orcl\product\11.2.0\client_1\network\admin\tsnames.ora
添加
ORCL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.248.110)(PORT = 1521))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)12345678910

这里的HOST写的是scan-ip

sqlplus sys/oracle@RAC_ORCL as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 4月 14 14:37:30 2016

Copyright (c) 1982, 2010, Oracle. All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select instance_name, status from v$instance;

INSTANCE_NAME STATUS

---

orcl1 OPEN

当开启第二个命令行窗口连接时，发现实例名为orcl2，可以看出，scan-ip的加入可以具有负载均衡的作用。

如果连接的时候有报错：ORA-12545
解决方案：
a.可以在本机hosts文件添加双机的vip
b.或者修改数据库的监听配置
SQL> show parameter local

NAME TYPE VALUE

---

local_listener string (ADDRESS=(PROTOCOL=TCP)(HOST=scan-vip)(PORT=1521))
parallel_force_local boolean FALSE

修改local_listener主机名为VIP-IP地址
SQL> alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST= 192.168.149.202)(PORT=1521))' sid='orcl2';

System altered.

SQL> alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST= 192.168.149.201)(PORT=1521))' sid='orcl1';

System altered.

容灾演练（模拟单机运行）
参考文档
https://blog.csdn.net/yujiajia111/article/details/82998601
本文参考文档：
https://blog.csdn.net/u014595668/article/details/51160783
https://www.cnblogs.com/lijiaman/p/7673795.html