在VirtualBox Linux 7u2 中安装Oracle RAC 12.2.0.1.0
环境描述
Lenovo X270, Win 10，VirtualBox 5.2.12
所有的虚拟机安装在移动硬盘上，操作系统为Oracle Linux 7.2
数据库为 12.2.0.1.0，从官网下载，由于是RAC，因此需要下载 linuxx64_12201_database.zip和 linuxx64_12201_grid_home.zip两个安装包
两个虚拟机的主机名分别为ol7-122-rac1.localdomain和ol7-122-rac2.localdomain，分配配置3块网卡：NAT，Host Only和Internal，分别用于上网，对外服务和RAC内部通讯
配置第一个虚机
扩容root disk到40G空间
我的虚拟机模板是已制作好的，但根盘只有24G，比较小，因此需要扩容到40G。
我的磁盘是VMDK格式的，不能直接扩展（当然你也可以添加新的磁盘来扩展），因此先转换为VDI格式。转换前必须将磁盘从VM detach。
"c:\Program Files\Oracle\VirtualBox\VBoxManage" clonemedium disk --format VDI rootdisk1.vmdk rootdisk1.vdi
"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd --compact rootdisk1.vdi
"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd --compact rootdisk1.vdiclonemedium命令较慢，其它都很快，完成后再retach到VM。
可以看到我们的盘以及扩到40G，不过文件系统还没有扩
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 23.5G 0 part
├─ol-root 252:0 0 21.1G 0 lvm /
└─ol-swap 252:1 0 2.4G 0 lvm [SWAP]
sr0 11:0 1 1024M 0 rom
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 22G 3.4G 18G 16% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 140K 1.9G 1% /dev/shm
tmpfs 1.9G 8.8M 1.8G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 497M 211M 287M 43% /boot
tmpfs 370M 24K 370M 1% /run/user/0
所以先利用新增的16G的容量建立一个分区：
# fdisk /dev/sda
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): n
Partition type:
p primary (2 primary, 0 extended, 2 free)
e extended
Select (default p):
Using default response p
Partition number (3,4, default 3):
First sector (50331648-83886079, default 50331648):
Using default value 50331648
Last sector, +sectors or +size{K,M,G} (50331648-83886079, default 83886079):
Using default value 83886079
Partition 3 of type Linux and of size 16 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
创建PV：
# fdisk -l /dev/sda
Disk /dev/sda: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0005ec3b
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 50331647 24652800 8e Linux LVM
/dev/sda3 50331648 83886079 16777216 83 Linux
# partprobe /dev/sda
# pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created
扩展VG：
# vgdisplay
--- Volume group ---
VG Name ol
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 23.51 GiB
PE Size 4.00 MiB
Total PE 6018
Alloc PE / Size 6008 / 23.47 GiB
Free PE / Size 10 / 40.00 MiB
VG UUID yOD2tO-qDZe-d19l-1P6J-cSA4-VN0I-BXxChX
# vgextend ol /dev/sda3
Volume group "ol" successfully extended
扩展LV:
# vgdisplay | grep Free
Free PE / Size 4105 / 16.04 GiB
# lvdisplay
--- Logical volume ---
LV Path /dev/ol/swap
LV Name swap
VG Name ol
LV UUID 13PRF3-7Qwt-ssrv-eQNV-7LZw-dgDH-dyCjhY
LV Write Access read/write
LV Creation host, time localhost, 2016-10-22 21:25:09 +0800
LV Status available
# open 2
LV Size 2.40 GiB
Current LE 615
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 252:1
--- Logical volume ---
LV Path /dev/ol/root
LV Name root
VG Name ol
LV UUID rir5ib-Qrhd-e0e3-pEw6-1ffE-fXH2-5VRrF3
LV Write Access read/write
LV Creation host, time localhost, 2016-10-22 21:25:09 +0800
LV Status available
# open 1
LV Size 21.07 GiB
Current LE 5393
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 252:0
# lvextend -l +4105 /dev/ol/root
Size of logical volume ol/root changed from 21.07 GiB (5393 extents) to 37.10 GiB (9498 extents).
Logical volume root successfully resized.最后扩展文件系统：
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 22G 3.3G 18G 16% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 140K 1.9G 1% /dev/shm
tmpfs 1.9G 8.8M 1.8G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 497M 211M 287M 43% /boot
tmpfs 370M 28K 370M 1% /run/user/0
# xfs_growfs /
meta-data=/dev/mapper/ol-root isize=256 agcount=4, agsize=1380608 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=5522432, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2696, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 5522432 to 9725952
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 38G 3.3G 34G 9% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 140K 1.9G 1% /dev/shm
tmpfs 1.9G 8.8M 1.8G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 497M 211M 287M 43% /boot
tmpfs 370M 28K 370M 1% /run/user/0
配置虚拟机网络
配置3块网卡：NAT，Host Only和Internal，分别用于上网，对外服务和RAC内部通讯。设备名分别为enp0s3，enp0s8和enp0s9
网络配置信息如下：
hostname: ol7-122-rac1.localdomain
enp0s3 (eth0): DHCP (Connect Automatically)
enp0s8 (eth1): IP=192.168.56.101, Subnet=255.255.255.0
enp0s9 (eth2): IP=192.168.1.101, Subnet=255.255.255.0首先修改主机名为ol7-122-rac1.localdomain：
# echo ol7-122-rac1.localdomain > /etc/hostname
# cat /etc/hostname
ol7-122-rac1.localdomain然后通过GUI在system tools|settings|network中配置，NAT网卡enp0s3只需启用，即可自动配置/etc/resolv.conf并连接互联网。
配置完成后，显示如下：
# ifconfig -a
enp0s3: flags=4163 mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fe56:6319 prefixlen 64 scopeid 0x20
ether 08:00:27:56:63:19 txqueuelen 1000 (Ethernet)
RX packets 60 bytes 13212 (12.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 203 bytes 17576 (17.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s8: flags=4163 mtu 1500
inet 192.168.56.101 netmask 255.255.255.0 broadcast 192.168.56.255
inet6 fe80::a00:27ff:fe6c:93cb prefixlen 64 scopeid 0x20
ether 08:00:27:6c:93:cb txqueuelen 1000 (Ethernet)
RX packets 49 bytes 7446 (7.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 44 bytes 6501 (6.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s9: flags=4163 mtu 1500
inet 192.168.1.101 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::a00:27ff:fe59:828e prefixlen 64 scopeid 0x20
ether 08:00:27:59:82:8e txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52 bytes 7705 (7.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 8 bytes 624 (624.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 624 (624.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099 mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:21:c4:37 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5 bytes 931 (931.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0-nic: flags=4099 mtu 1500
ether 52:54:00:21:c4:37 txqueuelen 500 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 稍微修改一下网络配置文件：
# cd /etc/sysconfig/network-scripts/
# ls ifcfg-*
ifcfg-enp0s3 ifcfg-enp0s8 ifcfg-enp0s9 ifcfg-lo
# cat ifcfg-enp0s3
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp0s3
UUID=2531730c-a7b8-4e0c-b415-072ed3aac9dc
DEVICE=enp0s3
ONBOOT=no
# cat ifcfg-enp0s8
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.56.101
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp0s8
UUID=c29fe81b-3e1b-4bcd-9794-96ca5337ee90
ONBOOT=yes
# echo "DEVICE=enp0s8" >> ifcfg-enp0s8
# echo "DEVICE=enp0s9" >> ifcfg-enp0s9
# systemctl restart network
准备Oracle安装环境
如果你已建立oracle用户，请将其删除
# userdel -r oracle然后利用oracle提供的RPM即可：
# yum install oracle-database-server-12cR2-preinstall -y
# passwd oracle此包在安装过程中自动建立用户和组，配置系统参数等：
# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba)
# more /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
# oracle-database-server-12cR2-preinstall setting for fs.file-max is 6815744
fs.file-max = 6815744
# oracle-database-server-12cR2-preinstall setting for kernel.sem is '250 32000 100 128'
kernel.sem = 250 32000 100 128
......创建目录和设置环境变量
mkdir -p /u01/app/12.2.0.1/grid
mkdir -p /u01/app/oracle/product/12.2.0.1/db_1
chown -R oracle:oinstall /u01
chmod -R 775 /u01/在~oracle/.bash_profile中添加以下：
# Oracle Settings
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=ol7-122-rac1.localdomain
export ORACLE_UNQNAME=CDBRAC
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/12.2.0.1/grid
export DB_HOME=$ORACLE_BASE/product/12.2.0.1/db_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'创建文件db_env和db_env:
$ cat db_env
export ORACLE_SID=cdbrac1
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
$ cat grid_env
export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
配置DNS解析
这主要是为SCAN VIP使用。首先添加以下文本到/etc/hosts：
# Public
192.168.56.101 ol7-122-rac1.localdomain ol7-122-rac1
192.168.56.102 ol7-122-rac2.localdomain ol7-122-rac2
# Private
192.168.1.101 ol7-122-rac1-priv.localdomain ol7-122-rac1-priv
192.168.1.102 ol7-122-rac2-priv.localdomain ol7-122-rac2-priv
# Virtual
192.168.56.103 ol7-122-rac1-vip.localdomain ol7-122-rac1-vip
192.168.56.104 ol7-122-rac2-vip.localdomain ol7-122-rac2-vip
# SCAN
#192.168.56.105 ol7-122-cluster-scan.localdomain ol7-122-cluster-scan
#192.168.56.106 ol7-122-cluster-scan.localdomain ol7-122-cluster-scan
#192.168.56.107 ol7-122-cluster-scan.localdomain ol7-122-cluster-scan然后使用BIND配置DNS，参见过程：
# yum install bind-libs bind bind-utils修改文件/etc/named.conf如下，其中zone部分指明了解析文件为localdomain.zone：
options {
......
allow-query { localhost; 192.168.0.0;};
......
};
......
zone "localdomain." IN {
type master;
file "localdomain.zone";
allow-update { none; };
};
.......新建解析文件/var/named/localdomain.zone如下：
$TTL 86400
@ IN SOA localhost root.localhost (
42 ; serial (d. adams)
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum
IN NS localhost
localhost IN A 127.0.0.1
ol7-122-rac1 IN A 192.168.56.101
ol7-122-rac2 IN A 192.168.56.102
ol7-122-rac1-priv IN A 192.168.1.101
ol7-122-rac2-priv IN A 192.168.1.102
ol7-122-rac1-vip IN A 192.168.56.103
ol7-122-rac2-vip IN A 192.168.56.104
ol7-122-cluster-scan IN A 192.168.56.105
ol7-122-cluster-scan IN A 192.168.56.106
ol7-122-cluster-scan IN A 192.168.56.107
现在可以禁止NAT网卡enp0s3了，以免Network Manager重启时修改/etc/resolv.conf文件
# ifdown enp0s3
# cat /etc/resolv.conf
search localdomain
nameserver 127.0.0.1
# systemctl restart network
# 确认resolv.conf文件没有变化启动DNS服务：
# systemctl enable named
Created symlink from /etc/systemd/system/multi-user.target.wants/named.service to /usr/lib/systemd/system/named.service.
# systemctl start named
# systemctl status named
● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2018-08-23 17:19:27 CST; 6s ago
Process: 13449 ExecStop=/bin/sh -c /usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 13460 ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF} $OPTIONS (code=exited, status=0/SUCCESS)
Process: 13457 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)
Main PID: 13462 (named)
CGroup: /system.slice/named.service
└─13462 /usr/sbin/named -u named -c /etc/named.conf
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: managed-keys-zone: loaded serial 2
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: zone 0.in-addr.arpa/IN: loaded serial 0
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: zone localdomain/IN: loaded serial 42
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: zone localhost.localdomain/IN: loaded serial 0
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: zone localhost/IN: loaded serial 0
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: all zones loaded
Aug 23 17:19:27 ol7-122-rac1.localdomain named[13462]: running
Aug 23 17:19:27 ol7-122-rac1.localdomain systemd[1]: Started Berkeley Internet Name Domain (DNS).
验证：
# nslookup ol7-122-rac1
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: ol7-122-rac1.localdomain
Address: 192.168.56.101
# nslookup ol7-122-rac2-priv
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: ol7-122-rac2-priv.localdomain
Address: 192.168.1.102
# nslookup ol7-122-cluster-scan
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: ol7-122-cluster-scan.localdomain
Address: 192.168.56.107
Name: ol7-122-cluster-scan.localdomain
Address: 192.168.56.105
Name: ol7-122-cluster-scan.localdomain
Address: 192.168.56.106
禁用 SELINUX和防火墙
虽然不是必需，但这样方便很多。
编辑/etc/selinux/config即可，重启生效。用sestatus确认修改成功。
# systemctl stop firewalld
# systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
创建安装介质共享目录
此共享目录供RAC两个虚拟机一起访问介质用，以免将介质拷到每一个主机中。
记得勾选make permanent和Auto-mount。
将数据库安装文件解压到共享目录下，将Grid安装文件拷贝到共享目录下，但不解压。因为grid的安装即解压。
最后将vboxsf添加到用户oracle的辅助组中，使oracle用户可以访问共享目录
# usermod -a -G vboxsf oracle
# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),983(vboxsf),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba)
创建RAC共享磁盘
有UDEV和ASMLIB两种方式，此处使用的是前者。
总容量需超过40G，为节省时间，创建2个20G磁盘。
"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm1.vdi --size 20480 --format VDI --variant Fixed
"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm2.vdi --size 20480 --format VDI --variant Fixed
停止虚机，然后attach磁盘：
"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-122-rac1 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-122-rac1 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable启动虚机
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
├─sda1 8:1 0 500M 0 part /boot
├─sda2 8:2 0 23.5G 0 part
│ ├─ol-root 252:0 0 37.1G 0 lvm /
│ └─ol-swap 252:1 0 2.4G 0 lvm [SWAP]
└─sda3 8:3 0 16G 0 part
└─ol-root 252:0 0 37.1G 0 lvm /
sdb 8:16 0 20G 0 disk
sdc 8:32 0 20G 0 disk
sr0 11:0 1 1024M 0 rom
# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x70762ef1.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-41943039, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
# 对sdc执行系统操作
# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x6de98a4b.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-41943039, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
# ls -l /dev/sdb* /dev/sdc*
brw-rw---- 1 root disk 8, 16 Aug 23 21:09 /dev/sdb
brw-rw---- 1 root disk 8, 17 Aug 23 21:09 /dev/sdb1
brw-rw---- 1 root disk 8, 32 Aug 23 21:10 /dev/sdc
brw-rw---- 1 root disk 8, 33 Aug 23 21:10 /dev/sdc1
新建以下文件：
# cat /etc/scsi_id.config
options=-g
查看此两设备的SCSI ID，并新建文件如下：
# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VB6e537b29-59f0029b
# /usr/lib/udev/scsi_id -g -u -d /dev/sdc
1ATA_VBOX_HARDDISK_VB6547abe2-0b36cd13
# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB6e537b29-59f0029b", SYMLINK+="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB6547abe2-0b36cd13", SYMLINK+="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
#
更新并测试：
# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1
# udevadm test /block/sdb/sdb1
重启udev设备：
# udevadm control --reload-rules
# ll /dev/asm*
lrwxrwxrwx 1 root root 4 Aug 23 21:18 /dev/asm-disk1 -> sdb1
lrwxrwxrwx 1 root root 4 Aug 23 21:17 /dev/asm-disk2 -> sdc1
配置时间同步
可以使用NTP或CHRONY，此处使用后者，没有配置完全，之后的报错可以忽略
# systemctl enable chronyd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/chronyd.service to /usr/lib/systemd/system/chronyd.service.
# systemctl start chronyd.service
通过克隆配置第二个RAC节点
shutdown第一台虚机，并由其克隆第二台虚机。
"c:\Program Files\Oracle\VirtualBox\VBoxManage" clonehd "E:\RACLab\node1\rootdisk1.vdi" "E:\RACLab\node2\rootdisk1.vdi"创建虚机ol7-122-rac2，并attach克隆盘，然后添加共享盘。另外记得和第一台虚机一样添加3块网卡，分别为NAT，Host Only和Internal。
E:\>cd RACLab\ShardDisks
E:\RACLab\ShardDisks>"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-122-rac2 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
E:\RACLab\ShardDisks>"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-122-rac2 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
E:\>tree /F RACLab
卷 Xiaoyu 2016 Drive 的文件夹 PATH 列表
卷序列号为 82EB-274D
E:\RACLAB
├─Node1
│ rootdisk1.vdi
│
├─Node2
│ rootdisk1.vdi
│
└─ShardDisks
asm1.vdi
asm2.vdi调整主机名和网络IP地址
启动第二台虚机并进行调整，调整的项目包括修改主机名，网络和相关配置文件：
# echo ol7-122-rac2.localdomain > /etc/hostname
[root@ol7-122-rac2 ~]# cd /etc/sysconfig/network-scripts/
[root@ol7-122-rac2 network-scripts]# vi ifcfg-enp0s8 修改101到102
[root@ol7-122-rac2 network-scripts]# vi ifcfg-enp0s9 修改101到102
[root@ol7-122-rac2 network-scripts]# systemctl restart network
[root@ol7-122-rac2 network-scripts]# ifconfig -a
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:d4:4d:2d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.56.102 netmask 255.255.255.0 broadcast 192.168.56.255
inet6 fe80::a00:27ff:fe62:32d prefixlen 64 scopeid 0x20
ether 08:00:27:62:03:2d txqueuelen 1000 (Ethernet)
RX packets 2 bytes 303 (303.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 51 bytes 7529 (7.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.102 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::a00:27ff:fe56:ce13 prefixlen 64 scopeid 0x20
ether 08:00:27:56:ce:13 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 47 bytes 7115 (6.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 168 bytes 13560 (13.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 168 bytes 13560 (13.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:21:c4:37 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7 bytes 1378 (1.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0-nic: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 52:54:00:21:c4:37 txqueuelen 500 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
重启虚机。
检查并修正配置
确保resolv.conf如下：
# cat /etc/resolv.conf
search localdomain
nameserver 127.0.0.1如果不是请执行以下：
ifdown enp0s3
systemctl restart network
修改resolv.conf文件，使其与第一台主机一致修改~oracle目录下的db_env和grid_env文件，将其中的1替换为2，例如ASM1到ASM2。
验证DNS工作正常：
$ nslookup ol7-122-cluster-scan
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: ol7-122-cluster-scan.localdomain
Address: 192.168.56.107
Name: ol7-122-cluster-scan.localdomain
Address: 192.168.56.105
Name: ol7-122-cluster-scan.localdomain
Address: 192.168.56.106
确保共享设备正常：
$ ls -l /dev/asm*
lrwxrwxrwx 1 root root 4 Aug 23 21:49 /dev/asm-disk1 -> sdb1
lrwxrwxrwx 1 root root 4 Aug 23 21:49 /dev/asm-disk2 -> sdc1
启动第一台主机，确保以下的ping在两台主机上执行正常，也就是对外服务网络和私网可以互通：
ping -c 3 ol7-122-rac1
ping -c 3 ol7-122-rac1-priv
ping -c 3 ol7-122-rac2
ping -c 3 ol7-122-rac2-priv仿照第一台主机建立共享目录，因为我发现没有克隆过来
安装Grid Infrastructure
在节点1采用GUI安装。
$ ll /media/sf_share/
total 2924505
drwxrwx--- 1 root vboxsf 4096 Jan 27 2017 database
-rwxrwx--- 1 root vboxsf 2994687209 Aug 22 11:55 linuxx64_12201_grid_home.zip
解压安装包，需要一些时间：
export SOFTWARE_LOCATION=/media/sf_share/
cd /u01/app/12.2.0.1/grid
unzip -q $SOFTWARE_LOCATION/linuxx64_12201_grid_home.zip以root用户在两个节点安装GI软件：
su -
# Local node.
cd /u01/app/12.2.0.1/grid/cv/rpm
rpm -Uvh cvuqdisk*
# Remote node.
scp ./cvuqdisk* root@ol7-122-rac2:/tmp
ssh root@ol7-122-rac2 rpm -Uvh /tmp/cvuqdisk*然后使用oracle用户开始GUI安装
cd /u01/app/12.2.0.1/grid
./gridSetup.sh就不上图了。
- Configure Oracle Grid Infrastructure for a New Cluster
- Configure an Oracle Standalone Cluster
- cluster name=ol7-122-cluster，scan name=ol7-122-cluster-scan，port=1521
- 添加节点2到GI，并配置SSH 联通性
- 网卡部分，enp0s8为public，enp0s9为ASM & Private
- 磁盘部分，选择之前建立的两个共享盘，并设置external redundancy
- oracle base使用默认的/u01/app/oracle
- 先决条件检查 - NTP的错误可忽略。而avahi-daemon的错误可以修复
- 忽略这些警告，开始安装
- 执行root脚本比较耗时
- 最后一步会报错，查看日志是chrony的错，没有关系，继续
验证GI安装成功
安装完成后，较晚，就关机了，不过没关系。重启后GI也会自动起来
[oracle@ol7-122-rac1 ~]$ grid_env
[oracle@ol7-122-rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE OFFLINE ol7-122-rac1 STABLE
ONLINE ONLINE ol7-122-rac2 STABLE
ora.DATA.dg
ONLINE OFFLINE ol7-122-rac1 STABLE
ONLINE ONLINE ol7-122-rac2 STABLE
ora.LISTENER.lsnr
ONLINE OFFLINE ol7-122-rac1 STABLE
ONLINE INTERMEDIATE ol7-122-rac2 Not All Endpoints Re
gistered,STABLE
ora.chad
ONLINE OFFLINE ol7-122-rac1 STABLE
ONLINE OFFLINE ol7-122-rac2 STABLE
ora.net1.network
ONLINE ONLINE ol7-122-rac1 STABLE
ONLINE ONLINE ol7-122-rac2 STABLE
ora.ons
ONLINE OFFLINE ol7-122-rac1 STABLE
ONLINE ONLINE ol7-122-rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE INTERMEDIATE ol7-122-rac2 Not All Endpoints Re
gistered,STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE INTERMEDIATE ol7-122-rac2 Not All Endpoints Re
gistered,STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE INTERMEDIATE ol7-122-rac2 Not All Endpoints Re
gistered,STABLE
ora.MGMTLSNR
1 ONLINE ONLINE ol7-122-rac2 169.254.27.15 192.16
8.1.102,STABLE
ora.asm
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE ol7-122-rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE ol7-122-rac2 STABLE
ora.mgmtdb
1 ONLINE OFFLINE ol7-122-rac2 STARTING
ora.ol7-122-rac1.vip
1 ONLINE INTERMEDIATE ol7-122-rac2 FAILED OVER,STABLE
ora.ol7-122-rac2.vip
1 ONLINE ONLINE ol7-122-rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE ol7-122-rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE ol7-122-rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE ol7-122-rac2 STABLE
ora.scan3.vip
1 ONLINE ONLINE ol7-122-rac2 STABLE
--------------------------------------------------------------------------------
安装数据库
以oracle用户用GUI安装
$ cd /media/sf_share/database
$ ./runInstaller
- Install database software only
- Oracle Real Application Clusters database installation
- 选择两个节点
- 选择Enterprise Edition（7.5G）
- 接受默认的”/u01/app/oracle” as the Oracle base”和”/u01/app/oracle/product/12.2.0.1/db_1”
- 接受默认的group
- 先决条件检查，有NTP的警告，可以忽略
- 开始安装
- 最后在两个主机上执行脚本，很快。
- 成功 2018/8/24日9：27
空间使用情况
节点1
[oracle@ol7-122-rac1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 38G 21G 17G 56% /
devtmpfs 2.8G 8.0K 2.8G 1% /dev
tmpfs 2.8G 645M 2.2G 23% /dev/shm
tmpfs 2.8G 9.1M 2.8G 1% /run
tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup
/dev/sda1 497M 211M 287M 43% /boot
share 100G 89G 12G 89% /media/sf_share
tmpfs 572M 8.0K 572M 1% /run/user/54321
节点2
[oracle@ol7-122-rac2 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 38G 20G 18G 53% /
devtmpfs 2.8G 8.0K 2.8G 1% /dev
tmpfs 2.8G 216K 2.8G 1% /dev/shm
tmpfs 2.8G 9.1M 2.8G 1% /run
tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup
/dev/sda1 497M 211M 287M 43% /boot
share 100G 89G 12G 89% /media/sf_share
tmpfs 572M 8.0K 572M 1% /run/user/54321
开始安装前是3.3G，因此安装GI和数据库软件使用了17-18G左右。
参考
- Oracle Database 12c Release 2 (12.2) RAC On Oracle Linux 7 Using VirtualBox
- DEPLOYING ORACLE RAC DATABASE 12C RELEASE 2 ON RED HAT ENTERPRISE LINUX 7
- ASMFD: Implementing ASM Filter Driver after 12.2 GI Installation (文档 ID 2305707.1)
- https://oracle-base.com/articles/10g/asm-using-asmlib-and-raw-devices
- http://www.oracle.com/technetwork/topics/linux/index-101839.html