如何在云环境下部署oracle18c RAC(京东云公有云)

理论篇

安装oracle RAC　主要需要几个条件

• 私有网络和公共网络隔离（不在同一网段）
• 共享存储
• 网络需要支持mulitcast

我们可以利用京东云提供的基础设施解决以上问题

• 通过创建"创建无预设CIDR私有网络"的vpc进行网络隔离，实现public和private网段隔离
• 通过创建"多点挂载存储"实现ASM所需的共享存储
• 通过n2n点对点实现multicast支持

 

实施篇

 

网络配置

• 创建“创建无预设CIDR私有网络”vpc
  控制台=>网络=>私有网络(vpc)=>创建=>选择"创建无预设CIDR私有网络"
• 创建子网
  分别创建两个不同网段的子网
  10.0.1.0/24
  192.168.1.0/24

 

主机创建

 

创建共享磁盘

控制台=>弹性计算=>云硬盘=>创建=>选择"多点挂载"（目前仅华东有此选项）

 

创建弹性网卡

192.168.1.10 rac1private
192.168.1.20 rac2private

 

修改hosts文件

vi /etc/hosts

 

#oracle rac public ip
10.0.0.10 rac1
10.0.0.20 rac2

#oracle rac vip
10.0.0.11 rac1vip
10.0.0.21 rac2vip

#oracle rac scanip
10.0.0.30 scan-rac

#oracle rac private ip
192.168.２.10 rac1private
192.168.２.20 rac2private

这里可以利用弹性网卡来绑定多个ip地址来实现public ip和vip

 

创建用户

https://docs.oracle.com/en/database/oracle/oracle-database/18/cwlin/identifying-an-oracle-software-owner-user-account.html#GUID-0A95F4B1-1045-455D-9897-A23012E4E27F

 

用户和组：
/usr/sbin/groupadd -g 54321 oinstall
/usr/sbin/groupadd -g 54322 dba
/usr/sbin/groupadd -g 54323 oper
/usr/sbin/groupadd -g 54324 backupdba
/usr/sbin/groupadd -g 54325 dgdba
/usr/sbin/groupadd -g 54326 kmdba
/usr/sbin/groupadd -g 54327 asmdba
/usr/sbin/groupadd -g 54328 asmoper
/usr/sbin/groupadd -g 54329 asmadmin
/usr/sbin/groupadd -g 54330 racdba

/usr/sbin/useradd -u 54321 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,oper oracle
/usr/sbin/useradd -u 54331 -g oinstall -G dba,asmdba,asmoper,asmadmin grid

echo oracle | passwd --stdin oracle
echo oracle | passwd --stdin grid

11、目录：
mkdir -p /u01/app/18.3.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

12、环境变量

节点1：
--grid
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/18.3.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

--oracle用户
export ORACLE_SID=db1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/18.3.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib


节点2：
--gird
export ORACLE_SID=+ASM2
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/18.3.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

--oracle用户
export ORACLE_SID=db2
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/18.3.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

• 验证用户组

 

$ grep "oinstall" /etc/group
oinstall:x:54321:grid,oracle
You can then use the ID command to verify that the Oracle installation owners you intend to use have the Oracle Inventory group as their primary group. For example:

$ id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba), 
54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54327(asmdba),54330(racdba)

$ id grid
uid=54331(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),
54327(asmdba),54328(asmoper),54329(asmadmin),54330(racdba)

 

操作系统参数

• vi /etc/sysctl.conf
  sysctl -p /etc/sysctl.d/*.conf

 

fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 67108864
kernel.shmmax = 274877906944
kernel.shmmni = 4096
#kernel.sem = 250 32000 100 128
kernel.sem = 5010 641280 5010 128
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

• 限制文件：vi /etc/security/limits.conf

 

grid soft nproc 2047 
grid hard nproc 16384 
grid soft nofile 1024 
grid hard nofile 65536 
grid soft stack 10240 
oracle soft nproc 2047 
oracle hard nproc 16384 
oracle soft nofile 1024 
oracle hard nofile 65536 
oracle soft stack 10240
* soft memlock 524288000
* hard memlock 524288000

• vi /etc/pam.d/login 文件
  session required pam_limits.so

• vi /etc/fstab
  tmpfs /dev/shm tmpfs size=5000m 0 0

• /etc/sysconfig/network
  NOZEROCONF=yes

• vi /etc/profile

 

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
    if [ $SHELL = "/bin/ksh" ]; then
        ulimit -p 16384
        ulimit -n 65536
    else
        ulimit -u 16384 -n 65536
    fi
    umask 022
fi

 

安装必要包

 

yum install -y gcc-c++
yum install -y bc
yum install -y binutils
yum install -y compat-libcap1
yum install -y compat-libstdc++-33.i686    
yum install -y compat-libstdc++-33.x86_64
yum install -y glibc-2.17-36.el7.i686
yum install -y glibc-2.17-36.el7.x86_64
yum install -y glibc-devel.i686
yum install -y glibc-devel.x86_64
yum install -y ksh
yum install -y libaio.i686
yum install -y libaio.x86_64
yum install -y libaio-devel.i686
yum install -y libaio-devel.x86_64
yum install -y libX11.i686
yum install -y libX11.x86_64
yum install -y libXau.i686
yum install -y libXau.x86_64
yum install -y libXi.i686
yum install -y libXi.x86_64
yum install -y libXtst.i686
yum install -y libXtst.el7.x86_64
yum install -y libgcc.i686
yum install -y libgcc.x86_64
yum install -y libstdc++.i686
yum install -y libstdc++
yum install -y libstdc++-devel.i686
yum install -y libstdc++-devel.x86_64
yum install -y libxcb.i686
yum install -y libxcb.x86_64
yum install -y make
yum install -y nfs-utils 
yum install -y net-tools
yum install -y python
yum install -y python-configshell
yum install -y python-rtslib
yum install -y python-six
yum install -y smartmontools
yum install -y sysstat
yum install -y targetcli

• 为保证xwindow正常显示的必要安装

 

yum groupinstall "X Window System" "Fonts" -y 

 

准备asm磁盘组（所有节点操作）

• asmlib方式
  参考文档：https://docs.microsoft.com/zh-tw/azure/virtual-machines/workloads/oracle/configure-oracle-asm

  • 从 https://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel7-2773795.html下载rpm
  • 安装
    rpm -Uvh oracleasm-support-2.1.11-2.el7.x86_64.rpm
    yum -y install kmod-oracleasm.x86_64
    rpm -ivh oracleasmlib-2.0.12-1.el7.x86_64.rpm
  • 配置asm

  /usr/sbin/oracleasm configure -i
  

  Configuring the Oracle ASM library driver.
  
  This will configure the on-boot properties of the Oracle ASM library
  driver. The following questions will determine whether the driver is
  loaded on boot and what permissions it will have. The current values
  will be shown in brackets ('[]'). Hitting  without typing an
  answer will keep that current value. Ctrl-C will abort.
  
  Default user to own the driver interface []: grid
  Default group to own the driver interface []: asmadmin
  Start Oracle ASM library driver on boot (y/n) [n]: y
  Scan for Oracle ASM disks on boot (y/n) [y]: y
  Writing Oracle ASM library driver configuration: done
  
  

  此处需要reboot

  • 创建asm磁盘组

  oracleasm createdisk ocr /dev/vdb1
  oracleasm createdisk data /dev/vdc1
  oracleasm scandisks
  oracleasm listdisks
  

  • 清除磁盘数据

  dd if=/dev/zero of=/dev/vdb1 bs=1M count=1000
  

 

n2n安装及配置

• 安装

  yum install -y gcc
  yum install -y openssl-devel
  

  git clone https://github.com/ntop/n2n.git
  cd n2n
  git checkout 2.4-stable
  
  make
  make PREFIX=/opt/n2n install
  

• 安装完毕后启动supernode服务

  nohup /opt/n2n/sbin/supernode -l 65530 &
  

• 在RAC节点安装ntop软件（计划加入网络的节点都需要安装ntop）

  scp -r /opt/n2n 10.0.0.10:/opt
  scp -r /opt/n2n 10.0.0.20:/opt
  

  • 节点1

  /opt/n2n/sbin/edge -d edge1 -a 192.168.2.10 -s 255.255.255.0 -c dtstack -k  dtstack -l 192.168.1.40:65530 -E -r
  

  • 节点2

  /opt/n2n/sbin/edge -d edge1 -a 192.168.2.20 -s 255.255.255.0 -c dtstack -k  dtstack -l 192.168.1.40:65530 -E -r
  

• 测试multicast(非必要，主要是验证multicast无异常)

  ./mcasttest.pl -n rac1,rac2 -i edge1
  

 

ssh互信

• 分别在grid和oracle用户，每个节点上执行。

  • 创建ssh目录

  mkdir .ssh
  chmod 700 ~/.ssh
  cd ~/.ssh
  

  • 生成秘钥

  ssh-keygen -t rsa
  ssh-keygen -t dsa
  

• 在rac1上将密钥添加到授权文件中,grid和oracle用户分别执行

  • 本机的拷贝

  cat id_dsa.pub >> authorized_keys
  cat id_rsa.pub >> authorized_keys
  

  • 拷贝另一台主机的秘钥

  ssh rac2 cat ~/.ssh/id_dsa.pub >> authorized_keys
  ssh rac2 cat ~/.ssh/id_rsa.pub >> authorized_keys
  

  • 将本机的authorized_keys的文件拷贝到另一台主机上

  scp authorized_keys rac2:/home/grid/.ssh/
  scp authorized_keys rac2:/home/oracle/.ssh/
  

• 修改两个主机的授权密钥的权限

  chmod 600 ~/.ssh/authorized_keys
  

• 测试

  • 获取服务器公钥

  ssh-keyscan rac1  >> ~/.ssh/known_hosts
  ssh-keyscan rac2  >> ~/.ssh/known_hosts
  ssh-keyscan rac1private  >> ~/.ssh/known_hosts
  ssh-keyscan rac2private  >> ~/.ssh/known_hosts
  

  • 测试

  ssh rac1 date
  ssh rac2 date
  ssh rac1private date
  ssh rac2private date
  

 

安装前检查

 

$ORACLE_HOME/runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

• 错误处理
  • PRVE-0421
    修改/etc/fstab

  tmpfs  /dev/shm tmpfs  size=5000m   0 0
  

 

安装GI--grid用户下

• 解压安装压缩包

  cd /u01/app/12.2.0/grid/grid
  unzip /oraclesoftware/linuxx64_12201_grid_home.zip
  

• 安装cvudisk RPM包
  root 用户安装
  cvuqdisk存于oracle GI安装介质的cv/rpm目录下

  cd /u01/app/18.3.0/grid
  export CVUQDISK_GRP=asmadmin
  rpm -ivh cv/rpm/cvuqdisk-1.0.10-1.rpm
  
  scp cv/rpm/cvuqdisk-1.0.10-1.rpm 10.0.0.20:/root
  export CVUQDISK_GRP=asmadmin
  rpm -ivh /root/cvuqdisk-1.0.10-1.rpm
  

• grid用户执行$HOME_ORACLE/gridSetup.sh

 

安装DB--oracle用户下

• 解压安装压缩包

  cd /u01/app/oracle/product/18.3.0/db_1
  unzip /oraclesoftware/LINUX.X64_180000_db_home.zip
  

• oracle用户执行$HOME_ORACLE/runInstaller.sh

 

查看crs服务状态

 

crsctl status res -t