目录
- 11g RAC概述
- SAN概述
- 集群架构
- 安装及调试工具
- storage服务器安装与配置
一。11g RAC概述：
11gR2集群又称为GI（Grid Infrastructure），保留了10g的核心功能，在结构上更标准，更精细。所有组件以资源形式存在，统一由代理进程管理。按照功能，可以分为集群初始化组件、集群存储管理组件、应用程序组件。
集群初始化组件主要包括：ohasd守护进程、bootstrap本地节点、ocssd进程。
集群存储管理组件主要负责管理共享存储ASM磁盘组。
应用程序组件负责管理集群中应用资源，包括crsd守护进程。按照集群功能特点划分，RAC属于一种hot failover 类型的集群。它的每个节点都处于活动状态，都能分担应用负载，某一节点出现问题，负载由其他节点承担，对用户完全透明。
按照数据共享特点划分，RAC属于一种share-everything集群，有自己的共享磁盘架构，有很好的可测量性和扩展性。但是存在数据并发读写的控制问题，需要引入额外的锁机制保证读、写的串行化。
如下是RAC的结构示意图
image.png
二。SAN概述：
- FC SAN有着高效、安全的特点，IP SAN有着较低的成本和好的网络适应性的特点。在数据访问要求不是特别高的场合和海量数据存储场合，我们一般采用IP SAN实现数据的远程访问。
- Linux I/O早在2011年就成为了Linux Kernel的一部分。 它前端支持FCoE, Fibre Channel, IEEE 1394, iSCSI, iSER, SRP and USB等多种互联。这使得LIO成为许多存储供应商的重要选择。我们这里使用LIO支持的iSCSI为数据库提供存储支持。
- 我们使用LIO的配置工具targetcli来方便的使用iSCSI。
三。集群架构
集群设计采用2台服务器，一台IP SAN服务器。如下图：
arch.jpg
四。安装及调试工具
1.ssh客户端：PuTTY0.70。下载
2.X window 客户端：Xming 6.9及以上版本。下载
3.操作系统：Oracle Linux 7.0版本。下载
4.虚拟化平台：VMware Workstation :12及以上。下载
五。storage服务器安装及配置
1.Oracle Linux 基本安装（省略）
为对外提供存储服务，storage虚拟机上应该添加额外的硬盘。这里我们额外添加了两块20G硬盘。
fdisk -l命令可以看到如下结果：
[root@storage ~]# fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2610 20860402+ 8e Linux LVM
Disk /dev/sdb: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 2610 20964793+ 8e Linux LVM
Disk /dev/sdc: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 2610 20964793+ 8e Linux LVM
2.关闭防火墙
为了简化安装，我们关闭系统防火墙。
如果是使用systemctl的系统了。就用systemctl关闭防火墙,并且disable防火墙:
[root@storage ~]# systemctl stop firewalld
[root@storage ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
Jun 06 13:59:33 storage systemd[1]: Starting firewalld - dynamic firewall .....
Jun 06 13:59:34 storage systemd[1]: Started firewalld - dynamic firewall d...n.
Jun 06 13:59:47 storage systemd[1]: Stopping firewalld - dynamic firewall .....
Jun 06 13:59:48 storage systemd[1]: Stopped firewalld - dynamic firewall d...n.
Hint: Some lines were ellipsized, use -l to show in full.
[root@storage ~]# systemctl disable firewalld
3.安装ISCSI管理工具
我们就安装和使用LIO targetcli。yum本地源配置如下：
[root@storage ~]# cat /etc/yum.repos.d/local.repo
[local]
name=local
baseurl=file:///mnt/cdrom
gpgcheck=0
enabled=1
安装LIO targetcli。
[root@192 Packages]# yum install targetcli
检查并启动targetcli，一般随机发布还有一个targetd包，这个是实现基于LIO的远程管理程序包，我们配置完成后，可以试试这个包的使用。
[root@storage ~]# systemctl status target
● target.service - Restore LIO kernel target configuration
Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2019-06-06 17:10:46 CST; 4s ago
Process: 22525 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS)
Main PID: 22464 (code=exited, status=0/SUCCESS)
Jun 06 17:09:00 storage systemd[1]: Starting Restore LIO kernel target configuration...
Jun 06 17:09:00 storage systemd[1]: Started Restore LIO kernel target configuration.
Jun 06 17:10:46 storage systemd[1]: Stopping Restore LIO kernel target configuration...
Jun 06 17:10:46 storage systemd[1]: Stopped Restore LIO kernel target configuration.
[root@storage ~]# systemctl start target
[root@storage ~]# systemctl status target
● target.service - Restore LIO kernel target configuration
Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
Active: active (exited) since Thu 2019-06-06 17:11:06 CST; 2s ago
Process: 22525 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS)
Process: 22550 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS)
Main PID: 22550 (code=exited, status=0/SUCCESS)
Jun 06 17:11:06 storage systemd[1]: Starting Restore LIO kernel target configuration...
Jun 06 17:11:06 storage systemd[1]: Started Restore LIO kernel target configuration.
4.准备创建pv，vg，lv
由于我们是在建立系统后添加的硬盘，所以需要扫描总线，发现新硬盘。
[root@storage ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 4.3G 0 rom /mnt/cdrom
sda 8:0 0 40G 0 disk
├─sda2 8:2 0 39G 0 part
│ ├─ol-swap 252:1 0 2G 0 lvm [SWAP]
│ └─ol-root 252:0 0 37G 0 lvm /
└─sda1 8:1 0 1G 0 part /boot
[root@storage ~]# echo "- - -"> /sys/class/scsi_host/host2/scan
[root@storage ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 40G 0 disk
sr0 11:0 1 4.3G 0 rom /mnt/cdrom
sda 8:0 0 40G 0 disk
├─sda2 8:2 0 39G 0 part
│ ├─ol-swap 252:1 0 2G 0 lvm [SWAP]
│ └─ol-root 252:0 0 37G 0 lvm /
└─sda1 8:1 0 1G 0 part /boot
[root@storage ~]#
建立磁盘分区。
[root@storage ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x5b5b6bcc.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1): 1
First sector (2048-83886079, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-83886079, default 83886079):
Using default value 83886079
Partition 1 of type Linux and of size 40 GiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
创建pv，vg
[root@storage ~]# pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created.
[root@storage ~]# vgcreate storagevg /dev/sdb1
Volume group "storagevg" successfully created
创建lv：第一个是lv_data,主要用于数据库文件存储，大小20G，第二个是lv_fra，用于Fast Recovery Area，大小是15G，第三个lv_crs，用于仲裁文件和OCR(Oracle Cluster Register)，大小2G。
[root@storage ~]# lvcreate -L 20G storagevg -n lv_data
Logical volume "lv_data" created.
[root@storage ~]# lvcreate -L 15G storagevg -n lv_fra
Logical volume "lv_fra" created.
[root@storage ~]# lvcreate -L 2G storagevg -n lv_crs
Logical volume "lv_crs" created.
[root@storage ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 40G 0 disk
└─sdb1 8:17 0 40G 0 part
├─storagevg-lv_crs 252:4 0 2G 0 lvm
├─storagevg-lv_data 252:2 0 20G 0 lvm
└─storagevg-lv_fra 252:3 0 15G 0 lvm
sr0 11:0 1 4.3G 0 rom /mnt/cdrom
sda 8:0 0 40G 0 disk
├─sda2 8:2 0 39G 0 part
│ ├─ol-swap 252:1 0 2G 0 lvm [SWAP]
│ └─ol-root 252:0 0 37G 0 lvm /
└─sda1 8:1 0 1G 0 part /boot
[root@storage ~]#
下面使用targetcli创建block设备：
[root@storage ~]# targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> cd /backstores/block
/backstores/block> create data /dev/storagevg/lv_data
Created block storage object data using /dev/storagevg/lv_data.
/backstores/block> create fra /dev/storagevg/lv_fra
Created block storage object fra using /dev/storagevg/lv_fra.
/backstores/block> create crs /dev/storagevg/lv_crs
Created block storage object crs using /dev/storagevg/lv_crs.
/backstores/block>
创建iqn和lun：
/backstores/block> cd /iscsi
/iscsi> create
Created target iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> cd /iscsi/iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de/tpg1/luns
/iscsi/iqn.20...8de/tpg1/luns> create /backstores/block/
/backstores/block/crs /backstores/block/data /backstores/block/fra
/iscsi/iqn.20...8de/tpg1/luns> create /backstores/block/data
Created LUN 0.
/iscsi/iqn.20...8de/tpg1/luns> create /backstores/block/fra
Created LUN 1.
/iscsi/iqn.20...8de/tpg1/luns> create /backstores/block/crs
Created LUN 2.
/iscsi/iqn.20...8de/tpg1/luns>
创建acl和portal：
/iscsi/iqn.20...8de/tpg1/luns> cd /iscsi/iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de/
/iscsi/iqn.20....b0a7e05a08de> cd tpg1/acls
/iscsi/iqn.20...8de/tpg1/acls> create iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de:client
Created Node ACL for iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de:client
Created mapped LUN 2.
Created mapped LUN 1.
Created mapped LUN 0.
/iscsi/iqn.20...8de/tpg1/acls> cd ../portals/
/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 1]
o- 0.0.0.0:3260 ............................................................................................................. [OK]
/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20.../tpg1/portals> create 192.168.237.128
Using default IP port 3260
Created network portal 192.168.237.128:3260.
/iscsi/iqn.20.../tpg1/portals>
/iscsi/iqn.20.../tpg1/portals> cd ..
/iscsi/iqn.20...e05a08de/tpg1> set attribute generate_node_acls=1
Parameter generate_node_acls is now '1'.
/iscsi/iqn.20...e05a08de/tpg1> set attribute authentication=0
Parameter authentication is now '0'.
/iscsi/iqn.20...e05a08de/tpg1> set attribute demo_mode_write_protect=0
Parameter demo_mode_write_protect is now '0'.
启动时启动ISCSID服务。
[root@192 ~]# systemctl enable iscsid
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.
支持IPSAN配置完毕。下一节我们开始集群节点的配置。