solaris修改shminfo_shmmax参数无效解决

环境介绍：

双机
操作系统：solaris 10
数据库版本：oracle 11g R1 64bit
当前主机的主机名称和IP：fly-db2  192.168.20.21
当前备机的主机名称和IP：fly-db1  192.168.20.20

背景：

       主备机的物理内存均为94G，近期对备机的物理内存扩容一倍，并修改了数据库层面的相关内存参数，以及主备机的操作系统参数shmsys:shminfo_shmmax，注意是主备的shmsys:shminfo_shmmax参数均已修改，详情请查看下面链接的博文
http://fly1116.blog.51cto.com/8301004/1343788
     今晚对fly-db1主机（目前的备机)的物理内存扩容一倍后，重启操作系统，因主备机的shmsys:shminfo_shmmax参数均已经修改，重启操作系统后，理论上该参数设置生效，通过双机软件将数据库从fly-db2切换到fly-db1，切换失败，在双机软件中，查看到out of memory错误，在数据库alert日志，查看到如下报错：

WARNING: The system does not seem to be configured
optimally. Creating a segment of size 0x0000000220000000
failed. Please change the shm parameters so that
a segment can be created for this size. While this is
not a fatal issue, creating one segment may improve performance
如果手工启动，则报错如下：
SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument
bash-3.00$ oerr ora 27102
27102, 00000, "out of memory"
// *Cause: Out of memory
// *Action: Consult the trace file for details

     从以上报错分析，因为使用的是同一个数据库，操作系统层面只修改了shmsys:shminfo_shmmax参数,定位为当前备机fly-db1该参数设置未生效

   1、对比主备机的/etc/system文件，是一致的

bash-3.00#hostname
fly-db1
bash-3.00# diff /etc/system /home/oracle/system_flydb2_131220.bak

     2、百度之solaris out of memory,查看到以下blog，文章中涉及/etc/project文件，我们可以这么理解/etc/project文件，如果/etc/system和/etc/project中设置了相同的参数，则操作系统会忽略/etc/system的设置，而采用/etc/project的设置，/etc/project文件可以针对特定用户设置只对自己生效的参数
https://blogs.oracle.com/mandalika/entry/oracle_on_solaris_10_fixing   Oracle on Solaris 10 : Fixing the 'ORA-27102: out of memory' Error

    3、查看到当前备机fly-db1主机的/etc/project中设置了max-shm-memory的最大大小为60G，即最大的共享内存段为60G，如下所示：

bash-3.00# hostname
fly-db1
bash-3.00# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::project.max-shm-memory=(privileged,64424509440,deny)
group.staff:10::::
bash-3.00# bc
scale=5
64424509440/1024/1024/1024
60.00000

    4、查看当前主机（fly-db2）的/etc/project内容，未设置max-shm-memory的最大值，所以在该操作系统，生效的是/etc/system的设置

bash-3.00#hostname
fly-db2
bash-3.00# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::

   5、将当前备机fly-db1的/etc/project修改成和当前主机的/etc/project一样

bash-3.00#hostname
fly-db1
bash-3.00# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::

   6、重启操作系统

bash-3.00# sync
bash-3.00# sync
bash-3.00# sync
bash-3.00# shutdown -i6 -g0 -y

  7、双机切换，成功切换到备机fly-db1

 

以下内容来自：
https://blogs.oracle.com/mandalika/entry/oracle_on_solaris_10_fixing
Symptom:
As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an ORA-27102:
out of memory error message. The system had enough free memory to serve the needs of Oracle.
SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument
Diagnosis
$ oerr ORA 27102
27102, 00000, "out of memory"
// \*Cause: Out of memory
// \*Action: Consult the trace file for details
Not so helpful. Let's look the alert log for some clues.
% tail -2 alert.log
WARNING: EINVAL creating segment of size 0x000000028a006000
fix shm parameters in /etc/system or equivalent
Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example)
responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in /etc/system.
Prior to Solaris 10, shmsys:shminfo_shmmax parameter has to be set in /etc/system with maximum memory segment value that can
be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10
and later. On a Solaris 10 (or later) system, it can be verified as shown below:
% prtconf | grep Mem
Memory size: 32760 Megabytes
% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)
% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
privileged      7.84GB      -   deny                                 -
system          16.0EB    max   deny                                 -
Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to
create a memory segment (10G) larger than 8G. Hence the failure.So, the solution is to configure the system with a value large
enough for the shared segment being created, so Oracle succeeds in starting up the database instance.
On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system
to pick up the new value.
set shminfo_shmmax = 0x000000028a006000
However shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter
in /etc/system even though it works as expected.
On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is
how we do it on Solaris 10 and later:
% prctl -n project.max-shm-memory -r -v 10G -i project 3
% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
privileged      10.0GB      -   deny                                 -
system          16.0EB    max   deny                                 -
Note that changes made with the prctl command on a running system are temporary, and will be lost when the system is rebooted. To make
the changes permanent, create a project with projadd command and associate it with the user account as shown below:
% projadd -p 3  -c 'eBS benchmark' -U oracle -G dba  -K 'project.max-shm-memory=(privileged,10G,deny)' OASB
% usermod -K project=OASB oracle
Finally make sure the project is created with projects -l or cat /etc/project commands.
% projects -l
...
...
OASB
projid : 3
comment: "eBS benchmark"
users  : oracle
groups : dba
attribs: project.max-shm-memory=(privileged,10737418240,deny)
% cat /etc/project
...
...
OASB:3:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)
With these changes, Oracle would start the database up normally.
SQL> startup
ORACLE instance started.
Total System Global Area 1.0905E+10 bytes
Fixed Size                  1316080 bytes
Variable Size            4429966096 bytes
Database Buffers         6442450944 bytes
Redo Buffers               31457280 bytes
Database mounted.
Database opened.