oracle service pdb 在各个node上切换 CDB开启、关闭下测试

The  -f option of srvctl would failover the running services to another instance, before shutting down the instance.
 

 Usage: srvctl stop instance -d {-n | -i }   [-o ] [-f]
 
     -d       Unique name for the database
     -n            Node name
     -i ""          Comma separated instance names
     -o         Options to shutdown command (e.g. NORMAL,TRANSACTIONAL, IMMEDIATE, or ABORT)
     -f                       This option fails the running services over to another instance. <<<<
     -h                       Print usage 

 
 Example:
 

 $ srvctl status database -d

Instance v11202_2 is running on node
Instance v11202_1 is running on node

$ srvctl status service -d

Service v11202_svc is running on nodes:

---------------不加-f 停不了

srvctl add service -d cdb -s pdb_srv -pdb pdb  -preferred cdb1 -available cdb2 这样建的话并不会切换 

[oracle@rac1 ~]$  srvctl stop instance -d cdb -n rac1
PRCD-1131 : Failed to stop database cdb and its services on nodes rac1
PRCR-1133 : Failed to stop database cdb and its running services
PRCR-1132 : Failed to stop resources using a filter
CRS-2974: unable to act on resource 'ora.cdb.db' on server 'rac1' because that would require stopping or relocating resource 'ora.cdb.pdb_srv.svc' but the appropriate force flag was not specified
[oracle@rac1 ~]$  srvctl stop instance -d cdb -n rac1 -f
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.

--------------

$ srvctl stop instance -d -i v11202_1 -f

<< after shutdown >>

$ srvctl status database -d

Instance v11202_2 is running on node
Instance v11202_1 is not running on node

$ srvctl status service -d

Service v11202_svc is running on nodes:

GOAL

NOTE: In the images, examples and document that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.

Sometimes user might want to start some of the pdbs in only one node for the following concern:

  1. User wants to reduce the load by not open the pdbs on both nodes.

  2. User wants some of the pdbs be activated on other nodes if the node they are running crashed.
 

SOLUTION

User can create service for a particular pdb,  let it run at preferred cdb instance, and let it to be able to failover to available cdb instance.

For example, there is one RAC system with two nodes: at node#1 ,  cdb db instance is orcl1,  at node#2, cdb db instance is orcl2. There is pluggable database pdb1 at the cdb.

User can create a service for this pdb, to let it run(open) at orcl1 instance, but not run (mount) at orcl2 instance:

$ srvctl add service -d orcl -s pdbtest -pdb pdb1 -preferred orcl1 -available orcl2

Here, the service name is pdbtest, and if this service started, then pdb1 will be opened only at orcl1 ,  if the orcl1 cdb instance crashed, then the service will failover to another node, in another word,  this pdb1 will be opened at cdb instance orcl2 at another node.

Start Service

The following is the command to start this service:

$ srvctl start service -db orcl -service pdbtest

When client application access the service via SCAN listener, then the connect request will be directed only to the preferred node.

The pdb can be opened by the starting of the service:

Before service starting:

At node#1:
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED

At node#2:
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED

After service starting:

At node#1:
$ srvctl start service -d orcl -s pdbtest

At node#1:
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO <-------- Changed
SQL>

At node#2:
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED <-------- Not changed
SQL>

Service Failover

There are two types of failover:  planned failover ,  unplanned failover.

Planned failover

If user want to make a planned failover (relocate), then a relocate operation can be done as the follows:

$ srvctl relocate service -db orcl -service pdbtest  -oldinst orcl1 -newinst orcl2

This will failover the pdb service pdbtest from orcl1 cdb instance on node#1 to orcl2 cdb instance on node#2.

The test result is as following:

At node#1:
SQL> show pdbs;
   CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
        2 PDB$SEED                       READ ONLY  NO
        3 PDB1                           READ WRITE NO
SQL>
At node#2:
SQL> show pdbs;
   CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
        2 PDB$SEED                       READ ONLY  NO
        3 PDB1                           MOUNTED
SQL>

At node#1:
$ srvctl relocate service -db orcl -service pdbtest  -oldinst orcl1 -newinst orcl2

At node#1:
SQL> show pdbs;
   CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
        2 PDB$SEED                        READ ONLY  NO
        3 PDB1                            READ WRITE NO <==== still alive
SQL>

At node#2:
SQL> show pdbs;
   CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
        2 PDB$SEED                       READ ONLY  NO
        3 PDB1                           READ WRITE NO <---- Changed
SQL>

The fact that the relocate operation does not close the pdb at the original node, is expected behavior.----带起了第二个 不关闭第一个，如果不做relocate 整个srv就关闭了。 必须要先relocate

If user wants the the pdb at node#1 to be closed, user need to stop the pdb on node#1 manually, eg: execute "alter pluggable database pdb1 close" on node#1.

After the relocate operation, we can see that the service shift to the other node(available node):

Service status before relocate:

$ crsctl stat res -t

ora.orcl.pdbtest.svc
     1        ONLINE  ONLINE       node1                     STABLE

 Service status after relocate:

$ crsctl stat res -t

ora.orcl.pdbtest.svc
     1        ONLINE  ONLINE       node2  <<<<               STABLE

Now, when client application access the service via SCAN listener, then the connect request will be directed only to the available instance (here it is orcl2). --orcl1反正也没有service呀

Unplanned Failover

If the node on which this pdb is running on crashes, then the service will failover to other node (available node), and the pdb will also be opened on this available node. We can simulate this by killing PMON process to see the effect:

At Node#1:
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO

At Node#2:
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED
SQL>

At Node#1:
$ crsctl stat res -t

ora.orcl.pdbtest.svc
      1        ONLINE  ONLINE       node1                    STABLE

At Node#1:
# ps -ef | grep pmon
oracle   1269214       1  0 00:43 ?        00:00:00 ora_pmon_orcl1
# kill -9 1269214  <---- killing the pmon process to make orcl1 instance crash

At Node#2:
$ crsctl stat res -t
ora.orcl.pdbtest.svc
      1        ONLINE  ONLINE       node2  <<<               STABLE <---- service shifted to node#2

At Node#2:
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO <-------- failover occurred
SQL>

After a while, let us check node#1 :

At Node#1:
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED <=== CDB restarted by cluster, but pdb not opened.
SQL>

Please note, that each of the following operation, is regarded as planned shutdown, so failover of the service / pdb will not occur.

SQL>alter pluggable database pdb close abort;

$ srvctl stop instance -d orcl -i orcl1 -f

But user can use "srvctl stop instance -d orcl -i orcl1 -force -failover" to make the service failover to the available node.

[oracle@rac1 ~]$ srvctl start service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   10181  9482  0 02:22 pts/0    00:00:00 grep --color=auto pmon
oracle   25763     1  0 02:06 ?        00:00:00 ora_pmon_cdb1
[oracle@rac1 ~]$ kill -9 25763
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl stop service -d cdb
[oracle@rac1 ~]$ s

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Feb 4 02:22:59 2024
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE YES
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0
[oracle@rac1 ~]$ srvctl start service -d cdb
PRCR-1120 : Resources are already running. 
CRS-5702: Resource 'ora.cdb.pdb_srv.svc' is already running on 'rac1'
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   10519     1  0 02:22 ?        00:00:00 ora_pmon_cdb1
oracle   12069  9482  0 02:24 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 10519

-----------node2 自动READ WRITE

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO
SQL> alter pluggable database pdb close abort;

Pluggable database altered.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO
SQL> 

------------------node2 上测试

SQL>  show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0
[oracle@rac2 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac2 ~]$ ps -ef|grep pmon
oracle    6356     1  0 02:16 ?        00:00:00 ora_pmon_cdb2
grid      7131     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM2
oracle   19028 14218  0 02:29 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac2 ~]$ kill -9 6356
[oracle@rac2 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac2 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac2 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac2 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac2 ~]$ 

------node1 自动起来了

[oracle@rac1 ~]$ s

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Feb 4 02:28:07 2024
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE YES
SQL> alter pluggable database pdb close abort;

Pluggable database altered.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            MOUNTED
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO
SQL> 

------------------PDB飞来飞去除非在mount到read write的时候报错，不需要重新连接-------------------

----------------- CDB开启时 srv切换另一个CDB上也是不需要重新连接--------------------------------------

----------------- CDB关闭时 等待CDB自动重启过程中 srv 重启后连接要重连了，可能和超时有关--- 

--------------------------------上面是PDB 的情况， 下面是CDB关闭的情况

-------------------------------CDB关闭的话SRV 不会切换！！！

---------------node 2 关闭不会切换
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle    8893     1  0 01:25 ?        00:00:00 ora_pmon_cdb1
oracle    9908  9482  0 01:26 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 8893
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1 自动启动后回到原来状态

---------------node 2 开启会切换

[oracle@rac1 ~]$ srvctl start instance -d cdb -i cdb2 
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   10077     1  0 01:26 ?        00:00:00 ora_pmon_cdb1
oracle   12371  9482  0 01:29 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 10077
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2----保持不变，需要重启或者relocate

[oracle@rac1 ~]$ srvctl stop service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl start service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1

-------------------重复

[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb2 -o abort
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   13057     1  0 01:30 ?        00:00:00 ora_pmon_cdb1
oracle   14961  9482  0 01:31 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 13057
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1

[oracle@rac1 ~]$ srvctl start instance -d cdb -i cdb2 
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   15298     1  0 01:31 ?        00:00:00 ora_pmon_cdb1
oracle   17093  9482  0 01:33 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 15298
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ 

----------------------服务不在，kill pmon后 这次node1 等很久才重启

Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   17272     1  0 01:33 ?        00:00:00 ora_pmon_cdb1
oracle   29639  9482  0 01:44 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 17272
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   29822  9482  0 01:44 pts/0    00:00:00 grep --color=auto pmon

。

。

。

[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   30113  9482  0 01:45 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ 

 Instance cdb1 is not running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl start service -d cdb -----服务不能再次启动来带动node1， 
PRCR-1120 : Resources are already running. 
CRS-5702: Resource 'ora.cdb.pdb_srv.svc' is already running on 'rac2'

 

[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1--------------一直都不再重启了，10分钟后才由GRID启动
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3

===========================================================
2024-02-03 12:44:40.822000 -05:00
License high water mark = 8
USER(prelim) (ospid: 29711): terminating the instance
2024-02-03 12:44:41.893000 -05:00
Instance terminated by USER(prelim), pid = 29711
2024-02-03 12:57:38.889000 -05:00
Starting ORACLE instance (normal) (OS id: 5012)
****************************************************
 Sys-V shared memory will be used for creating SGA
 ****************************************************
**********************************************************************
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
 Domain name: system.slice/ohasd.service
 Per process system memlock (soft) limit = UNLIMITED
 Expected per process system memlock (soft) limit to lock
 instance MAX SHARED GLOBAL AREA (SGA) into memory: 2274M
 Available system pagesizes:
  4K, 2048K

[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ 

------------------ instance 重启  -f后也不会自动切换服务的---------------

[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl stop service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl start service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ srvctl stop service -d cdb
[oracle@rac1 ~]$ srvctl start service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1

[oracle@rac2 ~]$ srvctl start service -d cdb-------------在2机上重启服务也是node1上启动服务oracle@rac2 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac2 ~]$ 

 
[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb1 -o abort
PRCD-1131 : Failed to stop database cdb and its services on nodes rac1
PRCR-1133 : Failed to stop database cdb and its running services
PRCR-1132 : Failed to stop resources using a filter
CRS-2974: unable to act on resource 'ora.cdb.db' on server 'rac1' because that would require stopping or relocating resource 'ora.cdb.pdb_srv.svc' but the appropriate force flag was not specified
[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb1 -o abort -f
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$  srvctl relocate service -db cdb -service pdb_srv  -oldinst cdb1 -newinst cdb2
[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb2 -o abort
PRCD-1131 : Failed to stop database cdb and its services on nodes rac2
PRCR-1133 : Failed to stop database cdb and its running services
PRCR-1132 : Failed to stop resources using a filter
CRS-2974: unable to act on resource 'ora.cdb.db' on server 'rac2' because that would require stopping or relocating resource 'ora.cdb.pdb_srv.svc' but the appropriate force flag was not specified
[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb2 -o abort -f
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running. 

-----------------kill -9 导致service 切换，即使node2时关闭的，但是其后会切回来---

--------------------------1        正常切换后再自动切回
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl stop service -d cdb
[oracle@rac1 ~]$ srvctl start service -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
---

[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   12272     1  0 02:24 ?        00:00:00 ora_pmon_cdb1
oracle   19120  9482  0 02:34 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 12272
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   19374  9482  0 02:34 pts/0    00:00:00 grep --color=auto pmon

[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3

[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2

---------------

[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   19642  9482  0 02:35 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl start database -d cdb
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is running on node rac2
Instance cdb3 is running on node rac3

---------------------2 node2 关闭下的切换 如果node1起早了也需要做 切换

[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb2 -o abort
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   19894     1  0 02:35 ?        00:00:00 ora_pmon_cdb1
oracle   27107  9482  0 02:39 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 19894
[oracle@rac1 ~]$ srvctl start instance -d cdb -i cdb2
[oracle@rac1 ~]$ srvctl status service -d cdb--------node1先起来了
Service pdb_srv is running on instance(s) cdb1

[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2

[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   27172     1  0 02:39 ?        00:00:00 ora_pmon_cdb1
oracle   31445  9482  0 02:46 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1-----切了两次

-------REPEAT--------------

[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb2 -o abort
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ ps -ef|grep pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   13057     1  0 01:30 ?        00:00:00 ora_pmon_cdb1
oracle   14961  9482  0 01:31 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1 ~]$ kill -9 13057
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is not running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.-------------两个node都没开，srv失败

[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status database -d cdb
Instance cdb1 is running on node rac1
Instance cdb2 is not running on node rac2
Instance cdb3 is running on node rac3
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1----node1 先起来了

 

----------------------3------------

[oracle@rac1 ~]$ srvctl stop instance -d cdb -i cdb2 -o abort
[oracle@rac1 ~]$  srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$  ps -ef|grep pmon
oracle     820  9482  0 02:49 pts/0    00:00:00 grep --color=auto pmon
grid      7182     1  0 Feb03 ?        00:00:00 asm_pmon_+ASM1
oracle   27172     1  0 02:39 ?        00:00:00 ora_pmon_cdb1
[oracle@rac1 ~]$ kill -9 27172
[oracle@rac1 ~]$  srvctl start instance -d cdb -i cdb2
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb2---node2先起来

--------------------4---------------重启后变回默认

[root@rac1 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'rac3' 
[root@rac1 ~]# crsctl start cluster -all
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is not running.
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1
[oracle@rac1 ~]$ srvctl status service -d cdb
Service pdb_srv is running on instance(s) cdb1