oracle crs中监听资源状态异常(ora.LISTENER.lsnr)

一：版本信息

操作系统版本：AIX 61009

数据库版本：11.2.0.3.11(RAC)

 

二：错误描述

1) crsctl stat res -t命令查看crs资源状态的时候，发现"ora.LISTENER.lsnr"资源状态异常：

ora.LISTENER.lsnr
               ONLINE  OFFLINE      ****1  ##实例1                                      
               ONLINE  OFFLINE      ****2  ##实例2                                     

2)检查监听状态正常

lsnrctl       

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 11-NOV-2014 10:53:26

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set password
Password: 
The command completed successfully
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
Start Date                06-NOV-2014 17:22:05
Uptime                    4 days 17 hr. 31 min. 48 sec
Trace Level               off
Security                  ON: Password
SNMP                      OFF
Listener Parameter File   /oracle/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /oracle/app/11.2.0/grid/log/diag/tnslsnr/bidb4/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.**)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.***)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "bidb" has 1 instance(s).
  Instance "bidb2", status READY, has 1 handler(s) for this service...
The command completed successfully

 

3)检查监听日志，发现目前监听还能正常接受连接

su - grid

cd $ORACLE_HOME/log/diag/tnslsnr/主机名/listener/alert

</msg>
<msg time='2014-11-11T10:46:02.602+08:00' org_id='oracle' comp_id='tnslsnr'
 type='UNKNOWN' level='16' <span style="color:#ff0000;">host_id='bidb4'</span>##数据库主机名
 <span style="color:#ff0000;">host_addr='***.***.**.**>##</span>数据库主机ip地址
 <txt>11-NOV-2014 10:46:02 * (CONNECT_DATA=(SERVICE_NAME=bidb)(CID=(PROGRAM=osh)(HOST=EGSNS-49-24.nd.cmgame.com)(USER=dsadm))(SERVER=dedicated)(INSTANCE_NAME=bidb2)) *
 (ADDRESS=(PROTOCOL=tcp)(<span style="color:#ff0000;">HOST=***.***.**.**)(</span>PORT=61378)) * establish * bidb * 0
 </txt>                  ##连接发起的IP地址

 

4)检查 CRS alert日志发现有如下报错：

/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f
or action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-11-07 15:03:21.139:
[/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f
or action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-11-07 15:03:21.267:
<span style="color:#ff0000;">[/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f
or action "check" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-11-07 15:08:01.278:</span>

##标红的部分报，监听检查报错，这个就跟我的问题联系上了。根据提示 查看oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/下的oraagent_grid日志(注意可能你要找的信息已经被刷出当前日志了，根据时间点查看相应日志。例如我需要的相关信息就在oraagent_grid.l02日志文件中)

5)查看oraagent_grid.l02日志

根据时间点，查找与"[ora.LISTENER.lsnr]"相关的日志，如下：

2014-11-07 15:03:21.143: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] <span style="color:#ff0000;"><span style="color:#ff0000;"><span style="color:#ff0000;">LsnrAgent::check </span>{ </span>##listener检查
</span>2014-11-07 15:03:21.143: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] <span style="color:#ff0000;">lsnrctl status LISTENER</span>

2014-11-07 15:03:21.143: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] getOracleHomeAttrib: oracle_home = /oracle/app/11.2.0/grid
2014-11-07 15:03:21.143: [    AGFW][2314] {2:2661:31941} Agent sending reply for: RESOURCE_START[ora.LISTENER.lsnr bidb4 1] ID 4098:431717
2014-11-07 15:03:21.144: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] getOracleHomeAttrib: oracle_home = /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::getCrsHome crsHome /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::execCmd 1 USR_ORA_ENV:ORACLE_BASE=/oracle/app/grid oracleHome:/oracle/app/11.2.0/grid
CrsHome:/oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::getCrsHome crsHome /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::getCrsHome crsHome /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Adding Environment Variables ORACLE_HOME=/oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Adding Environment Variables TNS_ADMIN=/oracle/app/11.2.0/grid/network/admin/
2014-11-07 15:03:21.146: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Adding Environment variable from USR_ORA_ENV ORACLE_BASE=/oracle/app/grid
2014-11-07 15:03:21.146: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2014-11-07 15:03:21.146: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] getOracleHomeAttrib: oracle_home = /oracle/app/11.2.0/grid
2014-11-07 15:03:21.265: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 07-NOV-
2014 15:03:21
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Copyright (c) 1991, 2011, Oracle.  All rights reserved.
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)<span style="color:#ff0000;"><span style="color:#ff0000;"><span style="color:#ff0000;">TNS-01169: The listener has not recognized the password </span></span>##检查报错(如果用过监听密码的人，因该对<span style="color:#ff0000;">TNS-01169</span>不会陌生，如果对监听器设置了密码，对监听进行操作时，需要输入密码)
</span>2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 1

2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] execCmd ret = 1
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 07-NOV-
2014 15:03:21
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Copyright (c) 1991, 2011, Oracle.  All rights reserved.
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)<span style="color:#ff0000;"><span style="color:#ff0000;"><span style="color:#ff0000;">TNS-01169: The listener has not recognized the password</span> </span>##检查报错
</span>2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] return val (clsagfwret) = 1
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] LsnrAgent::check }
2014-11-07 15:03:21.269: [    AGFW][2314] {2:2661:31941} <span style="color:#ff0000;"><span style="color:#ff0000;"><span style="color:#ff0000;">ora.LISTENER.lsnr bidb4 1 state changed from: STARTING to: OFFLINE</span> </span>##检查失败后将"<span style="color:#ff0000;">ora.LISTENER.lsnr</span>"资源状态改成<span style="color:#ff0000;">OFFLINE </span></span>

 

三：问题原因

  通过上面的分析，可以发现是因为为监听设置了密码，然后导致CRS在检查监听状态的时候报错(TNS-01169)，然后CRS将ora.LISTENER.lsnr资源状态修改为OFFLINE

然后紧接着又发现了第二个问题，同事在给listener设置密码，并重启后，listener中不会注册任何服务，执行'alter system register;'后检查还是没有任何服务被注册。如下：

lsnrctl status

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 12-NOV-2014 14:02:50

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
<span style="color:#ff0000;">TNS-01169: The listener has not recognized the password</span>

 

lsnrctl

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 12-NOV-2014 14:05:48

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set password
Password: 
The command completed successfully
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
Start Date                06-NOV-2014 17:27:51
Uptime                    5 days 20 hr. 38 min. 6 sec
Trace Level               off
Security                  ON: Password
SNMP                      OFF
Listener Parameter File   /oracle/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /oracle/app/grid/diag/tnslsnr/bidb3/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
The listener supports no services
The command completed successfully
LSNRCTL> 

 

 

四：解决方案

  查了一下MOS上有一篇NOTE "Deprecation of Listener Password in Oracle Database 11g Release 2 (Doc ID 1328725.1)"，上面说这样的描述

Oracle Net Services - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.
##适用范围，所有平台，11.2.0.1至11.2.0.3
In Oracle Database 11g Release 2 (11.2), the password feature is being deprecated. This does not cause a loss of security because authentication is enforced through local operating system authentication.
Please note, that this feature will be de-supported in the next major Oracle Database version (currently called 12g) and will then no longer be available.

##oracle 11.2版本，listener password特性已经不推荐适用。同时因为使用了Local OS Authentication功能，我们依然可以保证监听的安全性。在最新的oracle版本(12g)中listener password功能已经被取消

##查了很多文档，包括官方文档，都是类似的说法(在11.2以后已经不推荐使用 listener password了)。想想其实也是 新的版本中有Local OS Authentication功能(本地操作系统用户认证)，那么再给listener设置password纯属多余。如果你已经是安装用户，比如 grid用户，那么你就已经有权限去启停数据库了哪还用管什么监听

 

##原创作品，转载请注明出处！