TNS-12518 错误处理

一：版本信息

操作系统版本 AIX 6109

数据库版本  11.2.0.3(PSU11)

二：错误描述

    最近发现listener日志文件log.xml中频繁的出现如下告警信息：

 TNS-12518: TNS:listener could not hand off client connection
 TNS-12536: TNS:operation would block
  TNS-12560: TNS:protocol adapter error
   TNS-00506: Operation would block
    IBM/AIX RISC System/6000 Error: 11: Resource temporarily unavailable
 

##应用侧也反应，最近应用连接数据库有时会有问题

  我们来看一下MOS上对该问题原因的解释：

ORA-12518/TNS-12518 indicates a problem while listener hands off the client connection to the server process or dispatcher process.TNS-12518 is logged in the listener log. The client might receive ORA-12518 or some other disconnection errors like ORA-12537. Once TNS-12518 is noted in the listener log, follow the below steps to approach and resolve this error. 
##ORA-12518/TNS-12518 说明在监听程序把客户端连接转移到服务器进程，或者dispatcher 进程时出现了问题。TNS-12518错误被记录在listener日志里。客户端可能会收到ORA-12518或者其他一些断开连接的错误提示，如ORA-12537。
Let us have a small discussion about how actually database connections are made:
##让我们简单的看一下通过监听连接数据库的过程
In Dedicated mode, database client contacts listener and supplies the SERVICE NAME of the database. Then listener spawns a dedicated server process and hands off the client connection to this dedicated server process. TNS-12518 indicates a problem while handing off the client connection to the server process.
##在专有连接模式下，客户端连接监听并提供需要连接的SERVICE NAME，然后监听会产生一个专有服务器进程，并把客户端的连接转移给该服务器进程。TNS-12518说明在监听把客户端连接转移到服务器进程的时候出现了问题
In Shared Server mode, database client contacts listener and supplies the SERVICE NAME of the database. Then listener hands off the client connection to one of the dispatcher configured for that service. TNS-12518 indicates a problem while handing off the client connection to the dispatcher server process.
##在共享连接模式下，客户端连接监听并提供需要连接的SERVICE NAME，然后监听把客户端的连接转移给dispatcher。TNS-12518说明在监听把客户端连接转移到dispatcher进程的时候出现了问题
Though this error is logged in the listener log, the listener is just the messenger, ORA-12518/TNS-12518 is mostly related to RDBMS and OS resources.

##ORA-12518/TNS-12518错误大多数都跟数据库和操作系统资源有关

三：错误原因及解决方案

  下面是MOS上诊断该问题的详细步骤

1)Step 1. Is listener version compatible to the database  >>检查监听版本与数据库是否兼容

Check if you are using compatible listener version for your database version. >>检查监听版本同数据库版本是否兼容
If the database is 8i then use 8i or 9i listeners only. 10g listeners are not compatible to work with 8i databases. >>8i数据库只能使用8i或者9i的监听，同10g的监听不兼容
For 9i databases, 9i or 10g listeners can be used. >>9i数据库可以使用9i和10g的监听
For 10g databases, only 10g listeners can be used. >>10g的数据库只能使用10g的监听
The general rule is that use the higher version of the listener when there is a version mismatch between database and the listener. >>

2)Step 2. Gather more information from listener log  >>收集listener log中相关错误信息

The first place you would look for the TNS-12518 error is the listener log. Usually the listener log would be located under $ORACLE_HOME/network/log directory. You can use 'lsnrctl status' command output to look for the location of the listener log file.>>TNS-12518首先会记录在listener log中。通常listener log位于$ORACLE_HOME/network/log目录下，我们可以通过“lsnrctl status”查看监听日志文件路径

$lsnrctl status 
- - - 
- - - 
Listener Parameter File /ora10g/home_ora10g/network/admin/listener.ora 
Listener Log File /ora10g/home_ora10g/network/log/listener.log  >>监听日志路径
- - - 
- - -

listener log gives the complete error stack and the database service name to which the client tried to connect to.>>监听日志中给出了完全的错误信息包括SERVICE NAME信息

19-SEP-2007 13:55:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=test.oracle.com)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.3)(PORT=36030)) * establish * test.oracle.com * 12518 
TNS-12518: TNS:listener could not hand off client connection 
 TNS-12547: TNS:lost contact 
  TNS-12560: TNS:protocol adapter error 
   TNS-00517: Lost contact 
    Linux Error: 32: Broken pipe

In the above example, listener log shows the complete error stack, the bottom error being 32 is the OS error. It also shows that the jdbc client from IP 10.10.10.3 has tried to connect to the database service 'test.oracle.com' and failed with the error 12518.>>在上面的例子中，listener log给出了完整的错误信息，在最下面我们看到的是操作系统层面的错误(Linux Error: 32: Broken pipe)。这段错误显示了从10.10.10.3过来的jdbc 客户端尝试连接test.oracle.com服务，最后因为12518错误失败了。

Look for the lowest error in the stack. That is the error we have to concentrate on and try to resolve it. In the above example, the lower error is 'Linux Error: 32: Broken pipe'.>>对于错误信息我们应该关注最下面的一个错误，并尝试解决它。在上面的例子中，我们应该关注的是“Linux Error: 32: Broken pipe”这个错误

3)Step 3. Is service handlers in blocked state  >>监听是否处于阻塞状态

Check if the handlers are in blocked state. Check the output of the 'lsnrctl services'. Examine the status information under the database service name. From the listener log you would know which database service was affected by the 12518 error, now with the output of the 'lsnrctl services' under that service name gather more information.>>通过“lsnrctl services”检查相应service name的监听是否处于阻塞状态。通过监听日志你可以看到哪个service name的监听收到TNS-12518错误的影响

Service "test.oracle.com" has 1 instance(s). 
  Instance "db10g", status READY, has 2 handler(s) for this service... 
    Handler(s): 
      "DEDICATED" established:9 refused:0 state:ready 
         LOCAL SERVER 
      "D000" established:10 refused:0 current:0 max:972 state:ready 
         DISPATCHER  
         (ADDRESS=(PROTOCOL=tcp)(HOST=oid.mohan.com)(PORT=33487))

The highlighted state should be in 'ready' state for the connection to be successful. If the state is in 'blocked' then the connection are not possible. >>监听的状态应该是“read”，如果监听的状态是“blocked”那么连接将无法建立

The state of a handler could be in blocked state in the following scenario: >>在以下情况下监听的状态会处于blocked状态

i. The database parameter processes reached its value. >>数据库的processe资源达到参数限制
ii. The database is in the process of startup or shutting down. >>数据库在启动和关闭的过程中

In shared server mode, the number of dispatchers should be set according to the load that you expect.  'lsnrctl services'  output shows the maximum number(max:997) of connections that the dispatcher would accept and the number connections refused (refused:0) by this dispatcher. If any connections refused by the dispatcher, then consider increasing the number of dispatchers. >>在共享连接模式下(现在已经很少用共享模式)应该根据负载情况设置合适的dispatcher数量。“lsnrctl services”命令会显示dispatcher能够接受的最大连接数以及被拒绝的连接数。如果发现有连接被拒绝，那么要考虑增加dispatcher数量

If you are using PFILE edit init.ora and increase the dispatchers parameter. If you are using SPFILE you can dynamically increase the dispatchers parameter by the  'alter system set' command. >>如果你使用的参数文件是pfile可以直接修改pfile文件加大dispatchers参数值(重启生效)。如果你使用的是spfile参数文件，那么可以通过alter system set命令来动态的修改该参数的值。

4)Step 4. Is local BEQ connection successful >>检查本地BEQ协议连接是否正常

Check if local BEQ connection to the database works fine. It also verifies if the database is up and in good condition to accept the connection. If the database is down or in a hung state then a connection request to the database by the listener will not be possible. >>检查在本地使用BEQ协议能否正常连接数据库，同时还能验证数据库是否打开以及能否正常接收连接。如果数据库是关闭的或者hang住，通过监听连接数据库也会失败
Connect to the database server via telnet or ssh and check if a local bequeath SQL*Plus connection works. In other words, issue:>>通过如下命令检查能否使用BEQ协议正常连接数据库
sqlplus username/password [Enter]
This connection bypasses the listener and directly connects to the database via the BEQ (bequeath) protocol. If this fails, then the TNS-12518 listener error is simply a result of the database issue. >>上面的例子中连接会绕过监听直接通过BEQ协议连接数据库，如果连接失败，那么TNS-12518错误基本上就是因为数据库的问题了
One such error is:
ORA-12560: TNS:protocol adapter error

A possible cause for this error on Microsoft Windows servers, is that the Windows Database Service has not yet been created (common when creating a "standby" instance).
Resolution for this would be to create the Windows Service first by using the "oradim" command (see the Database Admin guide for details on oradim and service creation).

5)Step 5. Has number of processes reached its limit?  >>检查processes和sessions参数是否达到最大值限制
If local BEQ is successful, check the below query >>如果BEQ协议连接检查是正常的，那么检查processes和sessions参数是否达到最大值限制

SQL> select * from v$resource_limit; 

RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_AL LIMIT_VALU 
-------------------- ------------------- --------------- ---------- ---------- 
processes 249 250 250 250 
sessions 54 82 280 280 
- - - - - -
- - - - - -

Verify if the processes or sessions reached its limit value. If these database parameters reached its limit value, then consider increasing it accordingly. >>如果这些数据库参数达到了最大值限制，那么考虑加大这些参数的值

In the above example, the processes parameter has been set to 250. It's MAX_UTILIZATION has reached the limit value of 250, so the processes parameter should be increased further to accomodate the number of incoming connections. >>在上面的例子中processes参数已经达到了250最大值限制，所以这个参数应该被增大。

Edit the init.ora and set the processes parameter to a higher value. By default, if you just increase the processes it is enough, the sessions value would automatically be increased.>>默认的如果你增加了processes参数，sessions参数也会自动增加

Check the alert.log for a corresponding error such as "ksvcreate" process failed, etc.  Check timestamp against listener.log timestamps for causal relationship.>>检查alertlog中是否有"ksvcreate" process failed类似错误，并跟监听日志中错误时间进行比对。

Step 6. Are OS kernel parameters configured for optimum?  >>检查操作系统层面参数设置是否合理

Database is operated by a single user, normally it would be 'oracle' user. At the Operating System level, there is a limit for the number of process spawned by a user. And also there is a limit for the total number of process running on the entire OS.  >>数据库是被一个单独用户操作，通常这个用户是oracle。在操作系统层面上，会对某个用户可以产生的进程以及整个系统上可以产生的进程数据是有限制的。

The Oracle Database and the newly spawned processes would be owned by the 'oracle' user. And so make sure that these values are set accordingly.On Unix these values are configured through the configurable OS kernel parameters and is specific to Operating Systems. You will have check your corresponding OS documentation for your OS. >>根据相应的系统的文档检查你的数据库主机操作系统相应参数设置是否正常

For example for HP-UX the configurable kernel parameters are, >>例如在HP-UX中你需要关注如下两个参数的配置

maxuprc  Maximum number of processes for each non-root user >>指定非root用户可以产生的最大进程数
nproc      Maximum number of processes on the system            >>指定整个系统中可以产生的最大进程数

##我的系统中就是因为maxuprc 值设置过小导致TNS-12518错误，修改为官方文档建议值后，TNS-12518错误消失

lsattr -El sys0|grepmaxuproc  >>查看maxuprc参数值(AIX操作系统)

chdev -l sys0 -amaxuproc='16384' >>设置该参数为官方文档建议值(该参数修改即生效，无需重启操作系统)

Step 7.  Does alert log have any errors?  >>检查alert log中是否有错误信息

Look in alert log and look for any errors related to memory or process during the time the error TNS-12518 occurred in the listener log. If the alert log has any memory related errors, there there is a potential memory resource issue at the OS level. >>在TNS-12518发生时，检查alert log日志相应时间点是否有关于内存以及进程的错误。如果alert log中有一些和内存相关的错误，那么可能操作系统层面的内存资源出现了问题。

OS memory issues can be addressed by the below: 
i.   Make sure that the OS has been configured with the enough Swap memory. In case of Windows it is called as Virtual Memory. >>确保操作系统有足够的swap空间
ii.  Reduce the size of SGA, thus the newly spawned server process will have some more system memory available. >>减小SGA，这样操作系统可以使用的内存就增加了
iii. Reduce the PGA size, so that the newly spawned server process would occupy less memory. >>减小PGA，这样服务器进程占用的内存就会更少
iv. If you are in DEDICATED mode, try switching to MTS mode. >>如果你使用的专有连接模式，尝试转变为共享连接模式

To address memory issue for 32-bit Windows: Refer  Note 371983.1 >>有关32位的Window操作系统内存问题，请参考Note 371983.1

If there are any memory or process related error in the alert log during the time the TNS-12518 is logged in the listener log, then those errors in the alert log should be focused on and should be solved at first. Because, the errors in the alert log is the base error for the TNS-12518 in the listener log. 
However, the errors in the alert logs are not being discussed in this article, they are out of the scope of this article.

>>注意，如果TNS-12518错误发生的时候，在alert log相应时间点发现内存相关的错误，那么你需要首先关注并解决这些错误。因为alert日志里的错误是引起TNS-12518的根本原因。

8)Step 8. If using a statically defined SID_DESC in the listener.ora file for your sid, ensure that it is configured properly.    >>检查listener.ora中配置正确

A common mistake is to include a (PROGRAM=EXTPROC) parameter:

  (SID_LIST= 
    (SID_DESC = 
      (GLOBAL_DBNAME = ORCL.oracle.com) 
      (SID_NAME = ORCL) 
      (PROGRAM=extproc) 
      (ORACLE_HOME = D:\oracle\product\10.2.0\db_1)))

This misconfiguration occurs when the PLSExtproc SID_DESC is copied, pasted and edited in the listener.ora file.  The inclusion of the PROGRAM line will cause an ORA-12518 to be returned to the client.  Here's the corrected SID_DESC for our example SID:  >>当你copy了上面的内容，进行相关修改后放入自己的listener.ora文件中使用，你很可能忽略(PROGRAM=extproc)，如果在监听配置中包含这一行那么客户端可能会收到ORA-12518错误(当在监听中使用SID_NAME时)

  (SID_LIST= 
     (SID_DESC = 
     (GLOBAL_DBNAME = ORCL.oracle.com) 
     (SID_NAME = ORCL) 
     (ORACLE_HOME = D:\oracle\product\10.2.0\db_1)))

四：下面列出了一些引起TNS-12518错误的常见原因

Below shows example error stack that can be found in the listener log. The last line in the error stack shows the actual operating system name. Depending on the OS, only the operating system name would be different. For example, if you encounter error 32: Broken pipe, according to the OS, the last line in the error stack would be different only by the OS name, as shown below.

>>下面列出了一些在listener log中常见的TNS-12518相关的错误。在报错的最后部分显示了操作系统名称。对于不同的操作系统，最后一行报错信息只是操作系统名会不同。例如，如果你遇到了 error 32: Broken pipe，根据不同的操作系统，你看到的错误信息只是在操作系统名部分不同，如下所示：
Solaris Error: 32: Broken pipe
HPUX Error: 32: Broken pipe
Linux Error: 32: Broken pipe

_______________________________________________________________________________________________________________________________________

1) Error: 32: Broken pipe

Error stack in listener log:
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
IBM/AIX RISC System/6000 Error: 32: Broken pipe
Cause:
The error 32 indicates the communication has been broken while the listener is trying to hand off the client connection to the server process or dispatcher process. >>error 32错误说明listener在把客户端的连接转移给服务器进程或者dispatcher进程时发生错误。

Action:
1. One of reason would be processes parameter being low, and can be verified by the v$resource_limit view. >>一个可能的原因是processes参数值太小，通过v$resource_limit确认processes是否达到限制的最大值
2. In Shared Server mode, check the 'lsnrctl services' output and see if the dispatcher has refused any connections, if so, then consider increasing the number of dispatchers. >>如果是共享连接模式下，使用'lsnrctl services' 命令检查是否有被dispatcher拒绝的连接，如果有，那么考虑增加dispatcher
3. Check the alert log for any possible errors. >>检查alert日志是否有相关报错
4. Memory resource is also another cause for this issue. Check the swap, memory usage of the OS.>>内存资源使用问题也可能导致该错误，检查swap和操作系统内存使用情况
5. If RAC/SCAN or listener is running in separate home, check the following note:>>如果监听同 oracle database属于不同用户(比如RAC)，请参考下面的NOTE
Note: 1069517.1 ORA-12537 or TNS-12518 if Listener (including SCAN Listener) and Database are Owned by Different OS User
______________________________________________________________________________________________________________________________________

2) Error: 11: Resource temporarily unavailable

Error stack in listener log:
TNS-12518: TNS:listener could not hand off client connection
TNS-12549: TNS:operating system resource quota exceeded
TNS-12560: TNS:protocol adapter error
TNS-00519: Operating system resource quota exceeded
IBM/AIX RISC System/6000 Error: 11: Resource temporarily unavailable

Cause:
As the error indicates operating system resource has exceeded. >>这个错误说明超出了操作系统的资源限制

Action:
1. Increase the appropriate OS kernel parameters for 'maximum number of processes allowed per user'. >>增加主机侧对每个用户能够生成进程数的限制
For example for HP-UX the parameters are maxuprc and nproc. >>例如HP-UX上相应参数是maxuprc和nproc
_____________________________________________________________________________________________________________________________________

3) Error: 12: Not enough space

Error stack in listener log: 

TNS-12518: TNS:listener could not hand off client connection
TNS-12549: TNS:operating system resource quota exceeded
TNS-12560: TNS:protocol adapter error
TNS-00519: Operating system resource quota exceeded
IBM/AIX RISC System/6000 Error: 12: Not enough space

Cause:
This is a memory related issue. The error indicates that there is not enough memory available to spawn and hand off the client connections. >>这个是内存相关的错误
Typical problems are:
- Out of system memory / swap >>系统内存或者swap不足
- Out of process slots in the process table >>
- Streams resources depleted
- Out of File Handles
- sga memory usage
Action:
1. Check in the alert log for any possible memory related error. >>检查alert log中是否有跟内存相关的错误
2. Increase swap/Virtual memory if possible the available memory. >>增加swap和内存
3. SGA and PGA can be reduced to address the memory consumption. >>通过减少pga和sga解决内存不足
4. MTS mode can be used to reduce the amount of process and memory consumption. >>可以通过使用MTS连接模式，减少服务器进程和内存消耗
_____________________________________________________________________________________________________________________________________

4) Error: Connection Pooling limit reached

Error stack in listener log:
TNS-12518 TNS:listener could not hand off client connection
TNS-12564 TNS:connection refused
TNS-12602 TNS: Connection Pooling limit reached 

Action:
1. Try increasing initial number of dispatchers. >>增加dispatcher数量
_____________________________________________________________________________________________________________________________________

5) Error: 2: No such file or directory

Error stack in listener log:
TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:proto adapter error
TNS-00530: Proto adapter error
32-bit Windows Error: 2: No such file or directory

Error Description:
ERROR_FILE_NOT_FOUND
2
The system cannot find the file specified.
Cause:
This indicates the database service is not actually available >>该错误说明数据库服务不可用
Action:
1.Verify if the intended database really up and accepting local BEQ connections.>>首先确认数据库状态正常，然后检查本地BEQ连接能否成功

_______________________________________________________________________________________________________________________________________ 

6) Error: 24: Too many open files

Error stack in listener log:
TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
Linux Error: 24: Too many open files

or

TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
Solaris Error: 24: Too many open files

Error Description:
Out of file descriptors

Cause:
Can be caused by the following unpublished defect:>>由下面的bug引起

BUG 13078786 - LISTENER GOES DOWN SUDDENLY W/ LINUX ERROR: 32: BROKEN PIPE 

Action:
See Note 1527483.1 11.2 : ORA-12518 Listener Hangs and Reports "Too Many Open Files"
Use prescribed workaround OR apply one-off patch to your environment if available.

##注意对于windows 操作系统还有如下常见错误

This section briefly describes about the errors that are encountered on Windows Operating System. TNS-12518 most commonly occurs on 32-bit OS due to its memory constraint, however TNS-12518 can occur on 64-bit OS as well. See Note 873752.1 for more information on Windows memory addressing and the 3GB switch.

>>下面这部分描述了windows操作系统上TNS-12518相关错误。在32windows操作系统上经常因为内存使用的限制而引发TNS-12518错误，然后64位windows上也一样会出现TNS-12518错误。

_______________________________________________________________________________________________________________________________________

Error: 233: Unknown error

Error stack in listener log:
TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
32-bit Windows Error: 233: Unknown error


Error Description:
ERROR_PIPE_NOT_CONNECTED
233
No process is on the other end of the pipe.

Cause:
The communication has been broken while the listener is trying to hand off the client connection to the server process or dispatcher process.


Action:
Refer Note 371983.1

_______________________________________________________________________________________________________________________________________

Error: 54: Unknown error

Error stack in listener log:
TNS-12518: TNS:listener could not hand off client connection
TNS-12571: TNS:packet writer failure
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
32-bit Windows Error: 54: Unknown error


Error Description:
ERROR_NETWORK_BUSY
54
0x36
The network is busy.


Cause:
This indicates a bottleneck at the network layer(TCP/IP).


Action:
1.Try increasing dispatchers and shared servers.

##关于TNS-12518错误更多信息，请参考：

Troubleshooting Guide for TNS-12518 TNS listener could not hand off client connection (Doc ID 550859.1)

OERR: ORA-12518 / TNS-12518 "TNS:listener could not hand off client connection" Reference Note (Doc ID 71928.1)

ORA-12518 / TNS-12518 Troubleshooting (Doc ID 556428.1)