Listener Hanging - Information to Get For Resolving or Troubleshooting (Doc ID 230156.1)

In this Document

Purpose

Scope

Details

APPLIES TO:

Oracle Net Services - Version 8.1.7.4 to 11.2.0.4 [Release 8.1.7 to 11.2]
Information in this document applies to any platform.
***Checked for relevance on 7-MAY-2014.***

PURPOSE

This note is to assist and gather relevant information in troubleshooting issues concerning the TNS Listener hanging.  This note only applies to listeners hanging while running; it does not apply to listeners not starting.

The note is divided into three sections:

• Questions to be Addressed
• Technical Data to Obtain
• Recommendations to Immediately Provide Relief of a Hanging Listener

SCOPE

It is crucial that all questions and information be answered and supplied.

DETAILS

Questions to be Addressed
Example answers are given in the yellow blocks to demonstrate what type of information is expected.

1. What is the version and bit of the OS and of the Oracle software?

Example Answer: Linux Redhat AS4 64bit.  Oracle 10.2.0.3 64 bit

2. Can you predict when the hang will occur?

Example Answer: The listener seems to hang after it has been running for a few days

3. What is the system load at the time of the hang?

Example Answer: I have not noticed any heavy load at the time the listener hangs.  Just normal usage.

4. Can a bequeath connection be made to Oracle when the listener hangs?

Example Answer: Yes, "sqlplus system/manager" works while the listener is hung.

5. Is Shared Server being used

Example Answer: Yes, shared server is configured

6. Is this listener a part of a RAC environment.

Example Answer: No, this is a standalone listener.

Technical Data to Obtain

1. Level 16 trace file of the listener.

• If this is an 11g listener, add this parameter to the listener.ora file before adding any other tracing parameters.

DIAG_ADR_ENABLED_listenername = OFF

 

• If the listener hangs relatively quickly, add this parameter to the listener.ora file and restart the listener

TRACE_LEVEL_listenername = 16

 

• If the listener hangs after a long amount of time (more than 24 hrs), use cyclic tracing and add these parameters to the listener.ora file.  These parameters cycle between two 500MB trace files.

TRACE_LEVEL_listenername = 16
TRACE_FILELEN_listenername = 512000
TRACE_FILENO_listenername = 2

2. An RDA.  Follow these notes

Note:314422.1    "Remote Diagnostic Agent (RDA) 4 - Overview"

Optional Notes:
Note:330363.1    "Remote Diagnostic Agent (RDA) 4 - FAQ"
Note:330344.1    "Remote Diagnostic Agent (RDA) 4 - Training"
Note:330362.1    "Remote Diagnostic Agent (RDA) 4 - Troubleshooting Guide"

3. The listener.log file

4. Stack traces of the hung listener process.  Provide 3 stack traces - one about every 30 seconds.

Solaris	pstack <listener_pid>
AIX (pre 5.2)	Type: dbx -a <listener_pid> At the next prompt, type: where
AIX (5.2 and higher)	procstack <listener_pid>
HPUX	Type: gdb $ORACLE_HOME/bin/tnslsnr <listener_pid> At the next prompt, type: where
Linux	Type: gdb $ORACLE_HOME/bin/tnslsnr <listener_pid> At the next prompt, type: where

5. Memory usage of the listener.  These samples should be taken when the listener is started, after the listener is started during normal usage, and at the time of the hang.  The idea is to test for large growths of memory consumption.

Solaris	pmap -x <listener_pid>
AIX (5.2 +)	procmap <listener_pid>
HPUX	export UNIX95=1 ps -e -o pid, ruser, vsz, args | grep tnslsnr
Linux	cat /proc/<listener_pid>/status

6. CPU usage of the listener at the time the listener hangs:
Providing a "top" output showing the tnslsnr processes is the easiest way to provide this information


Recommendations to Immediately Provide Relief of a Hanging Listener

1. The listener should always be patched to the latest release of the current version being used. Ideally, the listener should be patched to the newest and latest production release of Oracle itself. There is no need to upgrade the database. Example upgrade paths (at the time of this writing):

• A 10.1.0.2 listener should be upgraded to 10.1.0.5
• A 11.1.0.6 listener should be upgraded to 11.1.0.7

2. Try running the listener on a different port (it is possible a 3rd party application may be interfering)

3. (Pre 11g) Keep the listener.log file small in size.  It is mandatory the listener.log file be less than 2GB in size in releases prior to 11g.  Not doing so results in process instability, random crashes, and random hangs

4. If listener load is high, try starting a second listener and configure load balancing to assist with connection load.

5. Add the parameter  DIRECT_HANDOFF_TTC_listenername=OFF to the listener.ora file and restart the listener.

6. In a non-RAC environment, add the parameter SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF to the listener.ora file and restart the listener.

7. Verify the operating system is running at its most current level.