10g11gConfiguration of TAF(Transparent Application Failover) and Load Balancing

10g & 11g :Configuration of TAF(Transparent Application Failover) and Load Balancing [ID 453293.1] 
 

In this Document
  Goal
  Solution
      CONCEPTS
     CONFIGURATION
     Client Side Load Balancing
     Connect Time Failover
     Client Side Load Balance & Connect-time Failover Connect String
     TAF - Client Side Configuration
     TAF - Server Side Configuration
     Troubleshooting Load Balance
     Troubleshooting TAF
  References

Applies to:
Oracle Net Services - Version: 10.1.0.2.0 to 11.1.0.7.0
Information in this document applies to any platform.
Goal

This note explains the configuration in 10g & 11g  for :

1) Client Side Connect Time Load Balance
2) Client Side Connect Time Failover
3) Server Side Listener Connection Load Balance
4) Transparent Application Failover (TAF)
Solution
CONCEPTS

(1) Client Side Connect-Time Load Balance

The client load balancing feature enables clients to randomize connection requests among the listeners.

Tnsnames Parameter: LOAD_BALANCE

The (load_balance=yes) instructs SQLNet to progress through the list of listener addresses in a random sequence, balancing the load on the various listeners. When set to OFF, instructs SQLNet to try the addresses sequentially until one succeeds.

Load balancing can be specified for an ADDRESS_LIST or associated with a set of ADDRESSes or set DESCRIPTIONs.

This parameter must be correctly coded in your net service name (connect descriptor).
By default, this parameter is set to ON for DESCRIPTION_LISTs.

If you use ADDRESS_LIST, (load_balance=yes) should be within the (ADDRESS_LIST=) portion. If you do not use ADDRESS_LIST, (load_balance=yes) should be within the (description=) portion.

(2) Client Side Connect-Time failover

The connect-time failover enables clients to connect to another listener if the initial connection to the first listener fails. The number of listener protocol addresses determines how many listeners are tried. Without
connect-time failover, Oracle Net attempts a connection with only one listener. The default is on.

Tnsnames Parameter: FAILOVER

(failover=on) is default for ADDRESS_LISTs, DESCRIPTION_LISTs, and a set of
DESCRIPTIONs., therefore, you do not have to specify it explicitly.


(3) Server Side Listener Connection Load Balance

The listener connection load balancing feature improves connection performance by balancing the number of active connections among multiple dispatchers and instances. In a single-instance environment, the listener selects the least loaded dispatcher to handle the incoming client requests. In an Oracle Real Application Clusters (RAC) environment, connection load balancing also has the capability to balance the number of active connections among multiple instances.
1. Least-loaded node
2. Least-loaded instance
3. Least-loaded dispatcher for that instance (in case of Shared server configuration)


There are two types of server-side load balancing:

Load Based — Server side load balancing redirects connections by default depending on node load. From 10.2 onwards you can work this with Load balancing advisory (LBA). This will not be effective for login storms as the connections are already in the listener after which PMON update on the load comes to the listener.
Session Based — Session based load balancing takes into account the number of sessions connected to each node and then distributes the connections to balance the number of sessions across the different nodes.
Session count balancing is used when you set a listener parameter, PREFER_LEAST_LOADED_NODE_<listener_name>=OFF (<listener_name> is the actual name of the listener which is different on each node in your cluster and by default is constructed as "listener_<nodename>"). Please be aware that connections bursts (large number of connections initiated very quickly one after another) may not be properly balanced due to statistics update latency.

(4) Transparent Application Failover (TAF)

Transparent Application Failover (TAF) is a feature of the Oracle Call Interface (OCI) driver at client side. It enables the application to automatically reconnect to a database, if the database instance to which the connection is made fails. In this case, the active transactions roll back.

Tnsnames Parameter: FAILOVER_MODE

When an instance to which a connection is established fails or is shutdown, the connection on the client side becomes stale and would throw exceptions to the caller trying to use it. TAF enables the application to transparently reconnect to a preconfigured secondary instance creating a fresh connection, but identical to the connection that was established on the first original instance.
CONFIGURATION

We will take a 2 node setup for this entire configuration example
Node1:

Hostname: node1.idc.oracle.com
VIP Hostname: node1-vip.idc.oracle.com
Database Service_names: service.idc.oracle.com
SID: sid1


Node2:

Hostname: node2.idc.oracle.com 
VIP Hostname: node2-vip.idc.oracle.com 
Database Service_names: service.idc.oracle.com 
SID: sid2


Note: Ensure that the node1-vip.idc.oracle.com and node2-vip.idc.oracle.com is resolvable & reachable from clients. If needed You may replace the hostnames with IP address in the entire configuration files.

Node1

Listener.ora
LISTENER_NODE1= 
(DESCRIPTION = 
(ADDRESS_LIST =  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521)(IP=FIRST))
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1.idc.oracle.com)(PORT = 1521)(IP=FIRST))    
) 
) 

Start the listener
$ lsnrctl start LISTENER_NODE1

Tnsnames.ora
NODE1_LOCAL=
(ADDRESS = (PROTOCOL = TCP)(HOST=node1-vip.idc.oracle.com)(PORT = 1521))    


Note: Make sure this tns alias NODE1_LOCAL is resolvable on the server, this ensures PMON picksup the entry without problem.
The following command should succeed on node1
$ tnsping node1_local

Set the local_listener for node1
sql > alter system set LOCAL_LISTENER='node1_local' scope=both sid='sid1' ;



Node2

Listener.ora
LISTENER_NODE2=   
(DESCRIPTION =  
(ADDRESS_LIST =          
(ADDRESS=(PROTOCOL=TCP)(HOST=node2.idc.oracle.com)(PORT=1521)(IP=FIRST))  
(ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip.idc.oracle.com)(PORT=1521)(IP=FIRST))  
)  
)

Start the listener
$ lsnrctl start LISTENER_NODE2

Tnsnames.ora

NODE2_LOCAL= 
   (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))


Note: Make sure this tns alias NODE2_LOCAL is resolvable on the server, this ensures PMON picksup the entry without problem.
The following command should succeed on node2
$ tnsping node2_local

Set the local_listener for node2

sql > alter system set LOCAL_LISTENER='node2_local' scope=both sid='sid2' ; 

Enable the server side Load balancing (if required)
You should make the the listeners aware of adjacent nodes load to do the server side load balance.
To make PMON to notify the load information to adjacent nodes,  you should set the REMOTE_LISTENER parameter.

Add the following tnsalias in both the nodes tnsnames.ora
Node1 & Node2 Tnsnames.ora
NODE_REMOTE =
(DESCRIPTION = 
(ADDRESS_LIST =  
  (ADDRESS = (PROTOCOL = TCP)(HOST=node1-vip.idc.oracle.com)(PORT = 1521))  
  (ADDRESS = (PROTOCOL = TCP)(HOST=node2-vip.idc.oracle.com)(PORT = 1521))  
) 
)


Note: Make sure this tns alias NODE_REMOTE is resolvable on all the servers, this ensures PMON picksup the entry without problem.
The following command should succeed on all the nodes
$ tnsping node_remote

Set the remote listeners to both the nodes
sql> alter system set REMOTE_LISTENER='node_remote' scope=both;


Client Side Load Balancing

Following is an example of TNS net service alias for client side load balancing
CLIENT_LOADBALANCE=
(DESCRIPTION =
  (LOAD_BALANCE = yes)
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521)) 
  (CONNECT_DATA = 
   (SERVICE_NAME = service.idc.oracle.com) 
  )
)
Connect Time Failover

Sample connect time failover TNS net service alias
address_list defaults the failover=on
CLIENT_FAILOVER=
(DESCRIPTION = 
(ADDRESS_LIST = 
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521)) 
)
(CONNECT_DATA =
   (SERVICE_NAME = service.idc.oracle.com)
)
)
Client Side Load Balance & Connect-time Failover Connect String
CLIENT_LOAD_FAILOVER=
(DESCRIPTION =
(ADDRESS_LIST =
  (LOAD_BALANCE = yes)
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))
)
(CONNECT_DATA =
  (SERVICE_NAME = service.idc.oracle.com)
)
)


TAF - Client Side Configuration

Failover Mode Type can be Either SESSION or SELECT.
Session failover will have just the session to failed over to the next available node.The select query will not be resumed.

You can refer the below document for various options on TAF
Oracle® Database Net Services Administrator's Guide
Chapter on Enabling Advanced Features of Oracle Net Services

Simple TAF can be configured on the client side tnsnames.ora with following entry
TAF=
(DESCRIPTION =
(ADDRESS_LIST =
  (LOAD_BALANCE = yes)
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))
)
(CONNECT_DATA =
   (SERVICE_NAME = service.idc.oracle.com)
   (failover_mode=(type=select)(method=basic))
    )
  )
TAF - Server Side Configuration

TAF can be configured with server side at the time of service creation also.
Refer Note 404644.1Configuration of Tansparent Application Failover(TAF) works with server side service.
Troubleshooting Load Balance

Incase of server side load balancing not happening properly, to start diagonisng the issue you may collect the below set of information:

a. Is session load balance turned on ?
b. What is the connection distribution on nodes ?
c. Is all the nodes are equal power w.r.t CPU / Memory / Hardware ?
d. Is there any application connecting locally (BEQueath connection) and doing big operations on any one of the node?
e. Are all the nodes are configured with proper settings of local / remote listeners?
Refer NOTE 263599.1 Understanding and Troubleshooting Instance Load Balancing
Troubleshooting TAF

Incase of client connectivity failure the ideal start point to diagonise the issue is with Client Sqlnet tracing at support Level

To enable the client tracing  Refer
NOTE 395525.1How to Enable Oracle SQLNet Client , Server , Listener , Kerberos and External procedure Tracing from Net Manager

While creating an Service Request with Oracle Support for TAF , provide the following information to diagonise the problem:

a. Failed connection client sqlnet tracing

b. Execute and provide the output of following commands from both the nodes:
    $ sql > show parameter Listener
    $ sql> show parameter service
    $ lsnrctl  services <listener_name>

c. Upload the tnsnames.ora & Listener logs (by default created with <listener_name>.log) from both nodes.
References
NOTE:226880.1 - Configuration of Load Balancing and Transparent Application Failover
NOTE:342419.1 - Ora-12520 When listeners on VIP in 10g RAC Setup
NOTE:395525.1 - How to Enable Oracle SQLNet Client , Server , Listener , Kerberos and External procedure Tracing from Net Manager