CRS-4402（Doc ID 1212703.1） 续

http://my.oschina.net/1272149624/blog/620653

因字数问题，接着

CRS-4402（Doc ID 1212703.1）

Configuring various network switches for multicast

While the most common reason for multicast based communication to fail with Oracle Grid Infrastructure 11.2.0.2 is probably the 230.0.1.0 multicast address being used by default, multicast being disabled or misconfigured on the network switches is an equally common cause. The mcasttest.pl-tool provided as part of this note will give you an indication whether or not multicasting has been disabled on the network switches, if it fails to use multicast on either the 230.0.1.0 or 224.0.0.251 multicast address for example. In general, work with your network administrator, if you suspect multicasting has been disabled on the network switches, affecting the private interconnect communication accordingly.

Cisco Nexus Switches - general 

The Cisco Nexus switches can be complicated to configure with multicasting. If multicasting cannot be properly configured, then these switches have the ability to turn off IGMP Snooping on a per-VLAN basis. This has the effect of having the switch handle multicasts like it would a broadcast. Since the network used for the interconnect is supposed to be private (where only the members of the cluster have access to this network), broadcasting every multicast packet to every NIC on the VLAN should pose no problem. The command to turn off IGMP Snooping on a VLAN is:

no ip igmp snooping 

(Consult the Cisco Nexus manual for the exact syntax of this command.)

Cisco Catalyst 

Turning off IGMP Snooping on a Cisco Catalyst switch is a more complex process, which is not covered in this note. For the Catalyst family of switches, IGMP Snooping can only be turned off globally and not on a per-VLAN basis. Consult the Cisco Catalyst manuals for how to properly configure multicasting for the switches over-all usage. Note that turning off IGMP Snooping on the Catalyst may cause other systems or programs to fail.

Cisco Nexus 7000 Switch 

A feature of the Cisco Nexus 7000 switch is a filter that effects large UDP messages. This command is designed to filter IP packets with the "do not fragment" bit set along with a non-zero "Fragment offset". Oracle RAC will send UDP messages that exceed the carrying capacity of a standard UDP packet. The IP layer sees this and sends the left-over data in IP Fragments. These IP Fragments will then have the "do not fragment" bit set along with a non-zero "Fragment offset". The switch will not deliver these packets as they are filtered out by this rule. 

To validate this issue, perform a tcpdump (with the -s 0 flag to capture the entire packet) on each node during the Oracle Clusterware installation at the time the root.sh script is run. The root.sh command will start the Oracle cluster services on the first node. When it is run on the second node, large UDP packets can occur, which will generate the IP Fragments that will be dropped. When one reads the two tcpdump files, on will see an IP Fragment packet leave the second node, yet it will not arrive on the first node. The first node will send out ICMP "Fragment reassembly time exceeded" packets as a result. To turn this rule off issue the following command:

no hardware ip verify fragment 

(Consult the Cisco Nexus manual for the exact syntax of this command.)

These settings apply only to the Cisco Nexus 7000 switch. This command may be available on other models of the Nexus switch, but not all of the Nexus models support it. Check the Cisco manual for the model of concern.

 

 

SOLUTION

For 11.2.0.2 Installations

It has been found that using a 230.0.1.0, port 42424, network address for multicasting can be problematic with some network configurations. Therefore, Oracle has released Patch: 9974223 on top of Oracle Grid Infrastructure 11.2.0.2. This patch enables multicasting on the 224.0.0.251 multicast address (port 42424) in addition to the 230.0.1.0 (port 42424) address used by default. Multicast must generally be enabled on one of these two addresses to allow Oracle Grid Infrastructure to successfully start on all nodes configured to join a particular cluster.

For 11.2.0.3.5, 11.2.0.3.6 and 11.2.0.3.7 GI PSU  and 12.1.0.1 Installations

11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 installations will ONLY be impacted if those installations were relying on the broadcast functionality that was initially introduced in 11.2.0.3 base.  For these installations the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address (only 1 of the 2 addresses needs to be functional).  For those installations that are unable to enable multicast, Patch 16547309 is available for some platforms to re-enable broadcast functionality.  If you require this patch (due to not being able to enable multicast) and it is not available for your version/platofrm please raise an SR with support to request the fix.  The fix for Bug 16547309 will be included in the 11.2.0.3.8 and 12.1.0.1.2 GI PSUs.

 

Problem validation

Before applying Patch: 9974223, Patch 16547309 or any subsequent Oracle Grid Infrastructure (GI) Bundle Patch including which include the mentioned fixes, understand that the issue described in this note does not apply under the following circumstances:

Note: The issue described in this note does not apply

on Windows

The Redundant Interconnect Usage feature does not exist on Windows.

on Solaris when using Solaris Cluster

When using Sun Cluster, Oracle Clusterware 11.2.0.2 will detect the presence of clprivnet0 and not enable any HAIPs. The Redundant Interconnect Usage feature is thereby disabled. clprivnet0 will provide the required availability for redundant network interfaces used for the interconnect.

on HP-UX or AIX when using Infiniband network cards

If on HP-UX or AIX using Infiniband network interfaces the installation of Oracle Grid Infrastructure will fail due to OS and network hardware related issues, unrelated to multicast.

These are known BUGs and Oracle is working with the respective OS vendors for a fix.

Contact Oracle Support if you plan on upgrading to Oracle Grid Infrastructure 11.2.0.2 on either of those platforms using Infiniband network cards.

using Primecluster

The Redundant Interconnect Usage feature is currently disabled when using Primecluster.

 If you have already run the root.sh (rootupgrade.sh) on any node of the cluster:

Note: The README for Patch: 9974223 and Patch 16547309 contain instructions for those installations in which root.sh (or rootupgrade.sh) has failed due to this issue.  A complete reinstall of Grid Infrastructure should  not be required.

Multicast can be vaildated using the mcasttest.pl tool attached to this note to validate whether or not multicast on the 230.0.1.0 or 224.0.0.251 address can be used on the private interconnect interface(s).  CLUFFY in 11.2.0.3 and above does contain similar checks but will report pass if the broadcast check passes, with mentioned issue which breaks broadcast in 11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 this pass is a false positive.  For this reason it is recommended to use mcasttest.pl program to validate multicast functionality before patching/upgrading/installing the mentioned releases.

Note:  Multicast based communication only needs to be successful on either the 230.0.1.0 address or 

the 224.0.0.251 address. A successful multicast communication on both addresses is not required. 

How to use the mcasttest-tool

Download the mcasttest.pl perl program from this note and extract the contents of the package to node 1 of your cluster:

# gunzip mcasttest.tgz 

# tar xvf mcasttest.tar

The mcasttest.pl program can be found in the mcasttest directory extracted in the previous step.

The mcasttest.pl program requires two arguments:

The node list (specified with -n)

The list of interfaces to be used for the private interconnect (specified with -i).

The mcasttest is only to be executed on a single node in the cluster. It will use password-less SSH based access (also required for the Grid Infrastructure software installation) to distribute 

and execute the program on the remote nodes. The command string to execute mcasttest is:

# perl mcasttest.pl -n <node1>,<node2>,<node_n...> -i <interface1>,<interface2><interface_n...>

The example below tests multicast for a two node cluster (ratlnx01 and ratlnx02), in 

which two network interfaces (eth1 and eth2) are to be used for the private interconnect:

# perl mcasttest.pl -n ratlnx01,ratlnx02 -i eth1,eth2

########### Setup for node ratlnx01 ##########

Checking node access 'ratlnx01'

Checking node login 'ratlnx01'

Checking/Creating Directory /tmp/mcasttest for binary on node 'ratlnx01'

Distributing mcast2 binary to node 'ratlnx01'

########### Setup for node ratlnx02 ##########

Checking node access 'ratlnx02'

Checking node login 'ratlnx02'

Checking/Creating Directory /tmp/mcasttest for binary on node 'ratlnx02'

Distributing mcast2 binary to node 'ratlnx02'

########### testing Multicast on all nodes ##########

Test for Multicast address 230.0.1.0

Nov 8 09:05:33 | Multicast Failed for eth1 using address 230.0.1.0:42000

Nov 8 09:05:34 | Multicast Failed for eth2 using address 230.0.1.0:42001

Test for Multicast address 224.0.0.251

Nov 8 09:05:35 | Multicast Succeeded for eth1 using address 224.0.0.251:42002

Nov 8 09:05:36 | Multicast Succeeded for eth2 using address 224.0.0.251:42003

As shown in the example above, the test has failed for the 230.0.1.0 address, but succeeded for the 224.0.0.251 multicast address. In this case, Patch: 9974223 must be applied to enable Oracle Grid Infrastructure to use the 224.0.0.251 multicast address.  

Interpreting the outcome of the mcasttest-tool correctly:

Should the mcasttest.pl test-tool have failed for both, the 230.0.1.0 and the 224.0.0.251 address

you must work with your System and / or Network Administrator to enable multicast on one of the addresses.

Should the mcasttest.pl test-tool have failed for the 230.0.1.0 address only

Apply Patch: 9974223 or any subsequent GI PSU (fix provided GI PSU 1 and above)

Should the mcasttest.pl test-tool have returned "success" for both, the 230.0.1.0 and the 224.0.0.251 address

No specific patch application is required and you can proceed with the installation.  

When and how to patch for 11.2.0.2

Ideally, Patch: 9974223 or the latest 11.2.0.2 GI PSU  (fix included in GI PSU 1 and above) is applied before any root.sh or rootupgrade.sh has ever been run on any of the nodes meant to form a cluster. It is highly recommended that the latest 11.2.0.2 GI PSU is installed to obtain the fix for Bug: 9974223 (instead of Patch: 9974223 standalone). In order to apply Patch: 9974223 either directly or as part of a Bundle Patch, perform the following steps:

Perform an Oracle Grid Infrastructure 11.2.0.2 installation or upgrade

Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.

Do not close the installer or abort an operation in progress.

Leave the current installation as-is and open a new terminal.

Download and prepare the application of Patch: 9974223 either directly or as part of a GI Bundle.

Unlike described in the patch readme (of the GI Bundle Patch),

do not use "opatch auto"

Instead, use: "opatch napply -local"

IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

 Note: as Opatch is used with the "-local" flag here, you need to perform this operation on every node.

After you have patched every node in the cluster, return to the original installation

Proceed to run the root.sh (rootupgrade.sh) on all nodes and follow the instructions on the screen.

When and how to patch for 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7

As previously mentioned, the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address to address th.  If this is not possible within your environment, the recommended solution is to install the fix for Bug 16547309 prior to executing "rootcrs.pl -patch".

When and how to patch for 12.1.0.1

Again, the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address to address this issue.  If this is not possible within your environment, the recommended solution is to install the fix for Bug 16547309 or the latest GI PSU containing the fix (12.1.0.1.2) prior to running root.sh or rootupgrade.sh.  In order to apply Patch 16547309 prior to the execution of root.sh or rootupgrade.sh, perform the following steps:

Perform an Oracle Grid Infrastructure 12.1.0.1 installation or upgrade

Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.

Do not close the installer or abort an operation in progress.

Leave the current installation as-is and open a new terminal.

Download and prepare the application of Patch 16547309 either directly or as part of a GI Bundle.

Unlike described in the patch readme (of the GI Bundle Patch),

do not use "opatch auto"

Instead, use: "opatch napply -local"

IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

 

Disclaimer and summary: Due to differences in network hardware and overall network topologies that may be used for the interconnect network, it is not possible for Oracle to provide a single solution for enabling multicast within your network environment. That being said, it is important that you work with your System and / or  Network Administrators to enable multicast functionality for the 230.0.1.0 or the 224.0.0.251 multicast address (using Patch: 9974223) . The mcasttest.pl program attached to this note should be used to ensure that multicast communication is functioning properly to support Oracle Grid Infrastructure.

 

Scalability RAC Community

To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Scalability RAC Community.