ipv6造成的死锁问题
Description
For sites using only IPv4, you may find better performance and simplicity in configuring the environment to use only IPv4, wherever possible. This wiki page provides guidance on forcing applications to use IPv4 when possible.
Many OSs will enable IPv6 by default, even if the environment is only using IPv4. Configuring the OS to disable IPv6 would help to prevent these sorts of problems.
Known Problems
There are some known problems with applications trying to use IPv6, when the environment is really only configured for IPv4.
1. IPv6 changes the way that Round-Robin A-records are used, and instead forces an ordered priority weighting of A-records. This can cause high-availability techniques dependent on round-robin A-records to fail or work incorrectly. If using IPv4/IPv6 dual-stack, then the JVM will use getaddrinfo() instead of gethostbyname(). getaddrinfo() will sort the DNS results in an ordered way, due to RFC 3484 [1]. Some OSs provide /etc/gai.conf configuration options in an attempt to configure record handling.
2. The JVM can be blocked by an infinite loop problem between the JVM and libc when using the IPv6 getaddrinfo() libraries with IPv4 addresses. This issue is discussed in more detail here:
http://bugzilla.zimbra.com/show_bug.cgi?id=68432
http://old.nabble.com/-Bug-libc-12926--New%3A-getaddrinfo%28%29-make_request%28%29-may-spin-forever-td31913044.html
http://sourceware.org/bugzilla/show_bug.cgi?id=12926
In the JVM, following is what the threads look like. You will see many threads in the thread dump file locked in a state similar to this:
"btpool0-41529" prio=10 tid=0x00002aaac45dd000 nid=0x7db9 in Object.wait() [0x0000000047155000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.net.InetAddress.checkLookupTable(InetAddress.java:1267)
- locked <0x00000006e385b000> (a java.util.HashMap)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1190)
These threads are waiting for the Lookup Table object to be released from another thread that holds it. Here's the thread holding it:
"btpool0-41526" prio=10 tid=0x00002aaac5904800 nid=0x7db6 runnable [0x0000000044c02000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201)
at java.net.InetAddress.getAllByName0(InetAddress.java:1154)
at java.net.InetAddress.getAllByName(InetAddress.java:1084)
at java.net.InetAddress.getAllByName(InetAddress.java:1020)
at java.net.InetAddress.getByName(InetAddress.java:970)
You'll see that's running Inet6AddressImpl.lookupAllHostAddr. Because of a bug between Java and libc, this lookup can enter an infinite loop when a certain race condition occurs. This occurs infrequently, but can cause deadlocks where all threads of one type (such as LMTP threads) or even all JVM threads can end up blocked.
With java.net.preferIPv4Stack set to true, Java will not execute this code and the problem should be avoided.
Configuration
1. Java processes can be configured to prefer the IPv4 stack. The default is to prefer the IPv6 stack, so it requires a specified JVM argument to prefer IPv4:
-Djava.net.preferIPv4Stack=true
This would need to be added to your existing mailboxd_java_options. Your existing configuration may vary depending on your performance tuning [see http://wiki.zimbra.com/wiki/Performance_Tuning_Guidelines_for_Large_Deployments], so be careful to append this option to whatever is there currently:
# su - zimbra
$ zmlocalconfig mailboxd_java_options
$ zmlocalconfig -e mailboxd_java_options="-server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:PermSize=192m -XX:MaxPermSize=192m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/zimbra/log -Djava.net.preferIPv4Stack=true"
For more reference background on this, please review here:
http://bugzilla.zimbra.com/show_bug.cgi?id=13161#c55
2. Configuring the OS to disable IPv6
Each OS may have unique recommendations for disabling IPv6. This article does not currently include all OS-level recommendations, but please do a web search and determine methods for disabling the IPv6 interfaces, modules, and stack for your OS of choice.
For sites using only IPv4, you may find better performance and simplicity in configuring the environment to use only IPv4, wherever possible. This wiki page provides guidance on forcing applications to use IPv4 when possible.
Many OSs will enable IPv6 by default, even if the environment is only using IPv4. Configuring the OS to disable IPv6 would help to prevent these sorts of problems.
Known Problems
There are some known problems with applications trying to use IPv6, when the environment is really only configured for IPv4.
1. IPv6 changes the way that Round-Robin A-records are used, and instead forces an ordered priority weighting of A-records. This can cause high-availability techniques dependent on round-robin A-records to fail or work incorrectly. If using IPv4/IPv6 dual-stack, then the JVM will use getaddrinfo() instead of gethostbyname(). getaddrinfo() will sort the DNS results in an ordered way, due to RFC 3484 [1]. Some OSs provide /etc/gai.conf configuration options in an attempt to configure record handling.
2. The JVM can be blocked by an infinite loop problem between the JVM and libc when using the IPv6 getaddrinfo() libraries with IPv4 addresses. This issue is discussed in more detail here:
http://bugzilla.zimbra.com/show_bug.cgi?id=68432
http://old.nabble.com/-Bug-libc-12926--New%3A-getaddrinfo%28%29-make_request%28%29-may-spin-forever-td31913044.html
http://sourceware.org/bugzilla/show_bug.cgi?id=12926
In the JVM, following is what the threads look like. You will see many threads in the thread dump file locked in a state similar to this:
"btpool0-41529" prio=10 tid=0x00002aaac45dd000 nid=0x7db9 in Object.wait() [0x0000000047155000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.net.InetAddress.checkLookupTable(InetAddress.java:1267)
- locked <0x00000006e385b000> (a java.util.HashMap)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1190)
These threads are waiting for the Lookup Table object to be released from another thread that holds it. Here's the thread holding it:
"btpool0-41526" prio=10 tid=0x00002aaac5904800 nid=0x7db6 runnable [0x0000000044c02000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201)
at java.net.InetAddress.getAllByName0(InetAddress.java:1154)
at java.net.InetAddress.getAllByName(InetAddress.java:1084)
at java.net.InetAddress.getAllByName(InetAddress.java:1020)
at java.net.InetAddress.getByName(InetAddress.java:970)
You'll see that's running Inet6AddressImpl.lookupAllHostAddr. Because of a bug between Java and libc, this lookup can enter an infinite loop when a certain race condition occurs. This occurs infrequently, but can cause deadlocks where all threads of one type (such as LMTP threads) or even all JVM threads can end up blocked.
With java.net.preferIPv4Stack set to true, Java will not execute this code and the problem should be avoided.
Configuration
1. Java processes can be configured to prefer the IPv4 stack. The default is to prefer the IPv6 stack, so it requires a specified JVM argument to prefer IPv4:
-Djava.net.preferIPv4Stack=true
This would need to be added to your existing mailboxd_java_options. Your existing configuration may vary depending on your performance tuning [see http://wiki.zimbra.com/wiki/Performance_Tuning_Guidelines_for_Large_Deployments], so be careful to append this option to whatever is there currently:
# su - zimbra
$ zmlocalconfig mailboxd_java_options
$ zmlocalconfig -e mailboxd_java_options="-server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:PermSize=192m -XX:MaxPermSize=192m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/zimbra/log -Djava.net.preferIPv4Stack=true"
For more reference background on this, please review here:
http://bugzilla.zimbra.com/show_bug.cgi?id=13161#c55
2. Configuring the OS to disable IPv6
Each OS may have unique recommendations for disabling IPv6. This article does not currently include all OS-level recommendations, but please do a web search and determine methods for disabling the IPv6 interfaces, modules, and stack for your OS of choice.