c++比例-libcurl多线程并发时的core

转自： https://www.cnblogs.com/edgeyang/articles/3722035.html

 

浅析libcurl多线程安全问题

背景：使用多线程libcurl发送请求，在未设置超时或长超时的情况下程序运行良好。但只要设置了较短超时（小于180s），程序就会出现随机的coredump。并且栈里面找不到任何有用的信息。

问题：1.为什么未设置超时，或者长超时时间（比如601s）的情况下多线程libcurl不会core？

问题：2.进程coredump并不是必现，是否在libcurl内多线程同时修改了全局变量导致？

 

先来看下官方libcurl的说明：

libcurl is free, thread-safe, IPv6 compatible, feature rich, well supported, fast, thoroughly documented and is already used by many known, big and successful companies and numerous applications.

可以看到官方自称licurl是线程安全的，是否真的如此？再来看看代码中用到的超时选项的说明：

CURLOPT_TIMEOUT

Pass a long as parameter containing the maximum time in seconds that you allow the libcurl transfer operation to take. Normally, name lookups can take a considerable time and limiting operations to less than a few minutes risk aborting perfectly normal operations. This option will cause curl to use the SIGALRM to enable time-outing system calls.

In unix-like systems, this might cause signals to be used unless CURLOPT_NOSIGNAL is set.

Default timeout is 0 (zero) which means it never times out.

选项提到了超时机制是使用SIGALRM信号量来实现的，并且在unix-like操作系统中又提到了另外一个选项CURLOPT_NOSIGNAL：

CURLOPT_NOSIGNAL

Pass a long. If it is 1, libcurl will not use any functions that install signal handlers or any functions that cause signals to be sent to the process. This option is mainly here to allow multi-threaded unix applications to still set/use all timeout options etc, without risking getting signals. The default value for this parameter is 0. (Added in 7.10)

If this option is set and libcurl has been built with the standard name resolver, timeouts will not occur while the name resolve takes place. Consider building libcurl with c-ares support to enable asynchronous DNS lookups, which enables nice timeouts for name resolves without signals.

Setting CURLOPT_NOSIGNAL to 1 makes libcurl NOT ask the system to ignore SIGPIPE signals, which otherwise are sent by the system when trying to send data to a socket which is closed in the other end. libcurl makes an effort to never cause such SIGPIPEs to trigger, but some operating systems have no way to avoid them and even on those that have there are some corner cases when they may still happen, contrary to our desire. In addition, usingCURLAUTH_NTLM_WB authentication could cause a SIGCHLD signal to be raised.

该选项说明提到，为了在多线程中允许程序去设置timeout选项，但不是使用signals，需要设置CURLOPT_NOSIGNAL为1 。

于是在代码中加上了这句，测试再没有发现有coredump的情况。

 1 easy_setopt(curl, CURLOPT_NOSIGNAL, (long)1); 

问题：3.timeout机制实现机制是什么，为什么设置了选项CURLOPT_NOSIGNAL线程就安全了？

 

为了解答上面的问题，需要查看libcurl的相关源代码，以下是DNS解析的函数：

  1 int Curl_resolv_timeout(struct connectdata *conn,
  2                         const char *hostname,
  3                         int port,
  4                         struct Curl_dns_entry **entry,
  5                         long timeoutms)
  6 {
  7 #ifdef USE_ALARM_TIMEOUT
  8 #ifdef HAVE_SIGACTION
  9   struct sigaction keep_sigact;   /* store the old struct here */
 10   volatile bool keep_copysig = FALSE; /* wether old sigact has been saved */
 11   struct sigaction sigact;
 12 #else
 13 #ifdef HAVE_SIGNAL
 14   void (*keep_sigact)(int);       /* store the old handler here */
 15 #endif /* HAVE_SIGNAL */
 16 #endif /* HAVE_SIGACTION */
 17   volatile long timeout;
 18   volatile unsigned int prev_alarm = 0;
 19   struct SessionHandle *data = conn->data;
 20 #endif /* USE_ALARM_TIMEOUT */
 21   int rc;
 22 
 23   *entry = NULL;
 24 
 25   if(timeoutms < 0)
 26     /* got an already expired timeout */
 27     return CURLRESOLV_TIMEDOUT;
 28 
 29 #ifdef USE_ALARM_TIMEOUT
 30   if(data->set.no_signal)
 31     /* Ignore the timeout when signals are disabled */
 32     timeout = 0;
 33   else
 34     timeout = timeoutms;
 35 
 36   if(!timeout)
 37     /* USE_ALARM_TIMEOUT defined, but no timeout actually requested */
 38     return Curl_resolv(conn, hostname, port, entry);
 39 
 40   if(timeout < 1000)
 41     /* The alarm() function only provides integer second resolution, so if
 42        we want to wait less than one second we must bail out already now. */
 43     return CURLRESOLV_TIMEDOUT;
 44 
 45   /*************************************************************
 46    * Set signal handler to catch SIGALRM
 47    * Store the old value to be able to set it back later!
 48    *************************************************************/
 49 #ifdef HAVE_SIGACTION
 50   sigaction(SIGALRM, NULL, &sigact);
 51   keep_sigact = sigact;
 52   keep_copysig = TRUE; /* yes, we have a copy */
 53   sigact.sa_handler = alarmfunc;
 54 #ifdef SA_RESTART
 55   /* HPUX doesn't have SA_RESTART but defaults to that behaviour! */
 56   sigact.sa_flags &= ~SA_RESTART;
 57 #endif
 58   /* now set the new struct */
 59   sigaction(SIGALRM, &sigact, NULL);
 60 #else /* HAVE_SIGACTION */
 61   /* no sigaction(), revert to the much lamer signal() */
 62 #ifdef HAVE_SIGNAL
 63   keep_sigact = signal(SIGALRM, alarmfunc);
 64 #endif
 65 #endif /* HAVE_SIGACTION */
 66 
 67   /* alarm() makes a signal get sent when the timeout fires off, and that
 68      will abort system calls */
 69   prev_alarm = alarm(curlx_sltoui(timeout/1000L));
 70 
 71   /* This allows us to time-out from the name resolver, as the timeout
 72      will generate a signal and we will siglongjmp() from that here.
 73      This technique has problems (see alarmfunc).
 74      This should be the last thing we do before calling Curl_resolv(),
 75      as otherwise we'd have to worry about variables that get modified
 76      before we invoke Curl_resolv() (and thus use "volatile"). */
 77   if(sigsetjmp(curl_jmpenv, 1)) {
 78     /* this is coming from a siglongjmp() after an alarm signal */
 79     failf(data, "name lookup timed out");
 80     rc = CURLRESOLV_ERROR;
 81     goto clean_up;
 82   }
 83 
 84 #else
 85 #ifndef CURLRES_ASYNCH
 86   if(timeoutms)
 87     infof(conn->data, "timeout on name lookup is not supported\n");
 88 #else
 89   (void)timeoutms; /* timeoutms not used with an async resolver */
 90 #endif
 91 #endif /* USE_ALARM_TIMEOUT */
 92 
 93   /* Perform the actual name resolution. This might be interrupted by an
 94    * alarm if it takes too long.
 95    */
 96   rc = Curl_resolv(conn, hostname, port, entry);
 97 
 98 #ifdef USE_ALARM_TIMEOUT
 99 clean_up:
100 
101   if(!prev_alarm)
102     /* deactivate a possibly active alarm before uninstalling the handler */
103     alarm(0);
104 
105 #ifdef HAVE_SIGACTION
106   if(keep_copysig) {
107     /* we got a struct as it looked before, now put that one back nice
108        and clean */
109     sigaction(SIGALRM, &keep_sigact, NULL); /* put it back */
110   }
111 #else
112 #ifdef HAVE_SIGNAL
113   /* restore the previous SIGALRM handler */
114   signal(SIGALRM, keep_sigact);
115 #endif
116 #endif /* HAVE_SIGACTION */
117 
118   /* switch back the alarm() to either zero or to what it was before minus
119      the time we spent until now! */
120   if(prev_alarm) {
121     /* there was an alarm() set before us, now put it back */
122     unsigned long elapsed_ms = Curl_tvdiff(Curl_tvnow(), conn->created);
123 
124     /* the alarm period is counted in even number of seconds */
125     unsigned long alarm_set = prev_alarm - elapsed_ms/1000;
126 
127     if(!alarm_set ||
128        ((alarm_set >= 0x80000000) && (prev_alarm < 0x80000000)) ) {
129       /* if the alarm time-left reached zero or turned "negative" (counted
130          with unsigned values), we should fire off a SIGALRM here, but we
131          won't, and zero would be to switch it off so we never set it to
132          less than 1! */
133       alarm(1);
134       rc = CURLRESOLV_TIMEDOUT;
135       failf(data, "Previous alarm fired off!");
136     }
137     else
138       alarm((unsigned int)alarm_set);
139   }
140 #endif /* USE_ALARM_TIMEOUT */
141 
142   return rc;
143 }

View Code

由此可见，DNS解析阶段timeout的实现机制是通过SIGALRM+sigsetjmp/siglongjmp来实现的。

解析前，通过alarm设定超时时间，并设置跳转的标记：

 1   /* alarm() makes a signal get sent when the timeout fires off, and that
 2      will abort system calls */
 3   prev_alarm = alarm(curlx_sltoui(timeout/1000L));
 4 
 5   /* This allows us to time-out from the name resolver, as the timeout
 6      will generate a signal and we will siglongjmp() from that here.
 7      This technique has problems (see alarmfunc).
 8      This should be the last thing we do before calling Curl_resolv(),
 9      as otherwise we'd have to worry about variables that get modified
10      before we invoke Curl_resolv() (and thus use "volatile"). */
11   if(sigsetjmp(curl_jmpenv, 1)) {
12     /* this is coming from a siglongjmp() after an alarm signal */
13     failf(data, "name lookup timed out");
14     rc = CURLRESOLV_ERROR;
15     goto clean_up;
16   }

View Code

在等到超时后，进入alarmfunc函数实现跳转：

 1 #ifdef USE_ALARM_TIMEOUT
 2 /*
 3  * This signal handler jumps back into the main libcurl code and continues
 4  * execution.  This effectively causes the remainder of the application to run
 5  * within a signal handler which is nonportable and could lead to problems.
 6  */
 7 static
 8 RETSIGTYPE alarmfunc(int sig)
 9 {
10   /* this is for "-ansi -Wall -pedantic" to stop complaining!   (rabe) */
11   (void)sig;
12   siglongjmp(curl_jmpenv, 1);
13   return;
14 }
15 #endif /* USE_ALARM_TIMEOUT */

View Code

而CURLOPT_NOSIGNAL选项的作用是什么呢？

1   case CURLOPT_NOSIGNAL:
2     /*
3      * The application asks not to set any signal() or alarm() handlers,
4      * even when using a timeout.
5      */
6     data->set.no_signal = (0 != va_arg(param, long))?TRUE:FALSE;
7     break;

再回过头看看DNS解析的那段代码，你会发现在超时设定前有如下代码：

 1 #ifdef USE_ALARM_TIMEOUT
 2   if(data->set.no_signal)
 3     /* Ignore the timeout when signals are disabled */
 4     timeout = 0;
 5   else
 6     timeout = timeoutms;
 7 
 8   if(!timeout)
 9     /* USE_ALARM_TIMEOUT defined, but no timeout actually requested */
10     return Curl_resolv(conn, hostname, port, entry);

没错！设置了CURLOPT_NOSIGNAL选项，会把超时时间设置为0，也就是DNS解析不设置超时时间！以此来绕过SIGALRM+sigsetjmp/siglongjmp的超时机制。以此引来新的问题，DNS解析没有超时限制，不过这个官方有推荐的解决方法了。

了解了这些选项的原理之后，回到问题3，为什么使用了CURLOPT_NOSIGNAL选项后就保证了线程安全？继续看sigsetjmp/siglongjmp实现就会发现：

1 #ifdef HAVE_SIGSETJMP
2 /* Beware this is a global and unique instance. This is used to store the
3    return address that we can jump back to from inside a signal handler. This
4    is not thread-safe stuff. */
5 sigjmp_buf curl_jmpenv;
6 #endif

sigsetjmp/siglongjmp使用的curl_jmpenv是个全局唯一的变量！多个线程都会去修改该变量，破坏了栈的内容并导致coredump。看来这还是libcurl的实现问题，如果每个线程都有一个sigjmp_buf变量，是否就可以解决上面的问题呢？

看到这里，问题2也有了答案：当多个线程同时修改sigjmp_buf会出现问题，但线程间是串行的sigsetjmp/siglongjmp并不会出现问题，这有一定的随机性。

问题1，未设置超时不会有问题这很好理解，但是为什么设置长超时也不会出现问题？

原因就是在libcurl超时前，apache服务器端先超时返回了。apache超时时间一般是180s。只要libcurl超时大于180s，libcurl客户端永远都不会触发超时。而是直接返回504的错误。

是否设置了CURLOPT_NOSIGNAL就可以保证线程安全了呢？官方文档还提到了另外两个函数：

CURL *curl_easy_init( );

This function must be the first function to call, and it returns a CURL easy handle that you must use as input to other easy-functions. curl_easy_init initializes curl and this call MUST have a corresponding call to curl_easy_cleanup(3) when the operation is complete.

If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup.

You are strongly advised to not allow this automatic behaviour, by calling curl_global_init(3) yourself properly. See the description in libcurl(3) of global environment requirements for details of how to use this function.

其中curl_easy_init函数体内会调用curl_global_init，而后者是非线程安全的。

在curl_easy_init函数体内，有且仅调用一次curl_global_init：

 1 /*
 2  * curl_easy_init() is the external interface to alloc, setup and init an
 3  * easy handle that is returned. If anything goes wrong, NULL is returned.
 4  */
 5 CURL *curl_easy_init(void)
 6 {
 7   CURLcode res;
 8   struct SessionHandle *data;
 9 
10   /* Make sure we inited the global SSL stuff */
11   if(!initialized) {
12     res = curl_global_init(CURL_GLOBAL_DEFAULT);
13     if(res) {
14       /* something in the global init failed, return nothing */
15       DEBUGF(fprintf(stderr, "Error: curl_global_init failed\n"));
16       return NULL;
17     }
18   }
19 
20   /* We use curl_open() with undefined URL so far */
21   res = Curl_open(&data);
22   if(res != CURLE_OK) {
23     DEBUGF(fprintf(stderr, "Error: Curl_open failed\n"));
24     return NULL;
25   }
26 
27   return data;
28 }

View Code

但是在curl_global_init函数体内，是非线程安全的。initialized++并非原子操作，有可能出现多个线程重复执行curl_global_init。

 1 CURLcode curl_global_init(long flags)
 2 {
 3   if(initialized++)
 4     return CURLE_OK;
 5 
 6   /* Setup the default memory functions here (again) */
 7   Curl_cmalloc = (curl_malloc_callback)malloc;
 8   Curl_cfree = (curl_free_callback)free;
 9   Curl_crealloc = (curl_realloc_callback)realloc;
10   Curl_cstrdup = (curl_strdup_callback)system_strdup;
11   Curl_ccalloc = (curl_calloc_callback)calloc;
12 #if defined(WIN32) && defined(UNICODE)
13   Curl_cwcsdup = (curl_wcsdup_callback)_wcsdup;
14 #endif
15 
16   if(flags & CURL_GLOBAL_SSL)
17     if(!Curl_ssl_init()) {
18       DEBUGF(fprintf(stderr, "Error: Curl_ssl_init failed\n"));
19       return CURLE_FAILED_INIT;
20     }
21 
22   if(flags & CURL_GLOBAL_WIN32)
23     if(win32_init() != CURLE_OK) {
24       DEBUGF(fprintf(stderr, "Error: win32_init failed\n"));
25       return CURLE_FAILED_INIT;
26     }
27 
28 #ifdef __AMIGA__
29   if(!Curl_amiga_init()) {
30     DEBUGF(fprintf(stderr, "Error: Curl_amiga_init failed\n"));
31     return CURLE_FAILED_INIT;
32   }
33 #endif
34 
35 #ifdef NETWARE
36   if(netware_init()) {
37     DEBUGF(fprintf(stderr, "Warning: LONG namespace not available\n"));
38   }
39 #endif
40 
41 #ifdef USE_LIBIDN
42   idna_init();
43 #endif
44 
45   if(Curl_resolver_global_init() != CURLE_OK) {
46     DEBUGF(fprintf(stderr, "Error: resolver_global_init failed\n"));
47     return CURLE_FAILED_INIT;
48   }
49 
50 #if defined(USE_LIBSSH2) && defined(HAVE_LIBSSH2_INIT)
51   if(libssh2_init(0)) {
52     DEBUGF(fprintf(stderr, "Error: libssh2_init failed\n"));
53     return CURLE_FAILED_INIT;
54   }
55 #endif
56 
57   if(flags & CURL_GLOBAL_ACK_EINTR)
58     Curl_ack_eintr = 1;
59 
60   init_flags  = flags;
61 
62   return CURLE_OK;
63 }

View Code

所以curl_global_init需要在单线程中执行，例如在程序的开头。

最后，贴一个官方给出的多线程例子，稍作修改（docs/examples/multithreads.c）：

 1 /***************************************************************************
 2  *                                  _   _ ____  _
 3  *  Project                     ___| | | |  _ \| |
 4  *                             / __| | | | |_) | |
 5  *                            | (__| |_| |  _ <| |___
 6  *                             \___|\___/|_| \_\_____|
 7  *
 8  * Copyright (C) 1998 - 2011, Daniel Stenberg, , et al.
 9  *
10  * This software is licensed as described in the file COPYING, which
11  * you should have received as part of this distribution. The terms
12  * are also available at http://curl.haxx.se/docs/copyright.html.
13  *
14  * You may opt to use, copy, modify, merge, publish, distribute and/or sell
15  * copies of the Software, and permit persons to whom the Software is
16  * furnished to do so, under the terms of the COPYING file.
17  *
18  * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
19  * KIND, either express or implied.
20  *
21  ***************************************************************************/
22 /* A multi-threaded example that uses pthreads extensively to fetch
23  * X remote files at once */
24 
25 #include 
26 #include 
27 #include 
28 
29 #define NUMT 4
30 
31 /*
32   List of URLs to fetch.
33 
34   If you intend to use a SSL-based protocol here you MUST setup the OpenSSL
35   callback functions as described here:
36 
37   http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION
38 
39 */
40 const char * const urls[NUMT]= {
41   "http://curl.haxx.se/",
42   "ftp://cool.haxx.se/",
43   "http://www.contactor.se/",
44   "www.haxx.se"
45 };
46 
47 static void *pull_one_url(void *url)
48 {
49   CURL *curl;
50 
51   curl = curl_easy_init();
52   curl_easy_setopt(curl, CURLOPT_URL, url);
53   curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30L);    /*timeout 30s,add by edgeyang*/
54   curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);    /*no signal,add by edgeyang*/
55   curl_easy_perform(curl); /* ignores error */
56   curl_easy_cleanup(curl);
57 
58   return NULL;
59 }
60 
61 
62 /*
63    int pthread_create(pthread_t *new_thread_ID,
64    const pthread_attr_t *attr,
65    void * (*start_func)(void *), void *arg);
66 */
67 
68 int main(int argc, char **argv)
69 {
70   pthread_t tid[NUMT];
71   int i;
72   int error;
73 
74   /* Must initialize libcurl before any threads are started */
75   curl_global_init(CURL_GLOBAL_ALL);
76 
77   for(i=0; i< NUMT; i++) {
78     error = pthread_create(&tid[i],
79                            NULL, /* default attributes please */
80                            pull_one_url,
81                            (void *)urls[i]);
82     if(0 != error)
83       fprintf(stderr, "Couldn't run thread number %d, errno %d\n", i, error);
84     else
85       fprintf(stderr, "Thread %d, gets %s\n", i, urls[i]);
86   }
87 
88   /* now wait for all threads to terminate */
89   for(i=0; i< NUMT; i++) {
90     error = pthread_join(tid[i], NULL);
91     fprintf(stderr, "Thread %d terminated\n", i);
92   }
93 
94   curl_global_cleanup();    /*add by edgeyang*/
95   return 0;
96 }

View Code

 

参考文章列表：

1. http://curl.haxx.se/libcurl/
2. http://blog.csdn.net/balderfan/article/details/7599554