Thread Local Storage Platform Issues and Dbstl's Solutions

1. Overview

Thread local storage(tls) is a feature provided by most modern operating
systems(OS) that allow multiple threads within a process to have its own "global"
data, but the scope of the "global" data is restricted within a thread
itself, i.e. tls variables are thread wide global variables.

TLS can be useful if we want to store global data which are grouped by
threads, each thread only accesses its own piece of data, and have no loss of
concurrency. Without tls, we would have to store such data in a global data
structure (process wide) and use locks to sychronize access to it.


2. Two Ways to Support TLS

The support to tls feature includes two ways: compiler keyword support and
thread library API support.

2.1 Compiler keyword support:

If we declare an integer variable P a tls variable on Windows, we can do it like
this:

__declspec(thread) int P;

Or if we want to define a static data member S of class C as a tls variable so
that each thread access its own copy of S when it accesses C::S, we will
declare it like this:

class C {
public:
static __declspec(thread) int S;
// The rest follow...
};

template <typename T>
class CT {
public:
static __declspec(thread) T* ST;
// The rest follow...
};

And define it like this:

__declspec(thread) int C::S = 0;

template <typename T>
__declspec(thread) T* CT::ST = NULL;


2.2 Portability

Unfortunately, different compilers and OS have different keywords, for MSVC
it is __declspec(thread); for gcc and icpc (intel c++ compiler), it's __thread;
and some other compilers like those on HPUX use __declspec(__thread).

So it takes some effort to
write portable tls code that can run on multiple platforms using multiple
compilers --- we need to provide the correct tls keyword for the target
platform and compiler.

A more terrible issue that makes writing such portable code harder is that
in the above CT<T>::ST example, some
compilers like icpc require the tls keyword only appear in the declaration,
if CT<T>::ST's definition also has the tls keyword, icpc can't build the code.
And having only a tls declaration is enough to make CT<T>::ST a tls variable.
But icpc require tls keyword appear in C::S's definition, strange enough.

However, some other compilers like the older ( < 4.0) versions of gcc require
the tls keyword
appear in both the declaration and definition, otherwise although the build
succeeds, the variable CT<T>::ST or C::S is not handled as a tls variable,
thus multiple thread access will have races and result in undefined behaviors;

Newer versions of gcc will take CT<T>::ST and C::S as a tls variable once it's
declared as tls, whether or not the definition has tls keywords. This is also
true for icpc if the class is not templated. Since icpc is trying to mimic gcc
as much as possible, I think it's icpc's bug that it can't accept a tls
keyword in CT<T>::ST's definition.

A portable implementation in dbstl implementation is to detect appropriate tls declaration and
definition keywords when configuring, using the m4 scripts. We will iterate
through the ("__declspec(thread)", "__thread", "__declspec(__thread)", "")
array for a combination that builds the class C and CT, and exclude the
("", "") combination of course. Also note that the "" must be placed at last
otherwise if the compiler is a older version of gcc, the tls keyword in
definition will be blank, and we will get a non-tls variable instead; when we
apply this technique, we will find icpc end up with no appropriate tls
keyword combinations.



2.3 Thread API Support

Another way to use tls is through thread API. For example, in pthread, we
first create a tls key, then use this key to get/set the thread-specific copy
of the tls variable assocaited with the key for each thread.
We create the tls key via the pthread_key_create call.
And in order to avoid multiple initializations to the same key, we use
pthread_once to make sure the pthread_key_create is called only once.

When we have the tls key, we can call pthread_setspecific to write to this
thread's tls variable, or call pthread_getspecific to read this
thread's tls variable.

Some platforms only has the thread API support, like Mac OSX; For others if they
don't have pthread API, or other thread library that support tls, they will
need to have a tls keyword to support this feature.

2.4 TLS in Dbstl Implementation

In dbstl, we always prepared both choices, and make the choice when
configuring -- if the platform and compiler has tls keyword support, we use it;
otherwise we use pthread tls API; If neither pthread API nor tls keywords
are available, this platform won't be able to use dbstl in multiple threads.

See the $(DB)/stl/dbstl_resource_manager.h/cpp for example code.