st(state-threads) coroutine和setjmp/longjmp的关系
st(state-threads) https://github.com/winlinvip/state-threads
以及基于st的RTMP/HLS服务器：https://github.com/winlinvip/simple-rtmp-server
st是实现了coroutine的一套机制，即用户态线程，或者叫做协程。将epoll（async，nonblocking socket）的非阻塞变成协程的方式，将所有状态空间都放到stack中，避免异步的大循环和状态空间的判断。
关于st的详细介绍，参考翻译：http://blog.csdn.net/win_lin/article/details/8242653
本文主要介绍了coroutine基于setjmp和longjmp的实现机制。
我将st进行了简化，去掉了其他系统，只考虑linux系统，以及i386/x86_64/arm/mips四种cpu系列，参考：https://github.com/winlinvip/simple-rtmp-server/tree/master/trunk/research/st
st最关键的地方在于需要重新分配stack，譬如在heap分配stack，支持超大并发。
mips和arm可以直接设置stack；而i386和x86_64的CPU体系在glibc2.4以上为了安全性考虑，jmp_buf的结构不是那么清楚，直接设置jmp_buf的sp是不可行的：
/*
* Starting with glibc 2.4, JB_SP definitions are not public anymore.
* They, however, can still be found in glibc source tree in
* architecture-specific "jmpbuf-offsets.h" files.
* Most importantly, the content of jmp_buf is mangled by setjmp to make
* it completely opaque (the mangling can be disabled by setting the
* LD_POINTER_GUARD environment variable before application execution).
* Therefore we will use built-in _st_md_cxt_save/_st_md_cxt_restore
* functions as a setjmp/longjmp replacement wherever they are available
* unless USE_LIBC_SETJMP is defined.
*/可以参考glic的代码：http://ftp.gnu.org/gnu/glibc/
MIPS
这种最简单，实际上setjmp的jmp_buf提供了sp和pc，只需要把jmp_buf的sp设置为分配的stack，把pc设置为main地址就可以。
#if defined(__mips__)
#define MD_STACK_GROWS_DOWN
#define MD_INIT_CONTEXT(_thread, _sp, _main) \
ST_BEGIN_MACRO \
MD_SETJMP((_thread)->context); \
_thread->context[0].__jmpbuf[0].__pc = (__ptr_t) _main; \
_thread->context[0].__jmpbuf[0].__sp = _sp; \
ST_END_MACROARM
ARM实际上st也是用的glibc的setjmp和longjmp，明显arm的glibc的jmp_buf是结构可知的，参考arm的setjmp头文件：
/**
/usr/arm-linux-gnueabi/include/bits/setjmp.h
#ifndef _ASM
The exact set of registers saved may depend on the particular core
in use, as some coprocessor registers may need to be saved. The C
Library ABI requires that the buffer be 8-byte aligned, and
recommends that the buffer contain 64 words. The first 28 words
are occupied by v1-v6, sl, fp, sp, pc, d8-d15, and fpscr. (Note
that d8-15 require 17 words, due to the use of fstmx.)
typedef int __jmp_buf[64] __attribute__((__aligned__ (8)));
the layout of setjmp for arm:
0-5: v1-v6
6: sl
7: fp
8: sp
9: pc
10-26: d8-d15 17words
27: fpscr
*/
/**
For example, on raspberry-pi, armv6 cpu:
(gdb) x /64 env_func1[0].__jmpbuf
v1, 0: 0x00 0x00 0x00 0x00
v2, 1: 0x00 0x00 0x00 0x00
v3, 2: 0x2c 0x84 0x00 0x00
v4, 3: 0x00 0x00 0x00 0x00
v5, 4: 0x00 0x00 0x00 0x00
v6, 5: 0x00 0x00 0x00 0x00
sl, 6: 0x00 0xf0 0xff 0xb6
fp, 7: 0x9c 0xfb 0xff 0xbe
sp, 8: 0x88 0xfb 0xff 0xbe
pc, 9: 0x08 0x85 0x00 0x00
(gdb) p /x $sp
$5 = 0xbefffb88
(gdb) p /x $pc
$4 = 0x850c
*/
调试的结果显示也是没有问题。st需要修改下JB_RSP，从20改成8，参考：https://github.com/winlinvip/simple-rtmp-server/issues/190
i386/x86_64
st在i386和x86_64下面，都定义了宏MD_USE_BUILTIN_SETJMP，也就是用st自己的md.S里面的setjmp和longjmp：
/*
* Starting with glibc 2.4, JB_SP definitions are not public anymore.
* They, however, can still be found in glibc source tree in
* architecture-specific "jmpbuf-offsets.h" files.
* Most importantly, the content of jmp_buf is mangled by setjmp to make
* it completely opaque (the mangling can be disabled by setting the
* LD_POINTER_GUARD environment variable before application execution).
* Therefore we will use built-in _st_md_cxt_save/_st_md_cxt_restore
* functions as a setjmp/longjmp replacement wherever they are available
* unless USE_LIBC_SETJMP is defined.
*/
#if defined(__i386__)
#define MD_STACK_GROWS_DOWN
#define MD_USE_BUILTIN_SETJMP
#if defined(__GLIBC__) && __GLIBC__ >= 2
#ifndef JB_SP
#define JB_SP 4
#endif
#define MD_GET_SP(_t) (_t)->context[0].__jmpbuf[JB_SP]
#else
/* not an error but certainly cause for caution */
#error "Untested use of old glibc on i386"
#define MD_GET_SP(_t) (_t)->context[0].__jmpbuf[0].__sp
#endif
#elif defined(__amd64__) || defined(__x86_64__)
#define MD_STACK_GROWS_DOWN
#define MD_USE_BUILTIN_SETJMP
#ifndef JB_RSP
#define JB_RSP 6
#endif
#define MD_GET_SP(_t) (_t)->context[0].__jmpbuf[JB_RSP]原因讲得很清楚，glibc2.4以上的jmp_buf的sp不能操作了，导致只能用st内建的setjmp和longjmp。
若使用glibc的setjmp和longjmp，即定义宏（参考下一章：ST宏定义）USE_LIBC_SETJMP，则出现segmentfault，gdb调试setjmp的jmp_buf：
(gdb) x /64xb env_func1[0].__jmpbuf
0x600ca0 : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x600ca8 : 0xf8 0xc1 0x71 0xe5 0xa8 0x88 0xb4 0x15
0x600cb0 : 0xa0 0x05 0x40 0x00 0x00 0x00 0x00 0x00
0x600cb8 : 0x90 0xe4 0xff 0xff 0xff 0x7f 0x00 0x00
0x600cc0 : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x600cc8 : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x600cd0 : 0xf8 0xc1 0x51 0xe5 0xa8 0x88 0xb4 0x15
0x600cd8 : 0xf8 0xc1 0xd9 0x2f 0xd7 0x77 0x4b 0xea
(gdb) p /x $sp
$4 = 0x7fffffffe380
明显可以看到，jmp_buf中没有一个数据是和sp能对应上的，所以st在setjmp后设置sp就会有问题。
ST宏定义
s在make时可以定义宏，指定EXTRA_CFLAGS参数即可，参考说明：
##########################
# Other possible defines:
# To use poll(2) instead of select(2) for events checking:
# DEFINES += -DUSE_POLL
# You may prefer to use select for applications that have many threads
# using one file descriptor, and poll for applications that have many
# different file descriptors. With USE_POLL poll() is called with at
# least one pollfd per I/O-blocked thread, so 1000 threads sharing one
# descriptor will poll 1000 identical pollfds and select would be more
# efficient. But if the threads all use different descriptors poll()
# may be better depending on your operating system's implementation of
# poll and select. Really, it's up to you. Oh, and on some platforms
# poll() fails with more than a few dozen descriptors.
#
# Some platforms allow to define FD_SETSIZE (if select() is used), e.g.:
# DEFINES += -DFD_SETSIZE=4096
#
# To use malloc(3) instead of mmap(2) for stack allocation:
# DEFINES += -DMALLOC_STACK
#
# To provision more than the default 16 thread-specific-data keys
# (but not too many!):
# DEFINES += -DST_KEYS_MAX=
#
# To start with more than the default 64 initial pollfd slots
# (but the table grows dynamically anyway):
# DEFINES += -DST_MIN_POLLFDS_SIZE=
#
# Note that you can also add these defines by specifying them as
# make/gmake arguments (without editing this Makefile). For example:
#
# make EXTRA_CFLAGS=-DUSE_POLL
#
# (replace make with gmake if needed).
#
# You can also modify the default selection of an alternative event
# notification mechanism. E.g., to enable kqueue(2) support (if it's not
# enabled by default):
#
# gmake EXTRA_CFLAGS=-DMD_HAVE_KQUEUE
#
# or to disable default epoll(4) support:
#
# make EXTRA_CFLAGS=-UMD_HAVE_EPOLL
#
########################## 譬如，st默认使用mmap分配栈，估计是为了提高性能，可以让st在heap分配栈，这样可以支持几百万个线程。编译时定义宏MALLOC_STACK：
make linux-debug EXTRA_CFLAGS="-DMALLOC_STACK"或者在Makefile中改变默认的DEFINES也可以。
SETJMP和LONGJMP
看完了st的线程调度和生命周期，我才完全明白setjmp和longjmp的栈切换方式，以及st使用自己分配的stack到底如何切换。参考：http://blog.csdn.net/win_lin/article/details/40978665
考虑一个单线程程序，实际上程序是流水线执行的，也就是从main开始执行，进入各种子函数然后退出。参考：https://github.com/winlinvip/simple-rtmp-server/blob/master/trunk/research/arm/jmp_flow.cpp
/*
# for all supports setjmp and longjmp:
g++ -g -O0 -o jmp_flow jmp_flow.cpp
*/
#include
#include
#include
jmp_buf context_level_0;
void func_level_0()
{
const char* level_0_0 = "stack variables for func_level_0";
int ret = setjmp(context_level_0);
printf("func_level_0 ret=%d\n", ret);
if (ret != 0) {
printf("call by longjmp.\n");
exit(0);
}
}
int main(int argc, char** argv)
{
func_level_0();
longjmp(context_level_0, 1);
return 0;
}
调试这个程序：
(gdb) f 0
#0 func_level_0 () at jmp_flow.cpp:16
16 if (ret != 0) {
(gdb) bt
#0 func_level_0 () at jmp_flow.cpp:16
#1 0x0000000000400725 in main (argc=1, argv=0x7fffffffe4b8) at jmp_flow.cpp:24
(gdb) i locals
level_0_0 = 0x400838 "stack variables for func_level_0"
ret = 0在setjmp之后，stack是有效的，上层的变量值也对。而 setjmp只是保存了各种指针，而没有保存完整的stack的拷贝。因此，若setjmp之后返回了，在longjmp回来时整个stack都是破坏的了（很显然）。
(gdb) f 0
#0 func_level_0 () at jmp_flow.cpp:16
16 if (ret != 0) {
(gdb) bt
#0 func_level_0 () at jmp_flow.cpp:16
#1 0x0000000000400734 in main (argc=1, argv=0x7fffffffe4b8) at jmp_flow.cpp:25
(gdb) i locals
level_0_0 = 0x1
ret = 1 原因是这个函数返回后，栈已经释放了，再重新跳到这个地方执行，执行位置（PC）是对的，栈指针也是对的，但是栈的内容肯定是不一样了。
因此，longjmp到某个地方时，这个函数的堆栈实际上无效，访问变量和返回地址也是不可用的，因此longjmp只能在继续longjmp，这也就是为何有_st_thread_main的原因，永远不会从这个函数返回。
或者说，longjmp到某个函数之后，可以调用子函数，但只能通过longjmp来回到这个函数之外的函数。或者说，第一次longjmp的函数（即函数的thread_main），永远不能返回，只能通过longjmp跳转。
或者说，longjmp的目标只能是同一个stack，在不改变sp的情况下。而st那样需要跳来跳去的方式，必须在堆上分配sp，让每个线程私有自己的sp。
不分配stack的段错误
既然longjmp之后不能返回，若再次longjmp到其他的线程，堆栈是公用的，这时候应该会导致堆栈混淆。
查看代码，参考：https://github.com/winlinvip/simple-rtmp-server/blob/master/trunk/research/arm/jmp_2flow.cpp
/*
# for all supports setjmp and longjmp:
g++ -g -O0 -o jmp_2flow jmp_2flow.cpp
*/
#include
#include
#include
jmp_buf context_thread_0;
jmp_buf context_thread_1;
void thread0_functions()
{
int ret = setjmp(context_thread_0);
// when ret is 0, create thread,
// when ret is not 0, longjmp to this thread.
if (ret == 0) {
return;
}
int age = 10000;
const char* name = "winlin";
printf("[thread0] age=%d, name=%s\n", age, name);
if (!setjmp(context_thread_0)) {
printf("[thread0] switch to thread1\n");
longjmp(context_thread_1, 1);
}
// crash, for the stack is modified by thread1.
// name = 0x2b67004009c8
printf("[thread0] terminated, age=%d, name=%s\n", age, name);
exit(0);
}
void thread1_functions()
{
int ret = setjmp(context_thread_1);
// when ret is 0, create thread,
// when ret is not 0, longjmp to this thread.
if (ret == 0) {
return;
}
int age = 11111;
printf("[thread1] age=%d\n", age);
if (!setjmp(context_thread_1)) {
printf("[thread1] switch to thread0\n");
longjmp(context_thread_0, 1);
}
printf("[thread1] terminated, age=%d\n", age);
exit(0);
}
int main(int argc, char** argv)
{
thread0_functions();
thread1_functions();
// kickstart
longjmp(context_thread_0, 1);
return 0;
}
这两个函数虽然longjmp成功，但是stack互相混淆，导致段错误：
Breakpoint 1, thread0_functions () at jmp_2flow.cpp:23
23 printf("[thread0] age=%d, name=%s\n", age, name);
(gdb) i locals
ret = 1
age = 10000
name = 0x400908 "winlin"第一次进入thread0时，堆栈是ok的。然后会跳到thread1，它会进行破坏；最后还是跳到thread0：
Breakpoint 2, thread0_functions () at jmp_2flow.cpp:31
31 printf("[thread0] terminated, age=%d, name=%s\n", age, name);
(gdb) i locals
ret = 1
age = 10000
name = 0x2b6700000001 这个时候从thread1跳回来时，thread0的name完全变了。
因此，longjmp在stack没有在堆开辟时，不能跳转到已经破坏的栈。譬如：
main(setjmp) => func1 => func2 (longjmp to main)
func2若longjmp到main，是没有问题的，这时候func2的栈不可用，但是main的没有破坏。
假设下面的跳转路径：
main => func1 => func2 (setjmp)
=> func3 (longjmp to func2)
func2返回了，然后func3再longjmp到func2时，栈的指针虽然是和func2在setjmp时一样，但是内容已经变了。这个时候就几乎会段错误。
也就是说，stack若不在堆上分配，每个线程有自己的stack时，setjmp的那个函数不能再次longjmp回来，这个时候肯定stack被破坏了。
最后的结论就是，st必须得自己分配stack，每个thread一个stack。