实时xenomai3.0.5/3.0.9+linux4.9.38/4.9.90简要配置及BUG大记实

作为一个linux小白，内核知识也不太懂，但实验室需要为了提高代码实时性，一路下来边踩坑边GOOGLE，现在对前一段经历进行总结。
平台：
硬件：Intel NUC8i7BEH
软件：基于ubuntu16.04 LTS（18.04近期准备测试…）

一、前期操作

安装主要参考链接如下：
1、(Beta) Xenomai 3.0.5 on Ubuntu 14.04/16.04/
2、GitLab_XENOMAI官方网址
3、这个老哥的纪录（有一次跟着官网没成功，跟着这个成功了…简直妙啊）
正常跟着提示安装即可，一定要确保每一步没有奇怪的报错，有的话肯定自己哪里没弄好，趁早停止下来找原因…接下来对几处关键地方进行记录：
1、configure the linux kernel部分：

Recommended options:

* General setup
  --> Local version - append to kernel release: -xenomai-3.0.5 #一定要与自己的xenomai版本对应
  --> Timers subsystem
      --> High Resolution Timer Support (Enable)
* Xenomai/cobalt
  --> Sizes and static limits
    --> Number of registry slots (512 --> 4096)
    --> Size of system heap (Kb) (512 --> 4096)
    --> Size of private heap (Kb) (64 --> 256)
    --> Size of shared heap (Kb) (64 --> 256)
    --> Maximum number of POSIX timers per process (128 --> 512)
  --> Drivers
    --> RTnet
        --> RTnet, TCP/IP socket interface (Enable)
            --> Drivers
                --> New intel(R) PRO/1000 PCIe (Enable)
                --> Realtek 8169 (Enable)
                --> Loopback (Enable)
        --> Add-Ons
            --> Real-Time Capturing Support (Enable)
* Power management and ACPI options
  --> CPU Frequency scaling
      --> CPU Frequency scaling (Disable)
  --> ACPI (Advanced Configuration and Power Interface) Support
      --> Processor (Disable)
  --> CPU Idle
      --> CPU idle PM support (Disable)
* Pocessor type and features
  --> Enable maximum number of SMP processors and NUMA nodes (Disable)
  // Ref : http://xenomai.org/pipermail/xenomai/2017-September/037718.html
  --> Processor family   
  **#执行“cat /proc/cpuinfo | grep family”确定自己应该选Generic-***还是选Core 2/newer** **Xeon**
      --> Core 2/newer Xeon (if "cat /proc/cpuinfo | grep family" returns 6, set as Generic otherwise)
  // Xenomai will issue a warning about CONFIG_MIGRATION, disable those in this order
  --> Transparent Hugepage Support (Disable)
  --> Allow for memory compaction (Disable)
  --> Contiguous Memory Allocation (Disable)
  --> Allow for memory compaction
    --> Page Migration (Disable)
* Device Drivers
  --> Staging drivers
      --> Unisys SPAR driver support
         --> Unisys visorbus driver (Disable)

2、电脑比较渣，后面构建编译过程比较费时间，奥利给！
3、Configure GRUB and reboot：自己配置后的grub文件如下，
一定要根据自己的CPU和GPU配置好其中的“GRUB_CMDLINE_LINUX_DEFAULT=*”。

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

#GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
#GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
#GRUB_CMDLINE_LINUX="locale=en_US"

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 4.9.38-xenomai-3.0.9"
#GRUB_DEFAULT=saved
#GRUB_SAVEDEFAULT=true
# Comment the following lines
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=5
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_rc6=0 i915.enable_dc=0 noapic xenomai.allowed_group=1234"
GRUB_CMDLINE_LINUX=""

4、跟着文档一路下来，最后就能愉快的测试安装咯：希望大家都能正常得到以下提示！

== Sampling period: 100 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT|  00:00:01  (periodic user-mode task, 100 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|      0.174|      0.464|      1.780|       0|     0|      0.174|      1.780
RTD|      0.088|      0.464|      1.357|       0|     0|      0.088|      1.780
RTD|      0.336|      0.464|      1.822|       0|     0|      0.088|      1.822
RTD|      0.342|      0.464|      1.360|       0|     0|      0.088|      1.822
RTD|      0.327|      0.462|      2.297|       0|     0|      0.088|      2.297
RTD|      0.347|      0.463|      1.313|       0|     0|      0.088|      2.297
RTD|      0.314|      0.464|      1.465|       0|     0|      0.088|      2.297
RTD|      0.190|      0.464|      1.311|       0|     0|      0.088|      2.297

二、BUG记录（希望有懂的大佬多多赐教）

1、前期使用xenomai3.0.5+linux4.9.38版本编译安装，虽然能成功，但随后用的时候总是会突然死机，键盘鼠标全都不能用…有时甚至有线无线网都不能正常连接，只能靠无线网卡苟着…
2、有次在运行xeno latency测试安装时候，出现如下报错：

vision@vision:~/xenomai-3.0.5/xenomai-3$ xeno latency
0"000.000| BUG in low_init(): [main] ABI mismatch: required r17, provided r16

官网这里有类似记录，可惜我当时没看懂，大概是说我辛辛苦苦配的linux内核版本过低，有冲突？？
解决办法：故随后换用了xenomai3.0.9+linux4.9.90重新装了一次，这次竟然成功了…很迷…
3、OROCOS RTT on Xenomai：
配置RTT ROS Integration 2.9 on Xenomai期间，执行以下命令时候会卸载之前装好的ROS…真坑，毕竟贫困山区的孩子装一次ROS不容易啊！一定要慎重！

rosdep install --from-paths ~/isir/rtt_ros-2.9_ws/src --ignore-src --rosdistro kinetic -y -r

4、终端terminal打不开
试图根据官网这里配置ROS Kinetic ++，执行以下命令并重启以后，红红的终端突然就没了…慎重啊兄弟们！

sudo locale-gen en_US #warnings might occur
sudo locale-gen en_US.UTF-8
sudo nano /etc/environment
# put theses lines
LANGUAGE=en_US
LC_ALL=en_US
# Reboot !

解决办法：/etc/environment文件中原来是啥就是啥，不懂的话不要轻易加东西（大佬除外…），去掉自己加入的两行语言配置就好了~
5、将实时性要求高的代码放入xenomai的实时进程
期间经常会遇到检索不到某个头文件，一般是路径的问题，没有正确include进来，可以到**/usr/xenomai/include/** 下多看看，xenomai的东西基本都在这里。
若遇到报错error: invalid use of incomplete type ‘const struct timespec’，我当时是把包含这个结构体定义的头文件include进入所需要用到的.cpp文件内即可解决～
6、socket：address family not supported by protocol
这个问题是在测试实时进程和普通进程间的通信时遇到的，参考的官网这里，直接找到路径/usr/xenomai/demo下的可执行文件xddp-label，淦！错误提示如下：

vision@vision:/usr/xenomai/demo$ sudo ./xddp-label 
socket：address family not supported by protocol

解决办法：
回到官网这里的内核配置部分，重新进行以下内核选项配置：

vision@vision:$ cd linux-4.9.38
vision@vision:/linux-4.9.38$ make menuconfig
----------------------------------------------------------------------------
# 更改以下地方配置
* Xenomai/cobalt (NEW) --->
	Drivers --->
		Real-time IPC drivers  --->
			<*> RTIPC protocol family   
				[*]   XDDP cross-domain datagram protocol (NEW) 
				[*]   IDDP intra-domain datagram protocol (NEW) 
				(32)    Number of IDDP communication ports (NEW)
				[*]   Buffer protocol (NEW)
				(32)    Number of BUFP communication ports (NEW) 
-----------------------------------------------------------------------------
# save the new configuration
vision@vision:/linux-4.9.38$ sudo make -j12 && sudo make -j12 modules && sudo make -j12 modules_install && sudo make -j12 install

之后重启系统，在开机时的内核切换界面会看到两个xenomai内核：linux-4.9.90-xenomai-3.0.9（修改后的）和linux-4.9.90-xenomai-3.0.9.old（原先的），选择新生成的内核（linux-4.9.90-xenomai-3.0.9）进入系统即可。
此时，在运行即可出现以下界面，显示实时进程与普通进程之间通信成功：

vision@vision:/usr/xenomai/demo$ sudo ./xddp-label 
realtime_thread2: NRT peer is reading from /dev/rtp0
realtime_thread2: sent 22 bytes, "Surfing With The Alien"
realtime_thread1: "Surfing With The Alien" relayed by peer
realtime_thread2: sent 14 bytes, "Lords of Karma"
realtime_thread1: "Lords of Karma" relayed by peer
realtime_thread2: sent 12 bytes, "Banana Mango"
realtime_thread1: "Banana Mango" relayed by peer
realtime_thread2: sent 13 bytes, "Psycho Monkey"
realtime_thread1: "Psycho Monkey" relayed by peer
realtime_thread2: sent 21 bytes, "Luminous Flesh Giants"
realtime_thread1: "Luminous Flesh Giants" relayed by peer
realtime_thread2: sent 15 bytes, "Moroccan Sunset"
realtime_thread1: "Moroccan Sunset" relayed by peer
realtime_thread2: sent 12 bytes, "Satch Boogie"
realtime_thread1: "Satch Boogie" relayed by peer
realtime_thread2: sent 22 bytes, "Flying In A Blue Dream"
realtime_thread1: "Flying In A Blue Dream" relayed by peer
realtime_thread2: sent 4 bytes, "Ride"
realtime_thread1: "Ride" relayed by peer
realtime_thread2: sent 11 bytes, "Summer Song"
realtime_thread1: "Summer Song" relayed by peer

但终止一次sudo ./xddp-label命令，再次运行时，会出现以下问题，附解决办法：

vision@vision:/usr/xenomai/demo$ sudo ./xddp-label 
[sudo] password for vision: 
bind: File exists
--------------------------------------------------------
# 解决方式：
vision@vision:/usr/xenomai/bin$ ps -A | grep xddp
 2858 ?        00:00:00 xddp-label
vision@vision:/usr/xenomai/bin$ sudo kill -9 2858
---------------------------------------------------------
# 再次运行即可正常通信
vision@vision:/usr/xenomai/demo$ sudo ./xddp-label 
realtime_thread2: NRT peer is reading from /dev/rtp0
realtime_thread2: sent 22 bytes, "Surfing With The Alien"
realtime_thread1: "Surfing With The Alien" relayed by peer

7、error while loading shared libraries: libcobalt.so.2: cannot open shared object file: No such file or directory。
这个问题出现在以下情景：自己尝试通过编写cmakelists.txt和xddp.cpp源文件，然后通过cmake方式进行编译之，从而测试xenomai实时进程和Linux普通进程间的通信情况。
最终发现是由于在cmakelists.txt文件中缺少了libcobalt.so.2动态库文件的引入而导致的，更改后的代码：
（1）xddp.cpp：

/*
 * Copyright (C) 2009 Philippe Gerum .
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA.
 *
 *
 * XDDP-based RT/NRT threads communication demo.
 *
 * Real-time Xenomai threads and regular Linux threads may want to
 * exchange data in a way that does not require the former to leave
 * the real-time domain (i.e. secondary mode). Message pipes - as
 * implemented by the RTDM-based XDDP protocol - are provided for this
 * purpose.
 *
 * On the Linux domain side, pseudo-device files named /dev/rtp
 * give regular POSIX threads access to non real-time communication
 * endpoints, via the standard character-based I/O interface. On the
 * Xenomai domain side, sockets may be bound to XDDP ports, which act
 * as proxies to send and receive data to/from the associated
 * pseudo-device files. Ports and pseudo-device minor numbers are
 * paired, meaning that e.g. port 7 will proxy the traffic for
 * /dev/rtp7. Therefore, port numbers may range from 0 to
 * CONFIG_XENO_OPT_PIPE_NRDEV - 1.
 *
 * All data sent through a bound/connected XDDP socket via sendto(2) or
 * write(2) will be passed to the peer endpoint in the Linux domain,
 * and made available for reading via the standard read(2) system
 * call. Conversely, all data sent using write(2) through the non
 * real-time endpoint will be conveyed to the real-time socket
 * endpoint, and made available to the recvfrom(2) or read(2) system
 * calls.
 *
 * ASCII labels can be attached to bound ports, in order to connect
 * sockets to them in a more descriptive way than using plain numeric
 * port values.
 *
 * The example code below illustrates the following process:
 *
 * realtime_thread1----------------------------->----------+
 *   =>  get socket                                        |
 *   =>  bind socket to port "xddp-demo                    |
 *   =>  read traffic from NRT domain via recvfrom()    <--+--+
 *                                                         |  |
 * realtime_thread2----------------------------------------+  |
 *   =>  get socket                                        |  |
 *   =>  connect socket to port "xddp-demo"                |  |
 *   =>  write traffic to NRT domain via sendto()          v  |
 *                                                         |  ^
 * regular_thread------------------------------------------+  |
 *   =>  open /proc/xenomai/registry/rtipc/xddp/xddp-demo  |  |
 *   =>  read traffic from RT domain via read()            |  |
 *   =>  mirror traffic to RT domain via write()           +--+
 */
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
pthread_t rt1, rt2, nrt;
#define XDDP_PORT_LABEL  "xddp-demo"
static const char *msg[] = {
        "Surfing With The Alien",
        "Lords of Karma",
        "Banana Mango",
        "Psycho Monkey",
        "Luminous Flesh Giants",
        "Moroccan Sunset",
        "Satch Boogie",
        "Flying In A Blue Dream",
        "Ride",
        "Summer Song",
        "Speed Of Light",
        "Crystal Planet",
        "Raspberry Jam Delta-V",
        "Champagne?",
        "Clouds Race Across The Sky",
        "Engines Of Creation"
};
static void fail(const char *reason)
{
        perror(reason);
        exit(EXIT_FAILURE);
}
static void *realtime_thread1(void *arg)
{
        struct rtipc_port_label plabel;
        struct sockaddr_ipc saddr;
        char buf[128];
        int ret, s;
        /*
         * Get a datagram socket to bind to the RT endpoint. Each
         * endpoint is represented by a port number within the XDDP
         * protocol namespace.
         */
        s = socket(AF_RTIPC, SOCK_DGRAM, IPCPROTO_XDDP);
        if (s < 0) {
                perror("socket");
                exit(EXIT_FAILURE);
        }
        /*
         * Set a port label. This name will be registered when
         * binding, in addition to the port number (if given).
         */
        strcpy(plabel.label, XDDP_PORT_LABEL);
        ret = setsockopt(s, SOL_XDDP, XDDP_LABEL,
                         &plabel, sizeof(plabel));
        if (ret)
                fail("setsockopt");
        /*
         * Bind the socket to the port, to setup a proxy to channel
         * traffic to/from the Linux domain. Assign that port a label,
         * so that peers may use a descriptive information to locate
         * it. For instance, the pseudo-device matching our RT
         * endpoint will appear as
         * /proc/xenomai/registry/rtipc/xddp/ in the
         * Linux domain, once the socket is bound.
         *
         * saddr.sipc_port specifies the port number to use. If -1 is
         * passed, the XDDP driver will auto-select an idle port.
         */
        memset(&saddr, 0, sizeof(saddr));
        saddr.sipc_family = AF_RTIPC;
        saddr.sipc_port = -1;
        ret = bind(s, (struct sockaddr *)&saddr, sizeof(saddr));
        if (ret)
                fail("bind");
        for (;;) {
                /* Get packets relayed by the regular thread */
                ret = recvfrom(s, buf, sizeof(buf), 0, NULL, 0);
                if (ret <= 0)
                        fail("recvfrom");
                printf("%s: \"%.*s\" relayed by peer\n", __FUNCTION__, ret, buf);
        }
        return NULL;
}
static void *realtime_thread2(void *arg)
{
        struct rtipc_port_label plabel;
        struct sockaddr_ipc saddr;
        int ret, s, n = 0, len;
        struct timespec ts;
        struct timeval tv;
        socklen_t addrlen;
        s = socket(AF_RTIPC, SOCK_DGRAM, IPCPROTO_XDDP);
        if (s < 0) {
                perror("socket");
                exit(EXIT_FAILURE);
        }
        /*
         * Set the socket timeout; it will apply when attempting to
         * connect to a labeled port, and to recvfrom() calls.  The
         * following setup tells the XDDP driver to wait for at most
         * one second until a socket is bound to a port using the same
         * label, or return with a timeout error.
         */
        tv.tv_sec = 1;
        tv.tv_usec = 0;
        ret = setsockopt(s, SOL_SOCKET, SO_RCVTIMEO,
                         &tv, sizeof(tv));
        if (ret)
                fail("setsockopt");
        /*
         * Set a port label. This name will be used to find the peer
         * when connecting, instead of the port number.
         */
        strcpy(plabel.label, XDDP_PORT_LABEL);
        ret = setsockopt(s, SOL_XDDP, XDDP_LABEL,
                         &plabel, sizeof(plabel));
        if (ret)
                fail("setsockopt");
        memset(&saddr, 0, sizeof(saddr));
        saddr.sipc_family = AF_RTIPC;
        saddr.sipc_port = -1;   /* Tell XDDP to search by label. */
        ret = connect(s, (struct sockaddr *)&saddr, sizeof(saddr));
        if (ret)
                fail("connect");
        /*
         * We succeeded in making the port our default destination
         * address by using its label, but we don't know its actual
         * port number yet. Use getpeername() to retrieve it.
         */
        addrlen = sizeof(saddr);
        ret = getpeername(s, (struct sockaddr *)&saddr, &addrlen);
        if (ret || addrlen != sizeof(saddr))
                fail("getpeername");
        printf("%s: NRT peer is reading from /dev/rtp%d\n",
               __FUNCTION__, saddr.sipc_port);
        for (;;) {
                len = strlen(msg[n]);
                /*
                 * Send a datagram to the NRT endpoint via the proxy.
                 * We may pass a NULL destination address, since the
                 * socket was successfully assigned the proper default
                 * address via connect(2).
                 */
                ret = sendto(s, msg[n], len, 0, NULL, 0);
                if (ret != len)
                        fail("sendto");
                printf("%s: sent %d bytes, \"%.*s\"\n",
                       __FUNCTION__, ret, ret, msg[n]);
                n = (n + 1) % (sizeof(msg) / sizeof(msg[0]));
                /*
                 * We run in full real-time mode (i.e. primary mode),
                 * so we have to let the system breathe between two
                 * iterations.
                 */
                ts.tv_sec = 0;
                ts.tv_nsec = 500000000; /* 500 ms */
                clock_nanosleep(CLOCK_REALTIME, 0, &ts, NULL);
        }
        return NULL;
}
static void *regular_thread(void *arg)
{
        char buf[128], *devname;
        int fd, ret;
        if (asprintf(&devname,
                     "/proc/xenomai/registry/rtipc/xddp/%s",
                     XDDP_PORT_LABEL) < 0)
                fail("asprintf");
        fd = open(devname, O_RDWR);
        free(devname);
        if (fd < 0)
                fail("open");
        for (;;) {
                /* Get the next message from realtime_thread2. */
                ret = read(fd, buf, sizeof(buf));
                if (ret <= 0)
                        fail("read");
                /* Relay the message to realtime_thread1. */
                ret = write(fd, buf, ret);
                if (ret <= 0)
                        fail("write");
        }
        return NULL;
}
int main(int argc, char **argv)
{
        struct sched_param rtparam = { .sched_priority = 42 };
        pthread_attr_t rtattr, regattr;
        sigset_t set;
        int sig;
        sigemptyset(&set);
        sigaddset(&set, SIGINT);
        sigaddset(&set, SIGTERM);
        sigaddset(&set, SIGHUP);
        pthread_sigmask(SIG_BLOCK, &set, NULL);
        pthread_attr_init(&rtattr);
        pthread_attr_setdetachstate(&rtattr, PTHREAD_CREATE_JOINABLE);
        pthread_attr_setinheritsched(&rtattr, PTHREAD_EXPLICIT_SCHED);
        pthread_attr_setschedpolicy(&rtattr, SCHED_FIFO);
        pthread_attr_setschedparam(&rtattr, &rtparam);
        /* Both real-time threads have the same attribute set. */
        errno = pthread_create(&rt1, &rtattr, &realtime_thread1, NULL);
        if (errno)
                fail("pthread_create");
        errno = pthread_create(&rt2, &rtattr, &realtime_thread2, NULL);
        if (errno)
                fail("pthread_create");
        pthread_attr_init(&regattr);
        pthread_attr_setdetachstate(&regattr, PTHREAD_CREATE_JOINABLE);
        pthread_attr_setinheritsched(&regattr, PTHREAD_EXPLICIT_SCHED);
        pthread_attr_setschedpolicy(&regattr, SCHED_OTHER);
        errno = pthread_create(&nrt, &regattr, &regular_thread, NULL);
        if (errno)
                fail("pthread_create");
        sigwait(&set, &sig);
        pthread_cancel(rt1);
        pthread_cancel(rt2);
        pthread_cancel(nrt);
        pthread_join(rt1, NULL);
        pthread_join(rt2, NULL);
        pthread_join(nrt, NULL);
        return 0;
}

（2）CMakeLists.txt：

cmake_minimum_required(VERSION 3.7.2)
project(xddp)
#file(GLOB SOURCES "/usr/xenomai/include/alchemy/*.*")
# set(CMAKE_CXX_STANDARD 11)

#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
add_compile_options(-std=c++11)

include_directories(/usr/xenomai/include/alchemy/)
include_directories(/usr/xenomai/include/)

include_directories(/usr/xenomai/include/rtdm)

include_directories(/usr/xenomai/cobalt/)
include_directories(/usr/xenomai/posix/)
include_directories(/usr/xenomai/boilerplate/)

link_directories(/usr/xenomai/lib)



set(xeno_cflags_params "--skin=posix" "--cflags")
execute_process(
    COMMAND xeno-config ${xeno_cflags_params}
    OUTPUT_VARIABLE xeno_cflags
    OUTPUT_STRIP_TRAILING_WHITESPACE)

set(xeno_ldflags_params "--skin=posix" "--ldflags")
execute_process(
    COMMAND xeno-config ${xeno_ldflags_params}
    OUTPUT_VARIABLE xeno_ldflags
    OUTPUT_STRIP_TRAILING_WHITESPACE)

# Compiler and linker options
set(CMAKE_C_FLAGS          "${CMAKE_CXX_FLAGS} ${xeno_cflags}")
set(CMAKE_CXX_FLAGS        "${CMAKE_CXX_FLAGS} ${xeno_cflags}")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${xeno_ldflags}")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${xeno_ldflags}")


add_executable(xddp xddp.cpp )
target_link_libraries(xddp ${XENO_NATIVE_LDFLAGS} /usr/xenomai/bin)

#target_link_libraries(xddp ${LIB_DIR}/libalchemy.so  ${LIB_DIR}/libanalogy.so 
 #                      ${LIB_DIR}/libcobalt.so ${LIB_DIR}/libmodechk.so ${LIB_DIR}/libcopperplate.so 
 #                      ${LIB_DIR}/libnative.so ${LIB_DIR}/libpsos.so ${LIB_DIR}/libpthread_rt.so 
 #                            ${LIB_DIR}/librtdm.so   ${LIB_DIR}/libsmokey.so ${LIB_DIR}/libtrank.so 
 #                            ${LIB_DIR}/libuitron.so ${LIB_DIR}/libvrtx.so ${LIB_DIR}/libvxworks.so ${LIB_DIR}/libxenomai.so) 
#