日志收集rsyslog+es+kibana，微服务APM管理Skywalking全Docker实践

经过某人@Anoyi 点拨一二

日志收集篇

1.本地测试环境:

Linux version 4.15.0-47-generic (buildd@lgw01-amd64-001) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3))
rsyslogd 8.32.0

2. Spring Boot里面配置Logback.xml

        139.24.217.xxx
        SYSLOG
        Mindgate:%date{"yyyy-MM-dd'T'HH:mm:ss,SSSXXX",UTC} %-5level [%thread] %logger{36} - %msg%n
    

3. 服务器上设置rsyslog.conf来收集日志

sudo vim rsyslog.conf

日志除了本地可以保存还可以发往elasticsearch保存，在rsyslog.conf 中加入module(load="omelasticsearch")，然后安装rsyslog-es的插件，才可以起作用。

sudo apt-get install rsyslog-elasticsearch

# provides UDP syslog reception
module(load="imudp")
module(load="omelasticsearch")
input(type="imudp" port="514" Ruleset="udpinput")

template(name="udppfile" type="string" string="/home/ubuntu/git/logs/%HOSTNAME%/%PROGRAMNAME%/%$year%-%$month%-%$day%.log")
template(name="timemsg" type="string" string="%timereported% %msg%\n")
template(name="gwelastic" type="list" option.json="on"){
constant(value="{")
constant(value="\"FACILITY\":\"")    property(name="syslogfacility-text")
constant(value="\",\"TAG\":\"")         property(name="syslogtag")
constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
constant(value="\",\"HOSTNAME\":\"") property(name="hostname")
constant(value="\",\"TIME\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"MSG\":\"") property(name="msg")
constant(value="\"}")

template(name="mindgate-index" type="list") {
    constant(value="mindgate-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

ruleset(name="udpinput"){
action(type="omfile" dynaFile="udppfile" template="gwelastic")
action(type="omelasticsearch" server="127.0.0.1" serverport="9200" template="gwelastic" searchIndex="mindgate-index" dynSearchIndex="on")

最后要重启下rsyslog服务:

service rsyslog restart

4. Elasticsearch Kibana Docker 安装(参考elastic官网)

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elasticsearch docker.elastic.co/elasticsearch/elasticsearch:6.7.1
docker run --link elasticsearch:elasticsearch -p 5601:5601 -e "elasticsearch.hosts=http://elasticsearch:9200" --name kibana docker.elastic.co/kibana/kibana:6.7.1

5. 遇到的坑

所以才在logback,xml文件里配置了mindgate:.png

APM：Skywalking全链路管理篇

1. 服务端Docker安装

git clone -b v6.0.0-GA https://github.com/apache/skywalking.git

进入到docker文件夹修改docker-compose.yml. 如下：
Note: 这里我去掉了默认的elasticsearch的配置，因为上面在做日志收集篇的时候我就已经安装了es，所以这里不用再安装和启动es的docker镜像了。如果你本地没有es，还是需要安装默认配置安装es(verison: 6.x)

youjia@cdi6s003:~/git/skywalking/docker$ cat docker-compose.yml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

version: '3.3'
services:
  oap:
    image: apache/skywalking-oap-server:latest
    container_name: sw-oap
    restart: always
    ports:
      - 11800:11800
      - 12800:12800
    volumes:
      - ./config:/skywalking/config:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
  ui:
    image: apache/skywalking-ui:latest
    container_name: sw-ui
    depends_on:
      - oap
    links:
      - oap
    restart: always
    ports:
      - 8080:8080
    environment:
      collector.ribbon.listOfServers: oap:12800

启动之前，需要到docker/config文件下修改application.yml文件，配置好es的ip地址。

storage:
  elasticsearch:
  # set the namespace in elasticsearch
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:139.24.217.xxx:9200}

启动命令，然后在浏览器输入http://yourip:8080，输入用户名/密码: Admin/Admin, 即可看到一个全部服务显示为0的页面。

docker-compose up -d

2. 客户端Java Agent安装

首先到skywalking的官网上下载agent
http://skywalking.apache.org/downloads/， 然后修改agent.conf文件。

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# The agent namespace
# agent.namespace=${SW_AGENT_NAMESPACE:default-namespace}

# The service name in UI
agent.service_name=${SW_AGENT_NAME:app-A}

# The number of sampled traces per 3 seconds
# Negative number means sample traces as many as possible, most likely 100%
agent.sample_n_per_3_secs=${SW_AGENT_SAMPLE:-1}

# Authentication active is based on backend setting, see application.yml for more details.
# agent.authentication = ${SW_AGENT_AUTHENTICATION:xxxx}

# The max amount of spans in a single segment.
# Through this config item, skywalking keep your application memory cost estimated.
# agent.span_limit_per_segment=${SW_AGENT_SPAN_LIMIT:300}

# Ignore the segments if their operation names start with these suffix.
# agent.ignore_suffix=${SW_AGENT_IGNORE_SUFFIX:.jpg,.jpeg,.js,.css,.png,.bmp,.gif,.ico,.mp3,.mp4,.html,.svg}

# If true, skywalking agent will save all instrumented classes files in `/debugging` folder.
# Skywalking team may ask for these files in order to resolve compatible problem.
# agent.is_open_debugging_class = ${SW_AGENT_OPEN_DEBUG:true}

# Backend service addresses.
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:139.24.217.xxx:11800}

# Logging level
logging.level=${SW_LOGGING_LEVEL:DEBUG}

主要是修改ip地址为服务端的ip地址，保证client采集到的数据可以送往server。

3. 启动客户端java程序

可以是IDEA启动，也可以是命令行启动。譬如：

java -javaagent:/path/to/your/apache-skywalking-apm-incubating/agent/skywalking-agent.jar=agent.service_name=yourAppMindgate -jar mindgate-1.11.3.jar

然后就很容易在刚刚的浏览器上看到Service已经从0 变成了 1了。

最后Docker用好，还是太方便了。
以上。