使用logstash配置nginx和tomcat日志统一收集到一台日志服务器

相关服务器安装jdk和redis请参考

http://blog.csdn.net/u013619834/article/details/38894649
http://blog.csdn.net/u013619834/article/details/38899405

负责发送日志的所有服务器安装logstash

1.下载安装包
wget https://download.elastic.co/logstash/logstash/logstash-2.4.1.tar.gz


2.安装
tar zxvf logstash-2.4.1.tar.gz
mv logstash-2.4.1 /usr/local/logstash
mkdir -p /usr/local/logstash/etc


3.修改配置文件
vim /usr/local/logstash/etc/logstash.cnf
添加

input {
    file {
        type => "tomcat-catalina"
        path => "/u02/8080-tomcat/logs/catalina.out"
        codec => multiline {
            pattern => "^\s"
            what => "previous"
        }
    }
}


output {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-tomcat-catalina"
    }
}

4.如果系统中没设置环境变量，需要添加JAVA_HOME环境变量
vim /usr/local/logstash/bin/logstash.lib.sh
添加
JAVA_HOME=/usr/java/jdk1.7.0_79

5.启动
mkdir -p /usr/local/logstash/logs
nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/etc/logstash.cnf >> /usr/local/logstash/logs/nohup.out 2>&1 &

6.到redis查看
redis-cli
LPOP "logstash-tomcat-catalina"

负责接收数据的logstash服务器安需要安装logstash，并使用以下配置文件

vim /usr/local/logstash/etc/logstash.cnf
添加

input {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-tomcat-catalina"
    }
}


filter {
    ruby {
        code => "event['filedatetag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
    }

}

output {
    file {
         path => "/data/log/tomcat/%{host}/catalina-%{filedatetag}.log"
         message_format=>"%{host}----%{message}"
    }
    #stdout{
    #    codec=>rubydebug
    #}

    if [message] =~ "Exception" {
        file {
            path => "/data/log/tomcat/exception/exception-%{filedatetag}.log"
        }
    }
}

收集nginx日志的配置文件

客户端上的配置

vim /usr/local/logstash/etc/nginx_log.cnf

input {
    file {
        type => "nginx_access_log"
        path => "/data/logs/nginx/access_log.log"
    }
    file {
        type => "nginx_access"
        path => "/data/logs/nginx/access.log"
    }
    file {
        type => "nginx_access_check"
        path => "/data/logs/nginx/access_check.log"
    }
}


output {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-nginx-log"
    }
}

日志服务器上的配置

vim /usr/local/logstash/etc/nginx_log.cnf

input {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-nginx-log"
    }
}


filter {
    ruby {
        code => "event['filedatetag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
    }

}

output {
    file {
         path => "/data/log/nginx/%{host}/%{type}/%{type}-%{filedatetag}.log"
         message_format=>"%{message}"
    }
}

input {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-nginx-log"
    }
}


filter {
    ruby {
        code => "event['filedatetag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
    }
    json {
        source => "message"
        target => "jsoncontent"
    }

}

output {
    file {
         path => "/data/log/nginx/all/%{type}/%{type}-%{filedatetag}.log"
         message_format=>"%{message}"
         #message_format=>"%{host}----%{type}----%{message}"
    }


    if [message] =~ "code=514" {
        file {
            path => "/data/log/nginx/zabbix_monitor/yunxin-code514.log"
            message_format=>"%{message}"
        }
    }


    exec {
        command => "/usr/local/redis/bin/redis-cli -h 127.0.0.1 incr zabbix_nginx_log_count_%{type}"
    }
    exec {
        command => "/usr/local/redis/bin/redis-cli -h 127.0.0.1 incr zabbix_nginx_log_count_%{type}_%{[jsoncontent][status]}"
    }
    #stdout{
    #    codec=>rubydebug
    #}
}