上一篇: docker搭建elk日志收集系统
非系统数据或者离线日志进行分析,可以采用端口导入elk方式来实现
测试数据
2020-06-27 06:19:04.856 [] [INFO] [main] [com.lyf.BaseApplication:50] - Starting BaseApplication on NVX1F8W7MJSQCJI with PID 11448 (E:\xianghaizing\SpringbootBase\target\classes started by Administrator in E:\xianghaizing\SpringbootBase)
2020-06-27 06:19:04.863 [] [INFO] [main] [com.lyf.BaseApplication:675] - No active profile set, falling back to default profiles: default
2020-06-27 06:19:06.141 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:244] - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-06-27 06:19:06.144 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:126] - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-06-27 06:19:06.236 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:182] - Finished Spring Data repository scanning in 84ms. Found 1 repository interfaces.
2020-06-27 06:19:06.251 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:244] - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-06-27 06:19:06.252 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:126] - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-06-27 06:19:06.279 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationExtensionSupport:363] - Spring Data Redis - Could not safely identify store assignment for repository candidate interface com.lyf.dao.ElasticRepository.
2020-06-27 06:19:06.279 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:182] - Finished Spring Data repository scanning in 13ms. Found 0 repository interfaces.
2020-06-27 06:19:06.416 [] [WARN] [main] [org.mybatis.spring.mapper.ClassPathMapperScanner:44] - Skipping MapperFactoryBean with name 'elasticRepository' and 'com.lyf.dao.ElasticRepository' mapperInterface. Bean already defined with the same name!
2020-06-27 06:19:06.847 [] [INFO] [main] [org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker:330] - Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$81d28fde] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-06-27 06:19:06.992 [] [INFO] [main] [org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker:330] - Bean 'redisConfig' of type [com.lyf.redis.RedisConfig$$EnhancerBySpringCGLIB$$aac139eb] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-06-27 06:19:07.651 [] [INFO] [main] [org.springframework.boot.web.embedded.tomcat.TomcatWebServer:90] - Tomcat initialized with port(s): 8765 (http)
2020-06-27 06:19:07.666 [] [INFO] [main] [org.apache.coyote.http11.Http11NioProtocol:173] - Initializing ProtocolHandler ["http-nio-8765"]
2020-06-27 06:19:07.676 [] [INFO] [main] [org.apache.catalina.core.StandardService:173] - Starting service [Tomcat]
2020-06-27 06:19:07.676 [] [INFO] [main] [org.apache.catalina.core.StandardEngine:173] - Starting Servlet engine: [Apache Tomcat/9.0.17]
2020-06-27 06:19:07.843 [] [INFO] [main] [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173] - Initializing Spring embedded WebApplicationContext
2020-06-27 06:19:07.843 [] [INFO] [main] [org.springframework.web.context.ContextLoader:296] - Root WebApplicationContext: initialization completed in 2851 ms
2020-06-27 06:19:09.878 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:190] - no modules loaded
2020-06-27 06:19:09.880 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2020-06-27 06:19:09.881 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2020-06-27 06:19:09.882 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2020-06-27 06:19:09.883 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2020-06-27 06:19:09.884 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020-06-27 06:19:13.168 [] [INFO] [main] [org.springframework.data.elasticsearch.client.TransportClientFactoryBean:88] - Adding transport node : 192.168.37.190:9300
2020-06-27 06:19:25.891 [] [ERROR] [main] [org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository:91] - failed to load elasticsearch nodes : org.elasticsearch.index.mapper.MapperParsingException: analyzer [ik_max_word] not found for field [title]
2020-06-27 06:19:26.912 [] [INFO] [main] [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor:171] - Initializing ExecutorService 'applicationTaskExecutor'
2020-06-27 06:19:27.100 [] [INFO] [main] [org.springframework.boot.autoconfigure.web.servlet.WelcomePageHandlerMapping:56] - Adding welcome page: class path resource [static/index.html]
2020-06-27 06:19:28.079 [] [INFO] [main] [org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver:59] - Exposing 2 endpoint(s) beneath base path '/actuator'
2020-06-27 06:19:28.240 [] [INFO] [main] [org.springframework.amqp.rabbit.connection.CachingConnectionFactory:460] - Attempting to connect to: [192.168.37.190:5672]
2020-06-27 06:19:28.651 [] [INFO] [main] [org.springframework.amqp.rabbit.connection.CachingConnectionFactory:496] - Created new connection: rabbitConnectionFactory#e784320:0/SimpleConnection@1be4a7e3 [delegate=amqp://[email protected]:5672/, localPort= 57385]
2020-06-27 06:19:28.938 [] [INFO] [main] [org.apache.coyote.http11.Http11NioProtocol:173] - Starting ProtocolHandler ["http-nio-8765"]
2020-06-27 06:19:28.954 [] [INFO] [main] [org.springframework.boot.web.embedded.tomcat.TomcatWebServer:204] - Tomcat started on port(s): 8765 (http) with context path ''
2020-06-27 06:19:28.957 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [main] [com.lyf.BaseApplication:59] - Started BaseApplication in 24.844 seconds (JVM running for 28.197)
2020-06-27 06:19:29.234 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(6)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:110] - UserHikariCP - Starting...
2020-06-27 06:19:29.236 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(8)-127.0.0.1] [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-27 06:19:29.236 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(8)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:524] - Initializing Servlet 'dispatcherServlet'
2020-06-27 06:19:29.254 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(8)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:546] - Completed initialization in 17 ms
2020-06-27 06:19:29.424 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(6)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:123] - UserHikariCP - Start completed.
2020-06-27 06:19:29.546 [670dfbbf-982e-4e10-981a-e1e11fd12262] [WARN] [RMI TCP Connection(6)-127.0.0.1] [org.springframework.boot.actuate.elasticsearch.ElasticsearchHealthIndicator:89] - Elasticsearch health check failed
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:72)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:54)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:44)
at org.springframework.boot.actuate.elasticsearch.ElasticsearchHealthIndicator.doHealthCheck(ElasticsearchHealthIndicator.java:80)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:98)
at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61)
at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:126)
at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:99)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:235)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:69)
at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:70)
... 38 common frames omitted
2020-06-27 06:19:29.960 [] [INFO] [RMI TCP Connection(6)-127.0.0.1] [io.lettuce.core.EpollProvider:104] - Starting without optional epoll library
2020-06-27 06:19:29.961 [] [INFO] [RMI TCP Connection(6)-127.0.0.1] [io.lettuce.core.KqueueProvider:104] - Starting without optional kqueue library
2020-06-27 07:25:25.480 [] [INFO] [main] [org.apache.coyote.http11.Http11NioProtocol:173] - Starting ProtocolHandler ["http-nio-8765"]
2020-06-27 07:25:25.498 [] [INFO] [main] [org.springframework.boot.web.embedded.tomcat.TomcatWebServer:204] - Tomcat started on port(s): 8765 (http) with context path ''
2020-06-27 07:25:25.501 [] [INFO] [main] [com.lyf.BaseApplication:59] - Started BaseApplication in 18.721 seconds (JVM running for 21.761)
2020-06-27 07:25:25.937 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(2)-127.0.0.1] [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-27 07:25:25.937 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:110] - UserHikariCP - Starting...
2020-06-27 07:25:25.938 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(2)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:524] - Initializing Servlet 'dispatcherServlet'
2020-06-27 07:25:25.963 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(2)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:546] - Completed initialization in 25 ms
2020-06-27 07:25:26.209 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:123] - UserHikariCP - Start completed.
2020-06-27 07:25:26.397 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [io.lettuce.core.EpollProvider:104] - Starting without optional epoll library
2020-06-27 07:25:26.400 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [io.lettuce.core.KqueueProvider:104] - Starting without optional kqueue library
logstash配置
input {
tcp {
mode => "server"
host => "0.0.0.0"
port => 4560
codec => json_lines
type => "business"
}
tcp {
mode => "server"
host => "0.0.0.0"
port => 4561
codec => multiline {
pattern => "^2020"
negate => true
what => "previous"
}
type => "debug"
}
}
filter {
if [type] == "debug" {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
}
date {
match => ["logdate", "yyyy-MM-dd HH:mm:ss.SSS"]
}
}
}
output {
elasticsearch {
hosts => ["es:9200"]
action => "index"
codec => json
index => "%{type}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug {
}
}
}
配置简要说明:
codec => multiline {
pattern => "^2"
negate => true
what => "previous"
}
- 使用
4561端口来接收离线数据 multiline设置多行合并,否则异常信息会被分成多行存入pattern设置合并规则,测试数据起始行都是2020,所以设置^20202020开头的为新的一行,否则合并显示!!!
filter {
if [type] == "debug" {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
}
date {
match => ["logdate", "yyyy-MM-dd HH:mm:ss.SSS"]
}
}
}
filter用来处理导入数据的时间戳grok匹配message中的时间,增加logdate字段date匹配logdate重新设置给@timestamp
导入数据
使用nc命令导入数据:
nc 127.0.0.1 4561 < log.log
没有nc,安装一下就行了
yum install -y nc
查看数据
修改显示时间
现在你会发现kibana显示时间与当前时间错8个小时, 不用慌,修改下设置就好!!!
找到高级设置
时区选择utc保存
查看日志时间,现在就一致了
参考:
- https://www.cnblogs.com/yangwenbo214/p/6189622.html
- http://doc.yonyoucloud.com/doc/logstash-best-practice-cn/filter/date.html
- https://www.elastic.co/guide/en/logstash/6.6/plugins-filters-date.html#plugins-filters-date-match
- https://www.cnblogs.com/sunxucool/p/3939701.html