Java异步日志使用介绍

近期开发了一个API供第三方使用接入开放平台。第三方在做压力测试时，发现平均响应时间在1m以上，不能满足他们的TPS。

在关键处理环节加上了日志输出。本人使用的日志实现是logback，经过实际测试比log4j性能要好。通过观察日志，发现代码执行比较慢，但是非常随机，有时候从一个方法进入之前和进入之后都需要花费100ms。为什么会这样呢？

经过分析，定位到是logback同步日志输出在压力比较大的时候占用了执行时间。后来在网上查询是否有logback异步日志使用的样例，通过Baidu没有搜到，后来到了logback官网找到了异步日志的实现。

经过时间测试，在压力测试（10000人登陆，TPS2000，平均响应时间100ms），持续10分钟压测，异步日志产生日志4G数据量但几乎没有对性能造成太大的影响。

 

AsyncAppender介绍使用现摘录如下：

URL：http://logback.qos.ch/manual/appenders.html#AsyncAppender

 

AsyncAppender logs ILoggingEvents asynchronously. It acts solely as an event dispatcher and must therefore reference another appender in order to do anything useful.

Lossy by default if 80% full AsyncAppender buffers events in a BlockingQueue. A worker thread created by AsyncAppender takes events from the head of the queue, and dispatches them to the single appender attached to AsyncAppender. Note that by default, AsyncAppender will drop events of level TRACE, DEBUG and INFO if its queue is 80% full. This strategy has an amazingly favorable effect on performance at the cost of event loss.

Application stop/redeploy Upon application shutdown or redeploy, AsyncAppender must be stopped in order to stop and reclaim the worker thread and to flush the logging events from the queue. This can be achieved by stopping the LoggerContext which will close all appenders, including any AsyncAppender instances.

Here is the list of properties admitted by AsyncAppender:

Property Name	Type	Description
queueSize	int	The maximum capacity of the blocking queue. By default, queueSize is set to 256.
discardingThreshold	int	By default, when the blocking queue has 20% capacity remaining, it will drop events of level TRACE, DEBUG and INFO, keeping only events of level WARN and ERROR. To keep all events, set discardingThreshold to 0.
includeCallerData	boolean	Extracting caller data can be rather expensive. To improve performance, by default, caller data associated with an event is not extracted when the event added to the event queue. By default, only "cheap" data like the thread name and the MDC are copied. You can direct this appender to include caller data by setting the includeCallerData property to true.

By default, event queue is configured with a maximum capacity of 256 events. If the queue is filled up, then application threads are blocked from logging new events until the worker thread has had a chance to dispatch one or more events. When the queue is no longer at its maximum capacity, application threads are able to start logging events once again. Asynchronous logging therefore becomes pseudo-synchronous when the appender is operating at or near the capacity of its event buffer. This is not necessarily a bad thing. The appender is designed to allow the application to keep on running, albeit taking slightly more time to log events until the pressure on the appenders buffer eases.

Optimally tuning the size of the appenders event queue for maximum application throughput depends upon several factors. Any or all of the following factors are likely to cause pseudo-synchronous behavior to be exhibited:

• Large numbers of application threads
• Large numbers of logging events per application call
• Large amounts of data per logging event
• High latency of child appenders

To keep things moving, increasing the size of the queue will generally help, at the expense of heap available to the application.

Lossy behavior In light of the discussion above and in order to reduce blocking, by default, when less than 20% of the queue capacilty remains, AsyncAppender will drop events of level TRACE, DEBUG and INFO keeping only events of level WARN and ERROR. This strategy ensures non-blocking handling of logging events (hence excellent performance) at the cost loosing events of level TRACE, DEBUG and INFO when the queue has less than 20% capacity. Event loss can be prevented by setting the discardingThreshold property to 0 (zero).

Example: AsyncAppender configuration (logback-examples/src/main/java/chapters/appenders/conc/logback-async.xml)

View as .groovy

  name="FILE"class="ch.qos.logback.core.FileAppender">
    myapp.log
    
      %logger{35} - %msg%n
    
  
  name="ASYNC"class="ch.qos.logback.classic.AsyncAppender">
    ref="FILE"/>
  
  level="DEBUG">
    ref="ASYNC"/>