现象
不定时出现（很难复现）今天的日志输出到了昨天的日志文件中，比如 今天是2018-11-27日，默认的日志输出文件是default.log， 会出现日志打印到了default.log.2018-11-26 然而 default.log却只有当天进行diamond推送的日志
由于应用和diamond进行了对接并和spring进行了集成，所以会一个异步监听diamond推送并对应用中
使用@Value注解的地方进行动态修改， 每次推送事件发生时都会打印日志
- @Slf4j编译后的代码
Logger log = org.slf4j.LoggerFactory.getLogger(Foo.class);
slf4j和logback的绑定和初始化
-
slf4j约定了绑定规范，需要在org.slf4j.impl包下有一个 StaticLoggerBinder implements LoggerFactoryBinder 的类，这个可以是任何日志框架的类，当然我们也可以自己搞一个
- LoggerFactoryBinder的接口定义
public interface LoggerFactoryBinder { /** * Return the instance of {@link ILoggerFactory} that * {@link org.slf4j.LoggerFactory} class should bind to. * * @return the instance of {@link ILoggerFactory} that * {@link org.slf4j.LoggerFactory} class should bind to. */ public ILoggerFactory getLoggerFactory(); /** * The String form of the {@link ILoggerFactory} object that this *LoggerFactoryBinderinstance is intended to return. * *This method allows the developer to interrogate this binder's intention * which may be different from the {@link ILoggerFactory} instance it is able to * yield in practice. The discrepancy should only occur in case of errors. * * @return the class name of the intended {@link ILoggerFactory} instance */ public String getLoggerFactoryClassStr();
}
```
-
当调用org.slf4j.LoggerFactory.getLogger(Foo.class);时，会通过StaticLoggerBinder的getLoggerFactory方法获取日志框架的ILoggerFactory实现，进而获取日志输出对象
- StaticLoggerBinder在类加载的时候会进行日志初始化，包括加载日志配置比如默认的lomback.xml
public class StaticLoggerBinder implements LoggerFactoryBinder { /** * Declare the version of the SLF4J API this implementation is compiled * against. The value of this field is usually modified with each release. */ // to avoid constant folding by the compiler, this field must *not* be final public static String REQUESTED_API_VERSION = "1.7.16"; // !final final static String NULL_CS_URL = CoreConstants.CODES_URL + "#null_CS"; /** * The unique instance of this class. */ // 这里是单例模式这个是slf4j的约定要求单例模式 private static StaticLoggerBinder SINGLETON = new StaticLoggerBinder(); private static Object KEY = new Object(); static { // 这个地方进行初始化，类加载的时候就会触发 SINGLETON.init(); } ... } -
应用通过org.slf4j.LoggerFactory.getLogger(Foo.class);获得了日志框架的Logger实现进行日志打印
问题排查
- 断点logback初始化的地方，StaticLoggerBinder静态语句块
static {
// 这个地方进行初始化，类加载的时候就会触发
SINGLETON.init();
}
-
发现断点被执行了多次
- 静态语句块只有类加载的时候仅且执行一次，这里被执行了多次，怀疑是被加载了多次
- 通过IDEA 的alt + F8，执行this.getClass().getClassLoader().getClass().getName()发现每次都不一样
- 发现应用连接diamond的方法被执行了两次
- 在diamond推送时，同样也会执行两次
- 跟踪发现一个是Pandora调用的，一个是系统启动第一次调用的
- 推送一次，因为有两个链接顾推送事件的日志会被打印两次，查看日志对象Logger发现一个是springboot加载的一个是Pandora加载的
- 在应用启动完毕之后，发现Pandora加载的bean是用的com.taobao.pandora.boot.loader.ReLaunchURLClassLoader
- springboot加载用的是org.springframework.boot.loader.LaunchedURLClassLoader
-
此时已经确定系统有两套独立的日志输出入口，并打印到同一个日志文件
logback源码跟踪
应用appender配置
${LOG_PATH}/txp-default.log
${FILE_LOG_PATTERN}
utf8
true
${LOG_PATH}/default.log.%d{yyyy-MM-dd}
15
INFO
源码追踪到日志跨天翻滚的地方
public class RollingFileAppender extends FileAppender {
...
@Override
protected void subAppend(E event) {
// The roll-over check must precede actual writing. This is the
// only correct behavior for time driven triggers.
// We need to synchronize on triggeringPolicy so that only one rollover
// occurs at a time
synchronized (triggeringPolicy) {
// 这个地方判断是否满足日志翻滚条件
// 我们的应用配置是按天输出日志，所以这里会进行跨天判断
if (triggeringPolicy.isTriggeringEvent(currentlyActiveFile, event)) {
rollover(); // 这个地方进行日志翻转
}
}
super.subAppend(event);
}
...
}
跨天判断的代码片
@NoAutoStart
public class DefaultTimeBasedFileNamingAndTriggeringPolicy extends TimeBasedFileNamingAndTriggeringPolicyBase {
public boolean isTriggeringEvent(File activeFile, final E event) {
// 获得当前时间
long time = getCurrentTime();
// nextCheck因为我们的配置是按天那么nextCheck的值就是明天凌晨
if (time >= nextCheck) {
Date dateOfElapsedPeriod = dateInCurrentPeriod;
addInfo("Elapsed period: " + dateOfElapsedPeriod);
elapsedPeriodsFileName = tbrp.fileNamePatternWithoutCompSuffix.convert(dateOfElapsedPeriod);
setDateInCurrentPeriod(time);
computeNextCheck();
return true;
} else {
return false;
}
}
}
日志翻转代码片段
public class RollingFileAppender extends FileAppender {
public void rollover() {
lock.lock();
try {
// Note: This method needs to be synchronized because it needs exclusive
// access while it closes and then re-opens the target file.
//
// make sure to close the hereto active log file! Renaming under windows
// does not work for open files.
// 这里关闭当前文件输出流
this.closeOutputStream();
// 翻转日志文件，根据应用的配置进行反转，本应用其实就是把文件重命名
// （用的类是TimeBasedRollingPolicy），比如重命名为default.log.2018-11-26
attemptRollover();
// 这个地方打开新文件default.log进行日志输出
attemptOpenFile();
} finally {
lock.unlock();
}
}
// 这个是打开新日志文件的逻辑
private void attemptOpenFile() {
try {
// update the currentlyActiveFile LOGBACK-64
currentlyActiveFile = new File(rollingPolicy.getActiveFileName());
// This will also close the file. This is OK since multiple close operations are safe.
this.openFile(rollingPolicy.getActiveFileName());
} catch (IOException e) {
addError("setFile(" + fileName + ", false) call failed.", e);
}
}
}
对日志文件进行重命名的代码
public class TimeBasedRollingPolicy extends RollingPolicyBase implements TriggeringPolicy {
public void rollover() throws RolloverFailure {
// when rollover is called the elapsed period's file has
// been already closed. This is a working assumption of this method.
String elapsedPeriodsFileName = timeBasedFileNamingAndTriggeringPolicy.getElapsedPeriodsFileName();
String elapsedPeriodStem = FileFilterUtil.afterLastSlash(elapsedPeriodsFileName);
if (compressionMode == CompressionMode.NONE) {
if (getParentsRawFileProperty() != null) {
// 这个地方对文件进行了重命名
renameUtil.rename(getParentsRawFileProperty(), elapsedPeriodsFileName);
} // else { nothing to do if CompressionMode == NONE and parentsRawFileProperty == null }
} else {
if (getParentsRawFileProperty() == null) {
compressionFuture = compressor.asyncCompress(elapsedPeriodsFileName, elapsedPeriodsFileName, elapsedPeriodStem);
} else {
compressionFuture = renameRawAndAsyncCompress(elapsedPeriodsFileName, elapsedPeriodStem);
}
}
if (archiveRemover != null) {
Date now = new Date(timeBasedFileNamingAndTriggeringPolicy.getCurrentTime());
this.cleanUpFuture = archiveRemover.cleanAsynchronously(now);
}
}
}
问题逻辑梳理
- 系统引入了Pandora，导致 diamond对接地方被执行两次，分别是Pandora加载和默认加载
- 其实类似tomcat 一个JVM有多个webapp，他们之间可能用了很多相同的类和框架等，但是又不能互相冲突，肯定需要一个webapp一个classloader, 而一些容器公共的东西就用公共父亲classloader加载
- 系统出现两套log， 这里叫pandora加载的为log1， 默认的加log2
- 跨天业务日志持续输出，log1触发跨天事件，关闭default.log输出流，重命名default.log为default.log.2018-11-26, 创建新文件default.log
- 这个时候发生diamond推送，那么就会触发日志打印表示有推送，由于步骤1， log1和log2都会打印
- log2触发跨天事件，重命名步骤3产生的default.log为default.log.2018-11-26，创建新文件default.log
- log1 持续输出日志到了default.log.2018-11-26， log2输出日志到default.log
问题复现
-
自定义一个classloader
package com.alibaba.log.test; import java.net.URL; import java.net.URLClassLoader; import java.util.HashMap; public class MyURLClassLoader extends URLClassLoader { public MyURLClassLoader(URL[] urls, ClassLoader parent) { super(urls, parent); } public MyURLClassLoader(URL[] urls) {super(urls);} HashMap -
日志输出类（此类有第一步的类加载加载）
package com.alibaba.log.test;
import org.slf4j.Logger;
public class WriteLog {
private static Logger log1 = org.slf4j.LoggerFactory.getLogger(WriteLog.class);
{
new Thread() {
@Override
public void run() {
for (int i = 0; i < 999999; i++) {
log1.info("log1-> " + log1.getClass().getClassLoader().getClass().getName() + " " + i);
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}.start();
}
}
3.程序入口类，此类也有一个日志打印Log不过是用默认类加载器加载
package com.alibaba.log.test;
import org.slf4j.Logger;
import java.io.File;
import java.net.URL;
import java.util.Scanner;
public class Main {
private static Logger log2 = org.slf4j.LoggerFactory.getLogger(WriteLog.class);
public static void main(String[] args) throws Exception {
test();
Scanner scanner = new Scanner(System.in);
System.out.println("输入任意字符触发log2打印日志(记得修改系统时间跨天)");
String line = scanner.nextLine();
log2.info("log2-> " + log2.getClass().getClassLoader().getClass().getName() + " " + line);
}
public static void test() throws Exception {
String[] paths = new String[]{
"/Users/shushangjin/.m2/txp/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar",
"/Users/shushangjin/.m2/txp/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar",
"/Users/shushangjin/.m2/txp/repository/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar",
"/Users/shushangjin/IdeaProjects/code1/testlog/target/classes/"
};
URL[] urls = new URL[paths.length];
for (int i = 0; i < paths.length; i++) {
urls[i] = new File(paths[i]).toURI().toURL();
}
MyURLClassLoader loader = new MyURLClassLoader (urls);
Class cl = Class.forName ("com.alibaba.log.test.WriteLog", true, loader);
System.out.println(cl.getClassLoader().getClass().getName());
cl.newInstance();
}
}
4.当程序跑起来之后，修改系统时间为明天，我这里修改为2018-11-28 5.在程序终端输入任意字符串 6.见证奇迹的时刻
解决方案
1， 排除程序中有多个load的实例进行日志输出，当然也包括多进程 2， 从logback源码层面解决此问题，比如跨天判断的时候，触发了翻转，昨天的日志已经生成，那么不做任何动作