使用log4j2对日志脱敏

Mask & filter sensitive data in log4j2

log4j V1 版本可以通过继承org.apache.log4j.PatternLayout类来实现日志的脱敏，如下所示.

public class CustomPatternLayout extends org.apache.log4j.PatternLayout {
    @Override
    public String format(LoggingEvent event) {
        String temp = super.format(event);
        return doFilteringStuff(temp);
    }
}

但是从log4j 2.x 开始，PatternLayout类被定义为了final类型，不能再被继承，因此想要在log4j 2.x版本中实现日志中敏感信息的filter或者mask的功能就需要通过其他方式来实现, 下面总结了几种方式：

• Filters

  log4j2 允许用户配置filters到指定的loggers、appenders或者全局配置中(应用到所有filter和appender上)，filter的机制是通过返回一个三种状态的枚举(ACCEPT | DENY | NEUTRAL)来决定log events的处理过程，log4j2内置了RegexFilter 和 ScriptFilter可供用户过滤log日志信息。

  • Regex filter example

    @Getter
    @Setter
    public class Customer {
        private String name;
        private String creditCardNo;
        private String password;
    	
        @Override
        public String toString() {
            return "Customer[name="+name+", creditCardNo="+creditCardNo+", password="+password+"]";
        }
    }
    

    应用中log的日志长如下样子：

    public class CustomerLoggingApp {
        public static void main(String[] args) {
            Logger log = LogManager.getLogger();
    
            Customer customer = new Customer();
            customer.setName("Rick");
            customer.setCreditCardNo("1111-2222-3333-4444");
    		customer.setPassword("112233");
            log.info("This is sensitive and should not be logged: {}", customer);
            log.info("But this message should be logged.");
        }
    }
    

    此时，如果你想例如如果你想将包含有“Customer“字符串后跟", creditCardNo="的日志都屏蔽不打印，那就可以在log4j2.xml文件中配置RegexFilter：

    
    <Configuration status="warn">
      <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
          <RegexFilter regex=".*Customer.*, creditCardNo=.*" onMatch="DENY" onMismatch="NEUTRAL"/>
          <PatternLayout>
            <pattern>%d %level %c %m%npattern>
          PatternLayout>
        Console>
      Appenders>
      <Loggers>
        <Root level="debug">
          <AppenderRef ref="Console" />
        Root>
      Loggers>
    Configuration>
    

  • Script filter example

    另外一种非常灵活的filter是 ScriptFilter，下面的例子使用的是Groovy，你也可以使用javaScript或其他java应用环境可用的脚本语言，还以上面应用的Customer类为例，下面log4j2.xml的配置将会过滤掉任何包含Customer全类名的log event.

    
    <Configuration status="warn">
      <ScriptFilter onMatch="DENY" onMisMatch="NEUTRAL">
        <Script name="DropSensitiveObjects" language="groovy">
                    parameters.any { p ->
                        // DENY log messages with Customer parameters
                        p.class.name == "Customer"
                    }
                  ]]>
        Script>
      ScriptFilter>
      <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
          <PatternLayout>
            <pattern>%d %level %c %m%npattern>
          PatternLayout>
        Console>
      Appenders>
      <Loggers>
        <Root level="debug">
          <AppenderRef ref="Console" />
        Root>
      Loggers>
    Configuration>
    

• Rewriting Log Events

  有时候你可能有将log中password 或者 银行卡号之类的信息替换为”***“的形式的需求，想要完成这个功能就可以创建一个RewriteAppender，从log4j2的手册中可以看到

  The RewriteAppender allows the LogEvent to manipulated before it is processed by another Appender. This can be used to mask sensitive information such as passwords or to inject information into each event. The RewriteAppender must be configured with a RewritePolicy. The RewriteAppender should be configured after any Appenders it references to allow it to shut down properly.

  rewrite policy样例:

  package com.rick.demo;
  
  import org.apache.logging.log4j.core.Core;
  import org.apache.logging.log4j.core.LogEvent;
  import org.apache.logging.log4j.core.appender.rewrite.RewritePolicy;
  import org.apache.logging.log4j.core.config.plugins.Plugin;
  import org.apache.logging.log4j.core.config.plugins.PluginElement;
  import org.apache.logging.log4j.core.config.plugins.PluginFactory;
  import org.apache.logging.log4j.core.impl.Log4jLogEvent;
  import org.apache.logging.log4j.message.Message;
  import org.apache.logging.log4j.message.ObjectMessage;
  import org.apache.logging.log4j.message.ParameterizedMessage;
  import org.apache.logging.log4j.message.ReusableMessage;
  
  @Plugin(name = "MaskSensitiveDataPolicy", category = Core.CATEGORY_NAME, 
          elementType = "rewritePolicy", printObject = true)
  public class MaskSensitiveDataPolicy implements RewritePolicy {
  
      private String[] sensitiveClasses;
  
      @PluginFactory
      public static MaskSensitiveDataPolicy createPolicy(
              @PluginElement("sensitive") final String[] sensitiveClasses) {
          return new MaskSensitiveDataPolicy(sensitiveClasses);
      }
  
      private MaskSensitiveDataPolicy(String[] sensitiveClasses) {
          super();
          this.sensitiveClasses = sensitiveClasses;
      }
  
      @Override
      public LogEvent rewrite(LogEvent event) {
          Message rewritten = rewriteIfSensitive(event.getMessage());
          if (rewritten != event.getMessage()) {
              return new Log4jLogEvent.Builder(event).setMessage(rewritten).build();
          }
          return event;
      }
  
      private Message rewriteIfSensitive(Message message) {
          // 确保已经通过设置系统属性`log4j2.enable.threadlocals` 为 `false`关闭了garbage-free logging
          // 否则可能传入ReusableObjectMessage, ReusableParameterizedMessage或
          // MutableLogEvent messages 导致不能重写。
          
          // Make sure to switch off garbage-free logging
          // by setting system property `log4j2.enable.threadlocals` to `false`.
          // Otherwise you may get ReusableObjectMessage, ReusableParameterizedMessage
          // or MutableLogEvent messages here which may not be rewritable...
          if (message instanceof ObjectMessage) {
              return rewriteObjectMessage((ObjectMessage) message);
          }
          if (message instanceof ParameterizedMessage) {
              return rewriteParameterizedMessage((ParameterizedMessage) message);
          }
          return message;
      }
  
      private Message rewriteObjectMessage(ObjectMessage message) {
          if (isSensitive(message.getParameter())) {
              return new ObjectMessage(maskSensitive(message.getParameter()));
          }
          return message;
      }
  
      private Message rewriteParameterizedMessage(ParameterizedMessage message) {
          Object[] params = message.getParameters();
          boolean changed = rewriteSensitiveParameters(params);
          return changed ? new ParameterizedMessage(message.getFormat(), params) : message;
      }
  
      private boolean rewriteSensitiveParameters(Object[] params) {
          boolean changed = false;
          for (int i = 0; i < params.length; i++) {
              if (isSensitive(params[i])) {
                  params[i] = maskSensitive(params[i]);
                  changed = true;
              }
          }
          return changed;
      }
  
      private boolean isSensitive(Object parameter) {
          return parameter instanceof Customer;
      }
  
      private Object maskSensitive(Object parameter) {
          Customer result = new Customer();
          result.setName((Customer) parameter).getName());
          result.setPassword("***");
          result.setCreditCardNo("****-****-****-****");
          return result;
      }
  }
  

  CAUTION: When running in garbage-free mode (the default), Log4j2 uses reusable objects for messages and log events. These are not suitable for rewriting. (This is not documented well in the user manual.) If you want to use the rewrite appender, you need to partially switch off garbage-free logging by setting system property log4j2.enable.threadlocals to false. 推荐在类路径下创建一个名为log4j2.component.properties的文件，加入log4j2.enable.threadlocals=false, 会被log4j框架自动加载

  使用自定义的MaskSensitiveDataPolicy配置rewriteAppender, 为了能让log4j2 能够识别自定义的插件，需要指定插件所在的包，如下所示：

  
  <Configuration status="warn" packages="com.rick.demo">
    <Appenders>
      <Console name="Console" target="SYSTEM_OUT">
        <PatternLayout>
          <pattern>%d %level %c %m%npattern>
        PatternLayout>
      Console>
  
      <Rewrite name="obfuscateSensitiveData">
        <AppenderRef ref="Console"/>
        <MaskSensitiveDataPolicy />
      Rewrite>
  
    Appenders>
    <Loggers>
      <Root level="debug">
        <AppenderRef ref="obfuscateSensitiveData"/>
      Root>
    Loggers>
  Configuration>
  

  这样就能够产出如下输出：

  2021-08-30 19:22:31,431 INFO CustomerLoggingApp This is sensitive and should not be logged: Customer[name=Rick, creditCardNo=****-****-****, password=***]
  2021-08-30 19:22:31,432 INFO CustomerLoggingApp But this message should be logged.
  

• LogEventFactory

  1. 通过实现LogEventFactory接口, 如下所示：

  package com.rick.log4j.factory.event;
  
  import org.apache.logging.log4j.Level;
  import org.apache.logging.log4j.Marker;
  import org.apache.logging.log4j.core.LogEvent;
  import org.apache.logging.log4j.core.config.Property;
  import org.apache.logging.log4j.core.impl.Log4jLogEvent;
  import org.apache.logging.log4j.core.impl.LogEventFactory;
  import org.apache.logging.log4j.message.Message;
  import org.apache.logging.log4j.message.SimpleMessage;
  
  import java.util.List;
  import java.util.Objects;
  import java.util.regex.Matcher;
  import java.util.regex.Pattern;
  
  /**
   * @author rick
   * E-mail:[email protected]
   * @version 2021/8/26 10:51
   */
  public class CustomLogEventFactory implements LogEventFactory {
  //public class CustomLogEventFactory extends ReusableLogEventFactory {
  
      private static final CustomLogEventFactory instance = new CustomLogEventFactory();
      /**
       * 
  
       * mask RegExp, to reserve the first three
       * and last four chars for search use.
       * 
       */
      private static final String MASK_REGEX = "(?<=.{3}).(?=.*....)";
      /**
       * Credit Card Type                 Prefix                          Length
       * American Express	                34, or 37	                    15
       * MasterCard	                    51 through 55	                16
       * Visa	                            4	                            13 or 16
       * Diners Club and Carte Blanche	36,38,or 300 through 305	    14
       * Discover	                        6011	                        16
       * JCB	                            2123 or 1800	                15
       * JCB	                            3	                            16
       */
      private Pattern creditCardPattern = Pattern.compile("((?:(?:4\\d{3})|(?:5[1-5]\\d{2})|6(?:011|5[0-9]{2}))(?:-?|\\040?)(?:\\d{4}(?:-?|\\040?)){3}|(?:3[4,7]\\d{2})(?:-?|\\040?)\\d{6}(?:-?|\\040?)\\d{5})");
  
      /**
       * @return
       */
      @SuppressWarnings("unused")
      public static CustomLogEventFactory getInstance() {
          return instance;
      }
  
      /**
       * Creates a log event.
       *
       * @param loggerName The name of the Logger.
       * @param marker     An optional Marker.
       * @param fqcn       The fully qualified class name of the caller.
       * @param level      The event Level.
       * @param message    The Message.
       * @param properties Properties to be added to the log event.
       * @param throwable  An optional Throwable.
       * @return The LogEvent.
       */
      public LogEvent createEvent(String loggerName, Marker marker, String fqcn, Level level,
                                  Message message, List<Property> properties, Throwable throwable) {
  
          String formattedMessage = message.getFormattedMessage();
          String obfuscatedMsg = obfuscateIfNecessary(creditCardPattern, formattedMessage);
          Message handledMsg = new SimpleMessage(obfuscatedMsg);
          return new Log4jLogEvent(loggerName, marker, fqcn, level, handledMsg, properties, throwable);
      }
  
      /**
       * 
  
       *  obfuscate the digital chars
       *  which exist in the given string with '*' character
       *  except the first 4 chars ahead and last 4 chars behind.
       * 
       *
       * examples:
       * 4111-2222-3333-4444  -> 4111-****-****-4444
       * 4111222233334444     -> 4111********4444
       * 4111 2222 3333 4444  -> 4111********4444
       *
       * @param sensitiveData
       * @return obfuscated string
       */
      @SuppressWarnings("unused")
      private String obfuscate(String sensitiveData) {
          char[] chars = sensitiveData.toCharArray();
          int prefixLength = 4;// reserve first four chars
          int suffixLength = 4;// reserve last four chars
          if ((chars.length <= prefixLength + suffixLength)) {
              return sensitiveData;
          }
          for (int i = prefixLength; i < chars.length - suffixLength; i++) {
              if (isDigital(chars[i])) {
                  chars[i] = '*';
              }
          }
          return new String(chars);
      }
  
      /**
       * obfuscate the digital chars which exist in the given string
       * with '*' character except the first three
       * chars ahead and last four chars behind.
       * eg.
       * 4111-2222-3333-4444  -> 411************4444
       * 4111222233334444     -> 411*********4444
       * 4111 2222 3333 4444  -> 411************4444
       *
       * @param sensitiveData
       * @return obfuscated string
       */
      private String obfuscateByRegex(String sensitiveData) {
          return sensitiveData.replaceAll(MASK_REGEX, "*");
      }
  
      /**
       * @param ch
       * @return whether the given char is numeric
       */
      private boolean isDigital(char ch) {
          return (ch >= 48 && ch <= 57);
      }
  
      /**
       * @param formattedMsg #{@link Message#getFormattedMessage()}
       * @return obfuscated string.
       */
      private String obfuscateIfNecessary(Pattern pattern, String formattedMsg) {
          Matcher matcher = pattern.matcher(formattedMsg);
          StringBuffer sb = new StringBuffer();
          while (matcher.find()) {
              String original = matcher.group();
              if (!Objects.isNull(original)) {
  //                String obfuscated = obfuscate(original);
                  String obfuscated = obfuscateByRegex(original);
                  matcher.appendReplacement(sb, obfuscated);
              }
          }
          if (sb.length() != 0) {
              matcher.appendTail(sb);
              return sb.toString();
          }
          return formattedMsg;
      }
  }
  
  

  2. 在类路径下创建一个名为log4j2.component.properties的文件，加入一行Log4jLogEventFactory = com.rick.log4j.factory.event.CustomLogEventFactory指明要使用的Log4jLogEventFactory为自定义的Log4jLogEventFactory.

  3. 正常配置log4j2.xml，无需额外配置。

• LogEventPatternConverter

  1. 继承org.apache.logging.log4j.core.pattern.LogEventPatternConverter类实现format方法，可以对LogEvent 进行格式化，进而实现自定义的pattern

  package com.rick.log4j.converter;
  
  import com.rick.log4j.marker.CustomMarker;
  import org.apache.logging.log4j.Marker;
  import org.apache.logging.log4j.core.LogEvent;
  import org.apache.logging.log4j.core.config.plugins.Plugin;
  import org.apache.logging.log4j.core.pattern.ConverterKeys;
  import org.apache.logging.log4j.core.pattern.LogEventPatternConverter;
  
  import java.util.Objects;
  import java.util.regex.Matcher;
  import java.util.regex.Pattern;
  
  /**
   * @author rick
   * E-mail:[email protected]
   * @version 2021/8/25 16:26
   */
  @Plugin(name = "SensitiveDataConverter", category = "Converter")
  @ConverterKeys({"sc"})
  public class CustomLogEventPatternConverter extends LogEventPatternConverter {
  
      /**
       * pattern for credit card NO.
       */
      private static final Pattern CREDIT_CARD_PATTERN = Pattern.compile("((?:(?:4\\d{3})|(?:5[1-5]\\d{2})|6(?:011|5[0-9]{2}))(?:-?|\\040?)(?:\\d{4}(?:-?|\\040?)){3}|(?:3[4,7]\\d{2})(?:-?|\\040?)\\d{6}(?:-?|\\040?)\\d{5})");
  
      /**
       * 
  
       * mask RegExp, to reserve the first three
       * and last four chars for search use.
       * 
       */
      private static final String MASK_REGEX = "(?<=.{3}).(?=.*....)";
  
      /**
       * constructor
       *
       * @param options
       */
      public CustomLogEventPatternConverter(String[] options) {
          super("sc", "sc");
      }
  
      /**
       * 
   Unlike most other Plugins, Converters do not use a PluginFactory.
       * Instead, each Converter is required to provide a static newInstance
       * method that accepts an array of Strings as the only parameter.
       * The String array are the values that are specified within the curly
       * braces that can follow the converter key.
       *
       * @param options
       * @return
       */
      public static CustomLogEventPatternConverter newInstance(final String[] options) {
          return new CustomLogEventPatternConverter(options);
      }
  
      /**
       * @param event
       * @param toAppendTo
       */
      public void format(LogEvent event, StringBuilder toAppendTo) {
          String message = event.getMessage().getFormattedMessage();
          Marker marker = event.getMarker();
          // CustomMarker.SENSITIVE_DATA_MARKER.name() = "SENSITIVE_DATA_MARKER";
          if (Objects.isNull(marker)
                  || CustomMarker.SENSITIVE_DATA_MARKER.name().compareToIgnoreCase(marker.getName()) != 0) {
              toAppendTo.append(message);
              return;
          }
          Matcher matcher = CREDIT_CARD_PATTERN.matcher(message);
          StringBuffer sb = new StringBuffer();
          while (matcher.find()) {
              String original = matcher.group();
              if (!Objects.isNull(original)) {
                  String obfuscated = original.replaceAll(MASK_REGEX, "*");
                  matcher.appendReplacement(sb, obfuscated);
              }
          }
          if (sb.length() != 0) {
              matcher.appendTail(sb);
              toAppendTo.append(sb.toString());
              return;
          }
          toAppendTo.append(message);
      }
  }
  
  

  2. 在类路径下新建log4j2.component.properties配置文件，写入Log4jLogEventFactory = com.rick.log4j.factory.event.CustomLogEventFactory 配置

  3. 配置xml，在xml中使用%sc 替换掉%m（遇到%sc，log4j就会使用自定义的插件并调用format(LogEvent event, StringBuilder toAppendTo)方法。

  4. 应用中使用logger， 注意Marker的使用

     package com.rick.log4j.entrypoint;
     
     import com.rick.log4j.pojo.Customer;
     import org.slf4j.Logger;
     import org.slf4j.LoggerFactory;
     import org.slf4j.Marker;
     import org.slf4j.MarkerFactory;
     
     import java.util.Date;
     import java.util.HashMap;
     import java.util.Map;
     
     /**
      * @author rick
      * E-mail:[email protected]
      * @version 2021/8/30 14:04
      */
     public class CustomConverterTest {
         private static final Marker SENSITIVE_DATA_MARKER = MarkerFactory.getMarker("SENSITIVE_DATA_MARKER");
         private static Logger logger = LoggerFactory.getLogger(CustomConverterTest.class);
     
         public static void main(String[] args) {
     
     //        System.out.println("log4j2.enable.threadlocals = " + System.getProperty("log4j2.enable.threadlocals"));
             String cardNo = "4934-5322-4597-2245";
     
             Customer customer = new Customer();
             customer.setName("rick");
             customer.setCreditCardNo(cardNo);
     
             customer.setAge(30);
             customer.setCreditCardPassword(cardNo);
     
             Map<String, Object> map = new HashMap<>();
             map.put("phoneNo", 110);
             map.put("customer", customer);
             map.put("uid", cardNo);
     
             long start1 = System.currentTimeMillis();
             for (int i = 0; i < 10000; i++) {
                 logger.warn(SENSITIVE_DATA_MARKER, cardNo);
                 logger.info(SENSITIVE_DATA_MARKER, "string :{}", cardNo);
                 logger.info(SENSITIVE_DATA_MARKER, "map :{}", map);
                 logger.info(SENSITIVE_DATA_MARKER, "object :{}", customer.toString());
                 logger.info(SENSITIVE_DATA_MARKER, "date :{}", new Date());
                 logger.error(SENSITIVE_DATA_MARKER, "Map:{}, String :{}", map, cardNo);
             }
             long end1 = System.currentTimeMillis();
             System.out.println("end1 - start1(marker) = " + (end1 - start1));
         }
     }
     

参考资料:

​ Masking sensitive data in log4j2

​ How do I implement the Luhn algorithm?

​ java(log4j) logging filter by object type

​ Log4j2: How To Mask Logs Personal/Confidential/SPI Information

​ Editing log messages - LogEventFactory vs RewriteAppender

​ Masking credit card number using regex

​ How To Mask Sensitive Data

​ How to make a custom message converter for log4j2

​ java logging best practices

​ how to use log4j2.xml rewrite appender for modifying log event before it logs in file

​