基于CDC做全链路数据审计系统-入口设计(二)

由于我们本系统需要记录这次请求是从哪一个系统及哪个方法进来的,以便和本次请求引起的数据进行关联。
本节我们需要完成三件事:
1.对入口端的统一拦截生成traceId
2.对所入的api请求方法做拦截,记录相应的traceId,入参等重要信并异步发送到mq队列中

生成traceId

因为我们api这边是http入口,所以我们可以简单的实现一个filter接口,然后在filter里设置一个traceId,并放入请求上下文中(我们使用阿里的TransmittableThreadLocal来实现)

trace上下文,主要用来写入读取traceId

import com.alibaba.ttl.TransmittableThreadLocal;
import com.lang.oliver.web.api.util.TraceIdUtil;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.MDC;

@Slf4j
public class TraceContext {
    public static final String TRANSACTION_ID = "TransactionId";

    private static TransmittableThreadLocal transactionThreadLocal = new TransmittableThreadLocal<>();

    public static void setTraceId() {
        String traceId = TraceIdUtil.getTraceId();
        MDC.put(TRANSACTION_ID, traceId);
        transactionThreadLocal.set(traceId);
    }


    public static String getTraceId() {
        return transactionThreadLocal.get();
    }

}

TraceIdUtil 用来生成TraceId的,我们可以直接使用UUID来生成:

import java.util.UUID;

public class TraceIdUtil {

    /**
     *
     * @return
     */
    public static String getTraceId(){
        return UUID.randomUUID().toString().replace("-","");
    }

}

现在我们来实现web拦截filter

import com.lang.oliver.web.api.trace.TraceContext;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import java.io.IOException;

public class TraceFilter implements Filter {

    public TraceFilter() {
    }

    public void init(FilterConfig filterConfig) throws ServletException {
    }

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        TraceContext.setTraceId();
        chain.doFilter(request, response);
    }

    public void destroy() {
    }
}

我们的controller,比如我们要调用其他项目时我们就可以将这个传递过去,实际上你肯定使用的是aop拦截处理,我们本例因为没用feign,用的RestTemplate,所以需要手动设置traceId到RestTemplate里的header里。

import com.alibaba.fastjson.JSON;
import com.lang.oliver.web.api.mq.command.UpdateUserCommand;
import com.lang.oliver.web.api.mq.product.MqProduct;
import com.lang.oliver.web.api.trace.TraceContext;
import com.lang.oliver.web.api.util.TraceIdUtil;
import com.lang.oliver.web.api.vo.WebApiUserVo;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.MDC;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import java.util.Date;

@Slf4j
@RestController
@RequestMapping("/user")
public class ExampleController {

    @Autowired
    private RestTemplate restTemplate;

    @Autowired
    private MqProduct mqProduct;

    @Value("${biz.user.topic}")
    private String userTopic;

    @PutMapping("")
    public String saveUser(@RequestBody WebApiUserVo userVo) {
        log.info("userVo:{}", JSON.toJSONString(userVo));

        HttpHeaders headers = new HttpHeaders();
        headers.set(TraceContext.TRANSACTION_ID, MDC.get(TraceContext.TRANSACTION_ID));
        headers.setContentType(MediaType.APPLICATION_JSON);
        headers.set("Accept", "application/json");
        HttpEntity httpEntity = new HttpEntity<>(JSON.toJSONString(userVo), headers);

        Integer userId = restTemplate.exchange("http://127.0.0.1:9999/user", HttpMethod.POST,
                httpEntity, Integer.class).getBody();
        return String.valueOf(userId);
    }
}

如果你使用的是Mq,你可以把traceId这信息放到Mq的元信息中,或者是在你的事件里面定义一个header,然后将这些信息放进去,这样mq的消费者就可以消息的header拿到这些信息,从而让整个链路全部串起来了。后面的整合章节例子里有实际的完整代码。

到这里,我们的traceId的拦截生成和获取基本就完成了,现在我们还需要将请求的信息及traceId拦截并输出到mq中。

拦截方法

考虑到一些老系统各个包命名不统一(eg:com.aa.bb 和com.cc.xx),又不想每个项目都copy一份相同的代码,这样会导致后以后升级很麻烦,不利于维护。于是我们想做成一个统一的jar包来给各项目引用,这样每个项目只须要引用该jar,然后配置对应的切面值就可以了。

首先我们来定义拦截后的对象结构体:

import lombok.Data;

import java.util.Date;

@Data
public class EntryLogInfoRequest {

    /**
     * 登录人信息
     */
    private Integer customerId;
    /**
     * 每次请求产生的traceId
     */
    private String traceId;
    private String projectName;
    private String className;
    private String methodName;
    private Date requestTime;
    private String parameter;

    /**
     * 返回结果  该值需要根据不同的情况来判断要不要记录,比如返回数据比较大(导出操作)会严重影响性能
     */
    private String result;

    public EntryLogInfoRequest() {
    }

    public static EntryLogInfoRequest buildEntryLogInfoRequest(Integer customerId, String traceId, String className,
                                                               String methodName) {
        EntryLogInfoRequest entryLogInfoRequest = new EntryLogInfoRequest();
        entryLogInfoRequest.setCustomerId(customerId);
        entryLogInfoRequest.setTraceId(traceId);
        entryLogInfoRequest.setClassName(className);
        entryLogInfoRequest.setMethodName(methodName);
        entryLogInfoRequest.setRequestTime(new Date());
        return entryLogInfoRequest;
    }



}

其次,因为我们是使用的是异步发送消息,而且我们想监控线程情况,所以我们需要实现自己的线池程。

线程池工厂:

/**
 * 自定义线程池工厂,方便出现问题做监控
 */
public class ReqLogThreadFactory implements ThreadFactory {

    private static final AtomicInteger poolNumber = new AtomicInteger(1);
    private final ThreadGroup group;
    private final AtomicInteger threadNumber = new AtomicInteger(1);
    private final String namePrefix;

    public ReqLogThreadFactory() {
        SecurityManager securityManager = System.getSecurityManager();
        this.group = securityManager != null ? securityManager.getThreadGroup() : Thread.currentThread().getThreadGroup();
        this.namePrefix = "reqLog-comm-log-" + poolNumber.getAndIncrement() + "-thread-";
    }


    public Thread newThread(Runnable runnable) {
        Thread thread = new Thread(this.group, runnable, this.namePrefix + this.threadNumber.getAndIncrement(), 0L);
        if (thread.isDaemon()) {
            thread.setDaemon(false);
        }

        if (thread.getPriority() != 5) {
            thread.setPriority(5);
        }

        return thread;
    }

}

线程池及mq配置类:

/**
 *  拦截相关的配置信息
 *  优化:可以修改读取配置文件
 */

@Data
public class ReqLogConfigurerProperties {

    private String projectName;
    private String topicName = "common_req_trace_log";
    private int corePoolSize = 5;
    private int maximumPoolSize = 10;
    private long keepAliveTime = 60L;
    private BlockingQueue workQueue = new LinkedBlockingDeque(5000);
    private RejectedExecutionHandler handler = new ThreadPoolExecutor.DiscardPolicy();
    
}

请求拦截类:

/**
 * 请求拦截器 将入口数据写入mq 然后解析服务解析数据并写入仓储
 */
@Slf4j
public class ReqLogAdviceInterceptor implements MethodInterceptor {

    private ExecutorService fixedThreadPool;

    private KafkaTemplate kafkaTemplate;
    private ReqLogConfigurerProperties reqLogConfig;

    public ReqLogAdviceInterceptor(KafkaTemplate kafkaTemplate,ReqLogConfigurerProperties properties) {
        this.reqLogConfig = properties;
        this.kafkaTemplate = kafkaTemplate;
        this.fixedThreadPool = this.buildThreadPool(properties);
    }

    private ExecutorService buildThreadPool(ReqLogConfigurerProperties properties) {
        return new ThreadPoolExecutor(properties.getCorePoolSize(), properties.getMaximumPoolSize(),
                properties.getKeepAliveTime(), TimeUnit.SECONDS, properties.getWorkQueue(),
                new ReqLogThreadFactory(), properties.getHandler());
    }


    public Object invoke(MethodInvocation methodInvocation) throws Throwable {
        EntryLogInfoRequest entryLogInfoRequest = this.buildEntryLogInfoRequest(methodInvocation);
        entryLogInfoRequest.setProjectName(this.reqLogConfig.getProjectName());
        Object result = methodInvocation.proceed();

        try {
            this.fixedThreadPool.execute(() -> {
                try {
                    String string = (new ObjectMapper()).writeValueAsString(entryLogInfoRequest);
                    this.kafkaTemplate.send(this.reqLogConfig.getTopicName(), string);
                } catch (JsonProcessingException var3) {
                    log.error("日志发送kafka失败,data:{}", JSON.toJSONString(entryLogInfoRequest), var3);
                }
            });
        } catch (Exception e) {
            log.error("fixedThreadPool 执行任务失败",e);
        }

        return result;
    }


    private EntryLogInfoRequest buildEntryLogInfoRequest(MethodInvocation methodInvocation) {
        Method method = methodInvocation.getMethod();
        String className = method.getDeclaringClass().getName();
        String methodName = method.getName();
        Integer customerId = LoginSessionUtil.getLoginUserId();

        String traceId = TraceContext.getTraceId();

        return EntryLogInfoRequest.buildEntryLogInfoRequest(customerId, traceId, className, methodName);
    }

}

现在将上面的bean注入进来:

@Configuration
@Slf4j
public class CommonConfig {

    @Value("${point.cut:com.lang.oliver.web.api.controller}")
    private String pointCut;

    @Autowired
    private KafkaTemplate kafkaTemplate;

    @Bean
    public FilterRegistrationBean filterRegistrationBean() {
        FilterRegistrationBean registration = new FilterRegistrationBean();
        registration.setFilter(new TraceFilter());
        registration.addUrlPatterns("/*");
        registration.setName("TraceFilter");
        registration.setOrder(-100);
        return registration;
    }

    /**
     * @return
     */
    @Bean
    public ReqLogConfigurerProperties packageReqLofConfig() {
        ReqLogConfigurerProperties reqLofConfig = new ReqLogConfigurerProperties();
        reqLofConfig.setProjectName("k2-tmk-web-api");
        reqLofConfig.setTopicName("common_req_trace_log");
        return reqLofConfig;
    }


    @Bean
    public AspectJExpressionPointcutAdvisor configurabledvisor(ReqLogConfigurerProperties reqLogConfigurer) {
        String pointCutStr = "execution(public * " + pointCut + "..*.*(..))";
        AspectJExpressionPointcutAdvisor advisor = new AspectJExpressionPointcutAdvisor();
        advisor.setExpression(pointCutStr);
        advisor.setAdvice(new ReqLogAdviceInterceptor(kafkaTemplate,reqLogConfigurer));
        return advisor;
    }

    @Bean
    RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

至此,整个入口端的就完成了

你可能感兴趣的:(基于CDC做全链路数据审计系统-入口设计(二))