nginx 500状态排查

现有现象

• 我们的业务通过iframe嵌入在内部体系中
• 默认通过内部账号搜索框，跳转我们的页面(登录判断)
• 内部账号cookie过期时间默认为一天，不支持动态设置过期时间
• 内部账号过期后，通过指定cookie不能获取值，跳转登录页
• 登录页返回状态码为302，通过nginx日志反馈，则是500
• 运维新增告警规范，项目访问返回status >= 500 则会收到告警
  

问题分析

• 通过在内部账号体系搜索后的连接，拼接上参数 &traceId=uuid值
• 通过traceId+userId能确认唯一访问
  

加traceId原因：访问统一个用户，可能是公司内不同用户在同一时间访问的
用户强制刷新页面也能分析出，是否为用户强制刷新操作

• DispatcherServlet 为 SpringMVC 前置处理器

所有http请求都被拦截

• 使用过滤器

/**
 * UserInfoFilter
 *
 * @author weigang
 * @create 2020-07-30
 **/
@Slf4j
@Order(value = Ordered.HIGHEST_PRECEDENCE + 2)
@WebFilter(filterName = "userInfoFilter", urlPatterns = {
     "/*"})
public class UserInfoFilter implements Filter {
     
    @Override
    public void init(FilterConfig filterConfig) throws ServletException {
     
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
     
        HttpServletRequest req = (HttpServletRequest) request;
        HttpServletResponse res = (HttpServletResponse) response;
        try {
     
            Enumeration<String> names = req.getHeaderNames();
            Map<String, String> map = Maps.newHashMap();
            while (names.hasMoreElements()) {
     
                String nextElement = names.nextElement();
                map.put(nextElement, req.getHeader(nextElement));
            }

            log.info("headers map-> {}", JSON.toJSONString(map));
            chain.doFilter(request, response);
        } catch (Exception e) {
     
            e.printStackTrace();
        } finally {
     
            log.info("doFilter finally url->{} status-> {}", req.getRequestURI(), res.getStatus());
        }

    }

    @Override
    public void destroy() {
     

    }
}

缺点：当内部账号体系没登录时，则不会进入这个filter
最后，排查刚好是内部账号失效时，才会出现500 status

• ServletRequestHandledEvent 包含 Servlet 上下文信息

/**
 * ServletRequestListener
 *
 * @author weigang
 * @create 2020-07-28
 **/
@Slf4j
@Component
public class ServletRequestListener implements ApplicationListener<ServletRequestHandledEvent> {
     
    @Override
    public void onApplicationEvent(ServletRequestHandledEvent event) {
     
        String requestUrl = event.getRequestUrl();
        log.info("ServletRequestHandledEvent requestUrl-> {} address-> {} method-> {} servletName-> {} statusCode-> {} exception-> {}",requestUrl, event.getClientAddress(), event.getMethod(),event.getServletName(), event.getStatusCode(), event.getFailureCause());
    }
}

通过日志分析，没有状态码为500的
缺点：得到信息少，不能定位问题

• ServletContextRequestLoggingFilter 原生就支持写日志 这里重新日志格式和位置

@Order(value = Ordered.HIGHEST_PRECEDENCE)
@Slf4j
@WebFilter(filterName = "servletContextRequestLoggingExtFilter", urlPatterns = "/*")
public class ServletContextRequestLoggingExtFilter extends ServletContextRequestLoggingFilter {
     

    private ThreadLocal<HttpServletResponse> threadLocal = new ThreadLocal<>();

    @Autowired
    private BossUtil bossUtil;

    @Override
    protected void initFilterBean() throws ServletException {
     
        super.setIncludeQueryString(true);
        super.setIncludeClientInfo(true);
        super.setIncludePayload(true);
        super.setIncludeHeaders(true);
        super.setMaxPayloadLength(10000);

        //头信息
        super.setBeforeMessagePrefix("请求开始 [");
        super.setBeforeMessageSuffix("]");
        super.setAfterMessagePrefix("请求结束 [");
        super.setAfterMessageSuffix("]");
    }

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
     
        boolean isFirstRequest = !isAsyncDispatch(request);
        HttpServletRequest requestToUse = request;

        if (isIncludePayload() && isFirstRequest && !(request instanceof ContentCachingRequestWrapper)) {
     
            requestToUse = new ContentCachingRequestWrapper(request, getMaxPayloadLength());
        }

        boolean shouldLog = shouldLog(requestToUse);
        if (shouldLog && isFirstRequest) {
     
            beforeRequest(requestToUse, null);
        }
        try {
     
            filterChain.doFilter(requestToUse, response);
            threadLocal.set(response);
        } finally {
     
            if (shouldLog && !isAsyncStarted(requestToUse)) {
     
                afterRequest(requestToUse, null);
            }
        }
    }

    @Override
    protected void beforeRequest(HttpServletRequest request, String message) {
     
        MDC.put("traceId", CommonUtils.getUUID());
        log.info("ServletContextRequestLoggingExtFilter beforeRequest message-> " + super.createMessage(request, DEFAULT_BEFORE_MESSAGE_PREFIX, DEFAULT_BEFORE_MESSAGE_SUFFIX));
    }

    @Override
    protected void afterRequest(HttpServletRequest request, String message) {
     
        log.info("ServletContextRequestLoggingExtFilter afterRequest message-> {} response status-> {}",
                super.createMessage(request, DEFAULT_AFTER_MESSAGE_PREFIX, DEFAULT_AFTER_MESSAGE_SUFFIX), threadLocal.get().getStatus());
        MDC.remove("traceId");
    }
}

日志级别最高，设置 traceId，便于问题排查
重写日志便于观察，最终定位到response时，status为302所致

问题解决

• 在内部账号体系登录后(设置对应域cookie)，得到指定cookie，延长过期时间
• 则再次判断登录时，可以取到cookie的值，则校验登录成功