Java分布式跟踪系统Zipkin（四）：Brave源码分析-HttpTracing

所有博文均在个人独立博客http://blog.mozhu.org首发，欢迎访问！

上一篇博文中，我们分析了Tracing的相关源代码，这一篇我们来看看Brave是如何在Web项目中使用的

我们先来看看普通的servlet项目中，如何使用Brave，这对我们后面分析和理解Brave和SpringMVC等框架整合有帮助

首先Chapter1/servlet25项目中配置了FrontServlet和BackendServlet以及TracingFilter

web.xml

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
    http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">

  <display-name>Servlet2.5 Applicationdisplay-name>

  <filter>
    <filter-name>TracingFilterfilter-name>
    <filter-class>org.mozhu.zipkin.filter.BraveTracingFilterfilter-class>
  filter>
  <filter-mapping>
    <filter-name>TracingFilterfilter-name>
    <url-pattern>/*url-pattern>
  filter-mapping>

  <servlet>
    <servlet-name>BackendServletservlet-name>
    <servlet-class>org.mozhu.zipkin.servlet.BackendServletservlet-class>
    <load-on-startup>1load-on-startup>
  servlet>
  <servlet-mapping>
    <servlet-name>BackendServletservlet-name>
    <url-pattern>/apiurl-pattern>
  servlet-mapping>

  <servlet>
    <servlet-name>FrontendServletservlet-name>
    <servlet-class>org.mozhu.zipkin.servlet.FrontendServletservlet-class>
    <load-on-startup>1load-on-startup>
  servlet>
  <servlet-mapping>
    <servlet-name>FrontendServletservlet-name>
    <url-pattern>/url-pattern>
  servlet-mapping>
web-app>

TracingFilter

我们使用自定义的BraveTracingFilter作为入口，其init方法中，我们初始化了Tracing，然后创建HttpTracing对象，最后调用TracingFilter.create(httpTracing)创建了tracingFilter。
doFilter方法中，所有请求将被tracingFilter来处理

BraveTracingFilter

package org.mozhu.zipkin.filter;

import brave.Tracing;
import brave.context.log4j2.ThreadContextCurrentTraceContext;
import brave.http.HttpTracing;
import brave.propagation.B3Propagation;
import brave.propagation.ExtraFieldPropagation;
import brave.servlet.TracingFilter;
import zipkin2.codec.SpanBytesEncoder;
import zipkin2.reporter.AsyncReporter;
import zipkin2.reporter.Sender;
import zipkin2.reporter.okhttp3.OkHttpSender;

import javax.servlet.*;
import java.io.IOException;
import java.util.concurrent.TimeUnit;

public class BraveTracingFilter implements Filter {
    Filter tracingFilter;

    @Override
    public void init(FilterConfig filterConfig) throws ServletException {
        Sender sender = OkHttpSender.create("http://localhost:9411/api/v2/spans");
        AsyncReporter asyncReporter = AsyncReporter.builder(sender)
                .closeTimeout(500, TimeUnit.MILLISECONDS)
                .build(SpanBytesEncoder.JSON_V2);

        Tracing tracing = Tracing.newBuilder()
                .localServiceName(System.getProperty("zipkin.service", "servlet25-demo"))
                .spanReporter(asyncReporter)
                .propagationFactory(ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "user-name"))
                .currentTraceContext(ThreadContextCurrentTraceContext.create())
                .build();

        HttpTracing httpTracing = HttpTracing.create(tracing);
        filterConfig.getServletContext().setAttribute("TRACING", httpTracing);
        tracingFilter = TracingFilter.create(httpTracing);
        tracingFilter.init(filterConfig);
    }

    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
        tracingFilter.doFilter(servletRequest, servletResponse, filterChain);
    }

    @Override
    public void destroy() {
        tracingFilter.destroy();
    }

}

TracingFilter

TracingFilter在brave-instrumentation-servlet包中

public final class TracingFilter implements Filter {
  public static Filter create(Tracing tracing) {
    return new TracingFilter(HttpTracing.create(tracing));
  }

  public static Filter create(HttpTracing httpTracing) {
    return new TracingFilter(httpTracing);
  }

  final ServletRuntime servlet = ServletRuntime.get();
  final Tracer tracer;
  final HttpServerHandler handler;
  final TraceContext.Extractor extractor;

  TracingFilter(HttpTracing httpTracing) {
    tracer = httpTracing.tracing().tracer();
    handler = HttpServerHandler.create(httpTracing, new HttpServletAdapter());
    extractor = httpTracing.tracing().propagation().extractor(HttpServletRequest::getHeader);
  }
}

TracingFilter中几个重要的类
- HttpTracing - 包含Http处理相关的组件，clientParser，serverParser，clientSampler，serverSampler
- ServletRuntime - Servlet运行时类，包含根据环境来判断是否支持Servlet3异步调用等方法
- HttpServerHandler - Http处理的核心组件，基本上所有和trace相关的操作均在此类中完成
- HttpServletAdapter - HttpServlet的适配器接口，此类的引入可以让httpServerHandler类变得更为通用，因为它是一个泛型接口，跟具体的request和response无关，能和更多框架进行整合
- TraceContext.Extractor - TraceContext的数据提取器

doFilter方法

@Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
      throws IOException, ServletException {
    HttpServletRequest httpRequest = (HttpServletRequest) request;
    HttpServletResponse httpResponse = servlet.httpResponse(response);

    Span span = handler.handleReceive(extractor, httpRequest);
    Throwable error = null;
    try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
      chain.doFilter(httpRequest, httpResponse); // any downstream filters see Tracer.currentSpan
    } catch (IOException | ServletException | RuntimeException | Error e) {
      error = e;
      throw e;
    } finally {
      if (servlet.isAsync(httpRequest)) { // we don't have the actual response, handle later
        servlet.handleAsync(handler, httpRequest, span);
      } else { // we have a synchronous response, so we can finish the span
        handler.handleSend(httpResponse, error, span);
      }
    }
  }

• 首先调用handler.handleReceive(extractor, httpRequest)从request中提取Span信息
• 然后调用tracer.withSpanInScope(span)将Span包装成Tracer.SpanInScope，而Tracer.SpanInScope和前面博文中分析的CurrentTraceContext.Scope比较像，都实现了Closeable接口，这里的目的也一样，都是为了利用JDK7的try-with-resources的特性，JVM会自动调用close方法，做一些线程对象的清理工作。其区别是后者是SPI（Service Provider Interface），不适合暴露给真正的使用者。
  这样使得chain.doFilter(httpRequest, httpResponse)里的代码能用Tracer.currentSpan拿到从请求中提取（extract）的Span信息。
• 最后调用handler.handleSend(httpResponse, error, span)

下面来仔细分析下handler中handleReceive和handleSend两个方法
handleReceive方法

  public Span handleReceive(TraceContext.Extractor extractor, Req request) {
    return handleReceive(extractor, request, request);
  }

  public  Span handleReceive(TraceContext.Extractor extractor, C carrier, Req request) {
    Span span = nextSpan(extractor.extract(carrier), request);
    if (span.isNoop()) return span;

    // all of the parsing here occur before a timestamp is recorded on the span
    span.kind(Span.Kind.SERVER);

    // Ensure user-code can read the current trace context
    Tracer.SpanInScope ws = tracer.withSpanInScope(span);
    try {
      parser.request(adapter, request, span);
    } finally {
      ws.close();
    }

    boolean parsedEndpoint = false;
    if (Platform.get().zipkinV1Present()) {
      zipkin.Endpoint.Builder deprecatedEndpoint = zipkin.Endpoint.builder().serviceName("");
      if ((parsedEndpoint = adapter.parseClientAddress(request, deprecatedEndpoint))) {
        span.remoteEndpoint(deprecatedEndpoint.build());
      }
    }
    if (!parsedEndpoint) {
      Endpoint.Builder remoteEndpoint = Endpoint.newBuilder();
      if (adapter.parseClientAddress(request, remoteEndpoint)) {
        span.remoteEndpoint(remoteEndpoint.build());
      }
    }
    return span.start();
  }

• 首先调用nextSpan(extractor.extract(carrier), request)从request中提取TraceContextOrSamplingFlags，并创建Span，并将Span的kind类型设置为SERVER
• 然后调用parser.request(adapter, request, span)，将request的内容，将span的name改为request的method即GET或者POST，而且会将当前请求的路径以Tag(http.path)写入Span中，这样我们就能在Zipkin的UI界面中能清晰的看出某个Span是发起了什么请求。
• 最后为Span设置Endpoint信息，并调用start设置开始时间

handleSend方法

  public void handleSend(@Nullable Resp response, @Nullable Throwable error, Span span) {
    if (span.isNoop()) return;

    // Ensure user-code can read the current trace context
    Tracer.SpanInScope ws = tracer.withSpanInScope(span);
    try {
      parser.response(adapter, response, error, span);
    } finally {
      ws.close();
      span.finish();
    }
  }

handleSend比较简单，调用parser.response(adapter, response, error, span)，会将HTTP状态码写入Span的Tag(http.status_code)中，如果有出错，则会将错误信息写入Tag(error)中
最后会调用Span的finish方法，而finish方法中，会调用Reporter的report方法将Span信息上报到Zipkin。

接着看下nextSpan方法

  Span nextSpan(TraceContextOrSamplingFlags extracted, Req request) {
    if (extracted.sampled() == null) { // Otherwise, try to make a new decision
      extracted = extracted.sampled(sampler.trySample(adapter, request));
    }
    return extracted.context() != null
        ? tracer.joinSpan(extracted.context())
        : tracer.nextSpan(extracted);
  }

从请求里提取的对象extracted（TraceContextOrSamplingFlags），如果没有sampled信息，则由HttpSampler的trySample方法来决定是否采样
如果extracted中含有TraceContext信息，则由tracer调用joinSpan，加入已存在的trace，这种情况一般是客户端代码使用将trace信息放入header，而服务端收到请求后，则自动加入客户端发起的trace中，所以当backend的请求运行到这段代码，会joinSpan
如果extracted中不含TraceContext信息，则由tracer调用nextSpan，这种情况一般是我们用户发起的请求，比如浏览器发起，则请求header中肯定是没有trace信息的，所以当frontend的请求运行到这段代码，会新建一个span

joinSpan方法

  public final Span joinSpan(TraceContext context) {
    if (context == null) throw new NullPointerException("context == null");
    if (!supportsJoin) return newChild(context);
    // If we are joining a trace, we are sharing IDs with the caller
    // If the sampled flag was left unset, we need to make the decision here
    TraceContext.Builder builder = context.toBuilder();
    if (context.sampled() == null) {
      builder.sampled(sampler.isSampled(context.traceId()));
    } else {
      builder.shared(true);
    }
    return toSpan(builder.build());
  }

  public Span newChild(TraceContext parent) {
    if (parent == null) throw new NullPointerException("parent == null");
    return nextSpan(TraceContextOrSamplingFlags.create(parent));
  }

在joinSpan方法中，会共享调用方的traceId，如果调用者没有传入sampled信息，则由服务端自己决定是否采样，即sampler.isSampled(context.traceId())

nextSpan方法

  public Span nextSpan(TraceContextOrSamplingFlags extracted) {
    TraceContext parent = extracted.context();
    if (extracted.samplingFlags() != null) {
      TraceContext implicitParent = currentTraceContext.get();
      if (implicitParent == null) {
        return toSpan(newRootContext(extracted.samplingFlags(), extracted.extra()));
      }
      // fall through, with an implicit parent, not an extracted one
      parent = appendExtra(implicitParent, extracted.extra());
    }
    long nextId = Platform.get().randomLong();
    if (parent != null) {
      return toSpan(parent.toBuilder() // copies "extra" from the parent
          .spanId(nextId)
          .parentId(parent.spanId())
          .shared(false)
          .build());
    }
    TraceIdContext traceIdContext = extracted.traceIdContext();
    if (extracted.traceIdContext() != null) {
      Boolean sampled = traceIdContext.sampled();
      if (sampled == null) sampled = sampler.isSampled(traceIdContext.traceId());
      return toSpan(TraceContext.newBuilder()
          .sampled(sampled)
          .debug(traceIdContext.debug())
          .traceIdHigh(traceIdContext.traceIdHigh()).traceId(traceIdContext.traceId())
          .spanId(nextId)
          .extra(extracted.extra()).build());
    }
    // TraceContextOrSamplingFlags is a union of 3 types, we've checked all three
    throw new AssertionError("should not reach here");
  }

在nextSpan方法中，首先找出合适的parent，当parent存在时，则新建一个child Span，否则返回new Span

到这里服务端接受到请求后，是如何记录Span信息的代码已经分析完毕，接下来我们看看作为客户端，我们是如何上报Span信息

FrontServlet

首先我们看到FrontServet中init方法里，我们初始化了OkHttpClient，并将TracingInterceptor拦截器添加到OkHttpClient的NetworkInterceptor拦截器栈中，然后还用CurrentTraceContext中的ExecutorService的包装方法，将Dispatcher中的ExecutorService包装后设置到OkHttpClient中。

package org.mozhu.zipkin.servlet;

import brave.http.HttpTracing;
import brave.okhttp3.TracingInterceptor;
import okhttp3.Dispatcher;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;

public class FrontendServlet extends HttpServlet {

    private final static Logger LOGGER = LoggerFactory.getLogger(FrontendServlet.class);

    private OkHttpClient client;

    @Override
    public void init(ServletConfig config) throws ServletException {
        super.init(config);
        HttpTracing httpTracing = (HttpTracing) config.getServletContext().getAttribute("TRACING");
        client = new OkHttpClient.Builder()
                .dispatcher(new Dispatcher(
                        httpTracing.tracing().currentTraceContext()
                                .executorService(new Dispatcher().executorService())
                ))
                .addNetworkInterceptor(TracingInterceptor.create(httpTracing))
                .build();
    }

    @Override
    protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        LOGGER.info("frontend receive request");
        Request request = new Request.Builder()
                .url("http://localhost:9000/api")
                .build();

        Response response = client.newCall(request).execute();
        if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);

        PrintWriter writer = resp.getWriter();
        writer.write(response.body().string());
        writer.flush();
        writer.close();
    }

}

public final class TracingInterceptor implements Interceptor {
  // ...

  final Tracer tracer;
  final String remoteServiceName;
  final HttpClientHandler handler;
  final TraceContext.Injector injector;

  TracingInterceptor(HttpTracing httpTracing) {
    if (httpTracing == null) throw new NullPointerException("HttpTracing == null");
    tracer = httpTracing.tracing().tracer();
    remoteServiceName = httpTracing.serverName();
    handler = HttpClientHandler.create(httpTracing, new HttpAdapter());
    injector = httpTracing.tracing().propagation().injector(SETTER);
  }
}

TracingInterceptor中依赖Tracer，TraceContext.Injector，HttpClientHandler，HttpAdapter。
- TraceContext.Injector - 将Trace信息注入到HTTP Request中，即放到Http headers中
- HttpClientHandler - 和HttpServerHandler对应，也是Http处理的核心组件，基本上所有和trace相关的操作均在此类中完成
- HttpAdapter - 能从Http request中获得各种数据，比如method，请求Path，header值等

  @Override public Response intercept(Chain chain) throws IOException {
    Request request = chain.request();
    Request.Builder requestBuilder = request.newBuilder();

    Span span = handler.handleSend(injector, requestBuilder, request);
    parseServerAddress(chain.connection(), span);
    Response response = null;
    Throwable error = null;
    try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
      return response = chain.proceed(requestBuilder.build());
    } catch (IOException | RuntimeException | Error e) {
      error = e;
      throw e;
    } finally {
      handler.handleReceive(response, error, span);
    }
  }

这里代码和TracingFilter中doFilter比较相似，是一个相反的过程
- 首先将trace信息注入到request中，并创建Span对象
- 然后调用chain.proceed(requestBuilder.build())来执行发送http请求
- 最后handler.handleReceive(response, error, span)

接下来看看HttpClientHandler的handleSend方法和handleReceive方法
handleSend方法

  public Span handleSend(TraceContext.Injector injector, Req request, Span span) {
    return handleSend(injector, request, request, span);
  }

  public  Span handleSend(TraceContext.Injector injector, C carrier, Req request, Span span) {
    injector.inject(span.context(), carrier);
    if (span.isNoop()) return span;

    // all of the parsing here occur before a timestamp is recorded on the span
    span.kind(Span.Kind.CLIENT);

    // Ensure user-code can read the current trace context
    Tracer.SpanInScope ws = tracer.withSpanInScope(span);
    try {
      parser.request(adapter, request, span);
    } finally {
      ws.close();
    }

    boolean parsedEndpoint = false;
    if (Platform.get().zipkinV1Present()) {
      zipkin.Endpoint.Builder deprecatedEndpoint = zipkin.Endpoint.builder()
          .serviceName(serverNameSet ? serverName : "");
      if ((parsedEndpoint = adapter.parseServerAddress(request, deprecatedEndpoint))) {
        span.remoteEndpoint(deprecatedEndpoint.serviceName(serverName).build());
      }
    }
    if (!parsedEndpoint) {
      Endpoint.Builder remoteEndpoint = Endpoint.newBuilder().serviceName(serverName);
      if (adapter.parseServerAddress(request, remoteEndpoint) || serverNameSet) {
        span.remoteEndpoint(remoteEndpoint.build());
      }
    }
    return span.start();
  }

• 首先调用injector.inject(span.context(), carrier)将Trace信息注入request中，并将Span的kind类型设置为CLIENT
• 然后调用parser.request(adapter, request, span)，将request的内容，将span的name改为request的method即GET或者POST，而且会将当前请求的路径以Tag(http.path)写入Span中，这样我们就能在Zipkin的UI界面中能清晰的看出某个Span是发起了什么请求。
• 最后为Span设置Endpoint信息，并调用start设置开始时间

handleReceive方法

  public void handleReceive(@Nullable Resp response, @Nullable Throwable error, Span span) {
    if (span.isNoop()) return;
    Tracer.SpanInScope ws = tracer.withSpanInScope(span);
    try {
      parser.response(adapter, response, error, span);
    } finally {
      ws.close();
      span.finish();
    }
  }

handleReceive比较简单，当客户端收到服务端的响应后handleReceive方法会被调用，即调用parser.response(adapter, response, error, span)，会将HTTP状态码写入Span的Tag(http.status_code)中，如果有出错，则会将错误信息写入Tag(error)中
最后会调用Span的finish方法，而finish方法中，会调用Reporter的report方法将Span信息上报到Zipkin。

BackendServlet

最后看看BackendServlet，在收到请求后，将请求的header中参数user-name取出，添加到时间戳字符串尾部，并返回。
在上一篇博文中，我们看到如果我们向Frontend发送的请求中带有header user-name参数，Frontend会将这个值传递给Backend，然后backend会将它放到响应字符串中返回，以表明接收到该header。

package org.mozhu.zipkin.servlet;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Date;

public class BackendServlet extends HttpServlet {

    private final static Logger LOGGER = LoggerFactory.getLogger(BackendServlet.class);

    @Override
    protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        LOGGER.info("backend receive request");
        String username = req.getHeader("user-name");
        String result;
        if (username != null) {
            result = new Date().toString() + " " + username;
        } else {
            result = new Date().toString();
        }
        PrintWriter writer = resp.getWriter();
        writer.write(result);
        writer.flush();
        writer.close();
    }

}

至此，我们已经分析完Brave是如何在普通的web项目中使用的，分析了TracingFilter拦截请求处理请求的逻辑，也分析了OkHttpClient是如何将Trace信息放入request中的。
后面博文中，我们还会继续分析Brave和Spring Web项目的整合方法。