微服务治理之分布式链路追踪--4.opentelemetry实战

微服务治理之分布式链路追踪–4.opentelemetry实战

本节是基于zipkin分布式追踪系统搭建，因为对 scala 和 play framework 2 框架不熟悉，所以，没有采用opentelemetry 的sdk来实现play框架的追踪功能。scala有zio-telemetry库来实现opentelemetry方案，但是本人不太了解如何在play框架中引入zio-telemetry，如果有大佬熟悉这块可以留言评论。

---

---

前言

本次实验backend采用的是 jaeger，协议使用的是opentelemetry协议。play框架引入的是官方库: opentelemetry-java

实验环境：

• goframe: 1.16.6
• playframework: 2.8.8
• scala: 2.13.5
• golang: 1.17

---

一、环境构建

说明：jaeger(官网) 部署有多种方式，开发阶段可以采用podman单机部署all-in-one镜像。本次实验采用docker部署jaeger。

1. jaeger搭建（podman单机）

根据官网的operator部署方案部署jaeger。其中，jaeger的后端存储采用的是es.

jaeger.yaml:

podman run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.27

---

二、代码解析

1. goframe 实现

废话不多说，直接上关键代码。

代码如下（示例）：

go.mod:

require (
	github.com/gogf/gf v1.16.6
	go.opentelemetry.io/otel v1.0.0-RC2
	go.opentelemetry.io/otel/exporters/jaeger v1.0.0-RC2
	go.opentelemetry.io/otel/sdk v1.0.0-RC2
)

hello.go(controller)

package api

import (
	"context"

	"github.com/gogf/gf/frame/g"
	"github.com/gogf/gf/net/ghttp"
	"github.com/gogf/gf/net/gtrace"
)

var Hello = helloApi{}

type helloApi struct{}

// Index is a demonstration route handler for output "Hello World!".
func (*helloApi) Index(r *ghttp.Request) {
	g.Log().Line().Skip(1).Infof("trace-service-b msg: %s", "hello")
	ctx, span := gtrace.NewSpan(r.Context(), "Index")
	defer span.End()

	trace_request_go(ctx)
	r.Response.Writeln("Hello: world")
}
func trace_request_go(ctx context.Context) {
	ctx, span := gtrace.NewSpan(ctx, "trace_request_go")
	defer span.End()

	ctx = gtrace.SetBaggageValue(ctx, "name", "john")

	client := g.Client().Use(ghttp.MiddlewareClientTracing)

	content := client.Ctx(ctx).GetContent("http://127.0.0.1:9000/index")

	g.Log().Ctx(ctx).Line().Info(content)
}

注：9411是zipkin的端口

2. play framework实现

代码目录：

废话不多说，直接上关键代码。

build.sbt.：

name := "otel_test"
 
version := "1.0" 
      
lazy val `otel_test` = (project in file(".")).enablePlugins(PlayScala)

      
resolvers += "Akka Snapshot Repository" at "https://repo.akka.io/snapshots/"
      
scalaVersion := "2.13.5"

libraryDependencies ++= Seq(
  jdbc ,
  ehcache ,
  ws ,
  specs2 % Test ,
  guice,
  "io.opentelemetry" % "opentelemetry-sdk" % "1.7.0",
  "io.opentelemetry" % "opentelemetry-api" % "1.7.0",
  "io.opentelemetry" % "opentelemetry-exporter-jaeger" % "1.7.0",
  "org.apache.commons" % "commons-lang3" % "3.12.0",
  "io.grpc" % "grpc-netty" % "1.41.0"
)

application.yaml.：

# https://www.playframework.com/documentation/latest/Configuration
play.http.filters=filters.TracingFilter
play.filters {
  hosts.allowed = ["localhost:9000"]
}

play.http.secret.key = "qE50cTX2ZZ4PI1xcFZNC@;yjQKM=V5608O=HYP;EX6p4h^T_HIpDQPIUlEl7N1QE"

play.filters.disabled += play.filters.csrf.CSRFFilter
play.filters.disabled += play.filters.hosts.AllowedHostsFilter
# play.filters.disabled += play.filters.headers.SecurityHeadersFilter

otel{
  trace.host = "localhost"
  trace.port = 14250
}

logback.xml.：

<!-- https://www.playframework.com/documentation/latest/SettingsLogger -->
<configuration>

  <conversionRule conversionWord="coloredLevel" converterClass="play.api.libs.logback.ColoredLevel" />

  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>${application.home:-.}/logs/application.log</file>
    <encoder>
      <pattern>%date [%level] from %logger in %thread - %message%n%xException</pattern>
    </encoder>
  </appender>

  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>%coloredLevel %logger{15} - %message%n%xException{10}</pattern>
    </encoder>
  </appender>

  <appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="FILE" />
  </appender>

  <appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="STDOUT" />
  </appender>

  <logger name="play" level="INFO" />
  <logger name="application" level="DEBUG" />
  <logger name="opentelemetry" level="DEBUG" />
  <logger name="controllers" level="DEBUG" />
  <logger name="clients" level="DEBUG" />
  <logger name="services" level="DEBUG" />

  <!-- Off these ones as they are annoying, and anyway we manage configuration ourselves -->
  <logger name="com.avaje.ebean.config.PropertyMapLoader" level="OFF" />
  <logger name="com.avaje.ebeaninternal.server.core.XmlConfigLoader" level="OFF" />
  <logger name="com.avaje.ebeaninternal.server.lib.BackgroundThread" level="OFF" />
  <logger name="com.gargoylesoftware.htmlunit.javascript" level="OFF" />

  <root level="WARN">
    <appender-ref ref="ASYNCFILE" />
    <appender-ref ref="ASYNCSTDOUT" />
  </root>

</configuration>

HomeController.scala.：

package controllers

import io.opentelemetry.api.trace.Span

import javax.inject._
import play.api.mvc._
import play.api.libs.json.Json
import opentelemetry.core.{OtelTracer, TraceImplicits}
import play.api.Logging
import play.api.libs.ws._

/**
 * This controller creates an `Action` to handle HTTP requests to the
 * application's home page.
 */
@Singleton
class HomeController @Inject()(cc: ControllerComponents)(val tracer: OtelTracer) extends AbstractController(cc)
  with Logging with TraceImplicits {

  /**
   * Create an Action to render an HTML page with a welcome message.
   * The configuration in the `routes` file means that this method
   * will be called when the application receives a `GET` request with
   * a path of `/`.
   */
  def index: Action[AnyContent] = Action {implicit request: Request[AnyContent] =>
      val childSpan = tracer.send(request2trace.span,null);
      logger.info("Hello function: index")
      test(childSpan,request)
      Ok(Json.obj("result" -> "ok"))
  }

  def test(parentSpan: Span, rh: RequestHeader): Unit = {
      logger.info("Hello function: test")
      var span = tracer.newSpan("play_test",parentSpan);
      tracer.send(span,null);
      tracer.wsRequest("play_ws_test",span,"http://localhost:5000/model/metadata")
  }

}

OtelFilter.scala.：

package opentelemetry.play

import akka.stream.Materializer

import javax.inject.Inject
import play.api.mvc.{Filter, Headers, RequestHeader, Result}
import play.api.routing.Router

import scala.concurrent.Future
import scala.util.Failure
import play.api.Logger
import opentelemetry.core.OtelTracer

/** A Zipkin filter.
 *
 * This filter is that reports how long a request takes to execute in Play as a
 * server span. The way to use this filter is following:
 * {{{
 * class Filters @Inject() (
 *   zipkin: ZipkinTraceFilter
 * ) extends DefaultHttpFilters(zipkin)
 * }}}
 *
 * @param tracer
 *   a Zipkin tracer
 * @param mat
 *   a materializer
 */
class OtelFilter @Inject()(tracer: OtelTracer)(implicit val mat: Materializer) extends Filter {

  import tracer.executionContext

  val logger: Logger = Logger(this.getClass)
  private val reqHeaderToSpanName: RequestHeader => String =
    OtelFilter.ParamAwareRequestNamer
  def apply(nextFilter: RequestHeader => Future[Result])(req: RequestHeader): Future[Result] = {
    logger.info(s"RequestHeader.header is ${req.headers}")
    val rootSpan = tracer.received(
      spanName = reqHeaderToSpanName(req),
      span = tracer.newSpan(req.headers)((headers, key) => headers.get(key))
    )
    val result = nextFilter(req.withHeaders(new Headers(
      _headers = (req.headers.toMap.view.mapValues(_.headOption getOrElse "")).toSeq
    )))
    result.onComplete {
      case Failure(t) => tracer.send(rootSpan,"failed" -> s"Finished with exception: ${t}")
      case _ => tracer.send(rootSpan,null)
    }
    result
  }
}

object OtelFilter {
  val ParamAwareRequestNamer: RequestHeader => String = { reqHeader =>
    import org.apache.commons.lang3.StringUtils
    val pathPattern = StringUtils.replace(
      reqHeader.attrs
        .get(Router.Attrs.HandlerDef)
        .map(_.path)
        .getOrElse(reqHeader.path),
      "<[^/]+>",
      ""
    )
    s"${reqHeader.method} - $pathPattern"
  }
}

OtelTracer.scala.：

package opentelemetry.core

import javax.inject.Inject
import com.typesafe.config.Config
import io.grpc.ManagedChannelBuilder
import io.opentelemetry.api.OpenTelemetry
import io.opentelemetry.api.common.Attributes
import io.opentelemetry.api.trace.{Span, SpanKind}
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator
import io.opentelemetry.context.Context
import io.opentelemetry.context.propagation.{ContextPropagators, TextMapGetter, TextMapSetter}
import io.opentelemetry.exporter.jaeger.JaegerGrpcSpanExporter
import io.opentelemetry.sdk.OpenTelemetrySdk
import io.opentelemetry.sdk.resources.Resource
import io.opentelemetry.sdk.trace.SdkTracerProvider
import io.opentelemetry.semconv.resource.attributes.ResourceAttributes
import play.api.Logger
import play.api.libs.ws.{WSClient, WSRequest}
import play.api.mvc.Headers
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor

import java.lang
import java.util.concurrent.TimeUnit
import scala.concurrent.ExecutionContext
import scala.language.postfixOps
import scala.util.{Failure, Success, Try}
import collection.JavaConverters._
import scala.collection.mutable

class OtelTracer @Inject()(val config: Config, val ws: WSClient, implicit val executionContext: ExecutionContext){

  val logger: Logger = Logger(this.getClass)

  private[core] var opentelemetry :OpenTelemetry = initOpenTelemetry();

  private[core] var tracer = opentelemetry.getTracer("opentelemetry.core.OtelSclalTracer")

  def trace[A](traceName: String, tags: (String, String)*)(f: TraceData => A)(implicit parentData: TraceData): A = {
    val childSpan = tracer.spanBuilder(traceName).setParent(Context.current().`with`(parentData.span)).setSpanKind(SpanKind.CLIENT).startSpan
    tags.foreach { case (key, value) => childSpan.setAttribute(key, value) }

    Try(f(TraceData(childSpan))) match {
      case Failure(t) =>
        childSpan.setAttribute("failed", s"Finished with exception: ${t.getMessage}")
        childSpan.end()
        throw t
      case Success(result) =>
        childSpan.end()
        result
    }
  }

  def initOpenTelemetry(): OpenTelemetry ={
    logger.info(s"otel.trace.host: ${config.getString("otel.trace.host")}, otel.trace.port: ${config.getString("otel.trace.port")}")

    val jaegerChannel = ManagedChannelBuilder.forAddress(config.getString("otel.trace.host"), config.getInt("otel.trace.port"))
      .usePlaintext().build
    // Export traces to Jaeger
    val jaegerExporter = JaegerGrpcSpanExporter.builder.setChannel(jaegerChannel).setTimeout(30, TimeUnit.SECONDS).build

    val serviceNameResource = Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "otel-jaeger-example"))

    val spanProcessor = BatchSpanProcessor.builder(jaegerExporter).build

    // Set to process the spans by the Jaeger Exporter
    val tracerProvider = SdkTracerProvider.builder.addSpanProcessor(spanProcessor)
      .setResource(Resource.getDefault.merge(serviceNameResource)).build
    val openTelemetry = OpenTelemetrySdk.builder.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance))
      .setTracerProvider(tracerProvider).build

    // it's always a good idea to shut down the SDK cleanly at JVM exit.
//    Runtime.getRuntime.addShutdownHook(new Thread()(tracerProvider.close()))
    Runtime.getRuntime.addShutdownHook(new Thread(new Runnable {
      override def run(): Unit = tracerProvider.close()
    }))
//    spanProcessor.shutdown
    openTelemetry
  }


  /**
   * Starts the server span. When a server received event has occurred, calling this.
   *
   * @param spanName the string name for this span
   * @param span the span to start
   * @return the server span that will later signal to the Zipkin
   */
  def received(spanName: String, span: Span): Span = {
    tracer.spanBuilder(spanName).setSpanKind(SpanKind.SERVER).startSpan
  }

  /**
   * Reports the server span complete. When a server sent event has occurred, calling this.
   *
   * @param span the server span to report
   * @param tags tags to add to the span
   * @return the server span itself
   */
  def send(span: Span, tags: (String, String)*): Span = {
    if (tags.apply(0) != null){
      tags.foreach { case (key, value) => span.setAttribute(key, value) }
    }
    span.end()
    span

  }

  /**
   * Creates a span from request headers. If there is no existing trace, creates a new trace.
   * Otherwise creates a new span within an existing trace.
   *
   * @param headers the HTTP headers
   * @param getHeader optionally returns the first header value associated with a key
   * @tparam A the HTTP headers type
   * @return a new span created from request headers
   */
  def newSpan[A](headers: A)(getHeader: (A, String) => Option[String]): Span = {
    val textMapPropagator = this.opentelemetry.getPropagators.getTextMapPropagator
    val context = textMapPropagator.extract(Context.current, headers, getter)
    val span = tracer.spanBuilder("root-span").setParent(context).setSpanKind(SpanKind.SERVER).startSpan
    span
  }

  /**
   * Creates a span from a parent context.
   * If the parent span is None, creates a new trace.
   *
   * @param parent the parent context
   * @return a new span created from the parent context
   */
  def newSpan(name:String,parent: Span): Span = {
      tracer.spanBuilder(name).setParent(Context.current().`with`(parent)).setSpanKind(SpanKind.CLIENT).startSpan
  }


  def wsRequest(spanName: String, parentSpan: Span,url:String): Span = {
    val carrier: mutable.Map[String, String] = mutable.Map().empty
    val textMapPropagator = this.opentelemetry.getPropagators.getTextMapPropagator
    val span = tracer.spanBuilder(spanName).setParent(Context.current.`with`(parentSpan)).setSpanKind(SpanKind.CLIENT).startSpan
    var request: WSRequest = ws.url(url)

    textMapPropagator.inject(Context.current.`with`(span), carrier, setter)
    carrier.foreachEntry((k,v)=>{
      request = request.addHttpHeaders(k->v)
    })
    request.withFollowRedirects(true).get
    span.end()
    span
  }

  val setter: TextMapSetter[mutable.Map[String, String]] = (carrier, key, value)=>{carrier.update(key, value)}

  //      (carrier, key, value) => ()carrier.getHeaders.set(key, value)
  private val getter: TextMapGetter[Any] = new TextMapGetter[Any]() {
    override def keys(carrier: Any): lang.Iterable[String] = carrier.asInstanceOf[Headers].keys.asInstanceOf[Iterable[String]].asJava

    override def get(carrier: Any, key: String): String = { //            System.out.println("key is " + key);
      assert(carrier != null)
      //            System.out.println("otel.trace is " + carrier.headers().getAll(key));
      if (carrier.asInstanceOf[Headers].hasHeader(key)) { //                System.out.println("otel.trace is " + carrier.headers().getAll(key).apply(0));
        return carrier.asInstanceOf[Headers].getAll(key).head
      }
      ""
    }
  }
}

TraceImplicits.scala.：

package opentelemetry.core

import io.opentelemetry.api.trace.Span
import play.api.mvc.RequestHeader

trait TraceImplicits {

  // for injection
  val tracer: OtelTracer

  /**
   * Creates a trace data including a span from request headers.
   *
   * @param req the HTTP request header
   * @return the trace data
   */
  implicit def request2trace(implicit req: RequestHeader): TraceData = {
    TraceData(
      span = tracer.newSpan(req.headers)((headers, key) => headers.get(key))
    )
  }

  /**
   * Creates a trace data including a span from request headers for Akka actor.
   *
   * @param req the HTTP request header
   * @return the trace data
   */
//  implicit def request2actorTrace(implicit req: RequestHeader): ActorTraceData = {
//    val span = tracer.newSpan(req.headers)((headers, key) => headers.get(key))
//    val oneWaySpan = tracer.newSpan(Some(span.context())).kind(Span.Kind.CLIENT)
//    ActorTraceData(span = oneWaySpan)
//  }

}

TracingFilter.scala.：

package filters

import opentelemetry.play.OtelFilter

import javax.inject.Inject
import play.api.http.DefaultHttpFilters

class TracingFilter @Inject()(traceFilter: OtelFilter) extends DefaultHttpFilters(traceFilter) {}

---

3. jaeger展示

trace timeline：

trace graph：

---

总结

本次实验演示了基于play framework和goframe框架构建基础的微服务系统，然后基于opentelemetry分布式链路追踪系统进行服务间的链路追踪。通过这次实验了解了大概的分布式链路追踪系统是如何构建，运行。但还遗留了以下几个问题：

1. 我司线上服务是run on k8s ，并且用到了istio服务网格，链路追踪系统在服务网格的微服务治理体系中如何发挥出该有的价值。
2. 微服务的可观察性这块是个比较大的课题，logging、tracing、metrics三者之间如何构建、运行、协调可以做个专题来研究。