云原生现场分析 Part4 —— eBPF 跟踪 Istio/Envoy 之 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程

image.png

注，原文来自 https://blog.mygraphql.com/zh/posts/low-tec/trace/trace-istio/trace-istio-part4/ 。如你看到的转载图片不清，请回到原文。

• 为何要了解 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程
• 逆向工程与云原生现场分析 系列介绍
• HTTP 反向代理的总流程
  • Downstream Read Request 模块协作
  • Upstream Write Request 模块协作
  • Upstream Read Response 模块协作
  • Downstream Write Response 模块协作
• BPF 脚本输出
• BPF 脚本
• 结尾

为何要了解 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程

想不到还真坚持写到 Part4 了。不管你信不信，每一 part 的“为何”一节，是我觉得最难写的 :) 。如果你是第一次读这个系列，不用担心，每一节都是比较独立的。

不出意外的话，你之前看到的 Envoy 简介是这样描述 Envoy 的特性的：

• C++ 编写，原生底层，没有 GC stop the world，所以性能优异
• 异步事件驱动，多路复用，完美解决 C10k problem
• 由于单线程负责多连接，在大连接量时，减少了大量线程的内存开销，和 CPU 上下文切换开销

这些描述，当然有其合理性。但很多远看很美丽的事物，在微距放大后，可能有很多有意思，有价值的东西。我相信只要分析得足够细致，总可能针对我们实际运行环境和流量特性对作一些有意义的优化。可能只是修改一个 Envoy / Kernel 的配置，也可能是修改一行 Envoy 的代码。或你的应用的流程相关行为，如每次写 socket 的 buffer 数据包的大小。

而这些，都需要建立在了解实现细节的基础上。除非觉得运气或经验值特别好，可以瞎猜。

逆向工程与云原生现场分析 系列介绍

开始前先做个预告，逆向工程与云原生现场分析 系列（将）包括：

• Part 1: eBPF 跟踪 Istio/Envoy 之学步
• Part 2: eBPF 跟踪 Istio/Envoy 之启动、监听与线程负载均衡
• Part 3: eBPF 跟踪 Istio/Envoy 之 downstream TCP 连接 accept、TLS 握手 与 filter_chain 的选择
• Part 4: eBPF 跟踪 Istio/Envoy 之 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程 （本文）
• Part 5: eBPF 跟踪 Istio/Envoy 之 L3/4 Network Fitler 互动机制
• Part 6: eBPF 跟踪 Istio/Envoy 之 http filter
• Part 7: eBPF 跟踪 Istio/Envoy 之 http router
• Part 8: eBPF 跟踪 Istio/Envoy 之 cluster、 connection pool 与 outbound load balance

在系列中，我将演示如何让 bpftrace "读" 懂运行期的由 C++ 11 编写成的 envoy 进程中的对象数据。
为免吓跑人，还是老套路，多图少代码，不过有的图有点点复杂。程序员大叔开始讲故事了。

HTTP 反向代理的总流程

整体看，Socket 事件驱动的 HTTP 反向代理总流程如下：

image.png

图中看出，有4种事件驱动了整个流程。后面几节会逐个分析。

为免一下子进入各个步骤细节而让人迷途，现在先看看所有步骤的总流程：

image.png

图：Istio/Envoy 模块协作总图

Downstream Read Request 模块协作

image.png

图：Downstream Read-Ready 模块协作

大概说明一下流程：

1. upstream socket 可读回调
2. Http::ConnectionManagerImpl 读取 socket，增量放入 Http1::ConnectionImpl
3. Http1::ConnectionImpl 调用 nghttp2 增量解释 HTTP 请求
4. 如果 nghttp2 认为已经 完整读取了 HTTP Request 请求，则调用 Http::ServerConnection::onMessageCompleteBase()
5. Http::ServerConnection::onMessageCompleteBase() 首先 停止 downstream ReadReady 监听
6. Http::ServerConnection::onMessageCompleteBase() 调用 Http::FilterManager ，发起 http filter chain 的 decodeHeaders 迭代流程
7. 一般，http filter chain 的最后一个 http filter 是 Router::Filter ，Router::Filter::decodeHeaders() 被调用
8. Router::Filter::decodeHeaders() 的逻辑就见下图了。

image.png

图：Downstream Request Router 模块协作

大概说明一下流程：

1. Router::Filter ，Router::Filter::decodeHeaders() 被调用
2. 根据配置的 Router 规则，匹配到 Cluster
3. 如 Cluster 连接池对象不存在，则新建
4. 新建 Envoy::Router::UpstreamRequest 对象。
5. 调用 Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream) ， encode HTTP header
6. 经过一系列的负载均衡算法，匹配到 upstream 的 host(endpoint)
7. 发现到选定的 upstream host 的连接不足，则：
   1. 打开一新的 socket fd(未连接)
   2. 注册 upstream socket FD 的 WriteReady / Connected 事件。 准备在事件回调时写 upstream request
   3. 用 socket fd 发起到 upstream host 的异步连接请求
8. 关联 downstream 与 upstream fd

Upstream Write Request 模块协作

[图片上传失败...(image-5b18c8-1649291572197)]
大概说明一下流程：

1. upstream socket write ready 回调
2. 发现是连接成功回调，关联 upstream socket 到 ConnectionPool::ActiveClient
3. upstream socket write ready 回调
4. 发现是连接可写，写入 upstream HTTP request

Upstream Read Response 模块协作

image.png

Downstream Write Response 模块协作

image.png

BPF 脚本输出

以上图片，不是只读源码就得出的。是需要通过 bpf 埋点观察：

1. 记录 downstream FD(socket 的 file descriptor)，可以认为是进程中的 socket id
2. 埋点进程级和 kernel 级的相关函数入口和出口，观察其出入参与 stack
3. 关联起 downstream FD 与 upstream FD

其中当然有很多细节，不过我不打算一个个讲了，实在有需要用，想深入了解的同学可以联系我探讨。下一节的 bpf 代码是最好的细节。

https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.3.log

#### Downstream Read Request: accept downstream socket， 可见 downstream socket 的 FD=41

OS handshaked TCP:
01:22:11 4215   wrk:worker_0   172.30.207.129                          58708 172.21.206.232                          15006 0/4096
sys_exit_accept4 fd=41

        accept4+96
        Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217
 

***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_

***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_

***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE

***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE

***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE

***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv

***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89

        recv+108


***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm

#### Downstream Read Request: 选择 network filter chain
***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Ev
comm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12

        Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107

#### Downstream Read Request: 监听 upstream FD 事件
***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function, Envoy::Event::FileTriggerType, unsigned int)+362
        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function, Envoy::Event::FileTriggerType, unsigned int)+284
        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function, Envoy::Event::FileTriggerType, unsigned int)+126
        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr >&&, std::__1::unique_ptr >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
        Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr >&&, std::__1::unique_ptr >&&, Envoy::StreamInfo::StreamInfo&, bool)+107
        Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr >&&, std::__1::unique_ptr >&&, Envoy::StreamInfo::StreamInfo&)+70
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr >&&, std::__1::unique_ptr >)+307
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr)+0
        std::__1::__function::__func, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector, std::__1::allocator > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr >&&, std::__1::unique_ptr >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr)+0
        virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr)+62
        std::__1::__function::__func, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector, std::__1::allocator > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr >&&, std::__1::unique_ptr >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr)+0
        std::__1::__function::__func, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector, std::__1::allocator > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr >&&, std::__1::unique_ptr >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr >&&, std::__1::unique_ptr >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100 
***************************

#### Downstream Read Request: 读 HTTP Request
***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005

#### Downstream Read Request: downstream 的 HTTP Reuest 已经完全读取（onMessageComplete()）。停止监听 downstream 的 ReadReady 事件
***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Network::ConnectionImpl::readDisable(bool)+1077
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

#### Downstream Read Request: 监听新建立的 upstream socket 的 FD=44 ， 这里 socket 还处于未连接状态。现在监听 upstream socket 的 所有事件。以备后面异步连接成功时收到回调
***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function, Envoy::Event::FileTriggerType, unsigned int)+362
        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function, Envoy::Event::FileTriggerType, unsigned int)+284
        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function, Envoy::Event::FileTriggerType, unsigned int)+126
        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr >&&, std::__1::unique_ptr >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
        Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr const&, std::__1::shared_ptr const&, std::__1::unique_ptr >&&, std::__1::shared_ptr, std::__1::allocator > > > const&)+283
        Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr, std::__1::shared_ptr, std::__1::unique_ptr >&&, std::__1::shared_ptr, std::__1::allocator > > > const&)+75
        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr)+561
        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr) const+126
        non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr) const+21
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr)+0
        Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr >&&, std::__1::shared_ptr, Envoy::Event::Dispatcher&)+357
        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr >&&, std::__1::shared_ptr, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70
        std::__1::__function::__func, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0    : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4. 

#### Downstream Read Request: 发现异步 upstream 连接
connecting to 8080:
01:23:36 4215     wrk:worker_0     127.0.0.6                               36383  172.21.206.232                          8080  

        connect+75
        Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr)+104
        Envoy::Network::SocketImpl::connect(std::__1::shared_ptr)+64
        Envoy::Network::ClientConnectionImpl::connect()+650
        Envoy::Http::CodecClient::connect()+543
        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr >&&, std::__1::shared_ptr, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694
        std::__1::__function::__func, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr, std::__1::allocator > > > const&, std::__1::shared_ptr const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

***** elapsed=1632608064: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1632618881: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1145219, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=92 
***************************


***** elapsed=1632635526: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1632654517: tid=5327,comm=wrk:worker_0: END:EventFired

#### Upstream Write Request: 异步 upstream 连接成功，回调。
***** elapsed=1632659321: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET

comm:wrk:worker_0,tid:5327: ConnPoolImplBase*attachStreamToClient: uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy14ConnectionPool16ConnPoolImplBase20attachStreamToClientERNS0_12ActiveClientERNS0_13AttachContextE 

        Envoy::ConnectionPool::ConnPoolImplBase::attachStreamToClient(Envoy::ConnectionPool::ActiveClient&, Envoy::ConnectionPool::AttachContext&)+0
        Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent(Envoy::ConnectionPool::ActiveClient&, absl::string_view, Envoy::Network::ConnectionEvent)+497
        Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent)+59
        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+511
        Envoy::Network::ConnectionImpl::onWriteReady()+1648
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

#### Upstream Write Request: 向 upstream socket 写 HTTP Request
***** elapsed=1632741863: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=44, ret=262

        writev+77
        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
        Envoy::Network::ConnectionImpl::onWriteReady()+1876
        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+529
        Envoy::Network::ConnectionImpl::onWriteReady()+1648
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1632756069: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1632764448: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=129648, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=60 
***************************

#### Upstream Read Response: upstream socket ReadReady 回调
***** elapsed=1637821708: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET

#### Upstream Read Response: 读 upstream HTTP Response
***** elapsed=1637898855: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=4511

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1637915998: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=-11

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

***** elapsed=1638220184: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80002004

#### Upstream Read Response: 已经完整读取 upstream HTTP Response，开始监听 downstream 的 WriteReady 事件，准备写 downstream HTTP Response
***** elapsed=1638231319: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Network::ConnectionImpl::readDisable(bool)+938
        Envoy::Http::Http1::StreamEncoderImpl::~StreamEncoderImpl()+112
        non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onEncodeComplete()+54
        Envoy::Http::Http1::StreamEncoderImpl::endEncode()+166
        Envoy::Http::Http1::StreamEncoderImpl::encodeData(Envoy::Buffer::Instance&, bool)+340
        Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData(Envoy::Buffer::Instance&, bool)+679
        Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState)+2138
        Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase()+619
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7705
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        Envoy::Http::Http1::ClientConnectionImpl::dispatch(Envoy::Buffer::Instance&)+29
        Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
        Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92


***** elapsed=1638898463: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1638934709: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1123611, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
***************************


***** elapsed=1638955421: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1638975366: tid=5327,comm=wrk:worker_0: END:EventFired

#### Downstream Write Response: downstream 的 WriteReady 事件回调。
***** elapsed=1638980295: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET

#### Downstream Write Response: 写 downstream HTTP Response
***** elapsed=1639253752: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=41, ret=4631

        writev+77
        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
        Envoy::Network::ConnectionImpl::onWriteReady()+1876
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1639293010: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1639308346: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=354486, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
***************************


***** elapsed=1643785802: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1643855427: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=0

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function, absl::optional const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

***** elapsed=1643880789: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
sys_enter_close fd=41
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98d38758
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98996380

***** elapsed=1644006858: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1644056452: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=289169, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=52 
***************************

^C


@epoll_ctl_op[1]: EPOLL_CTL_ADD
@epoll_ctl_op[2]: EPOLL_CTL_DEL
@epoll_ctl_op[3]: EPOLL_CTL_MOD


@fd2sockopt[44, 6, 1, 1]: 1
@fd2sockopt[41, 6, 1, 1]: 1


@filterManagerImpl2fd[94143954060408]: 41
@filterManagerImpl2fd[94143950909952]: 41

BPF 脚本

https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.bt

#!usr/bin/bpftrace
/*
IMPORT-ENV: $ENVOY_PID
args: $1=ENVOY_PID
*/

/*

#export PODID=`sudo crictl  pods | grep fortio-server | egrep '\bReady' | awk '{print $1}'`
#export CONTAINERID=`sudo crictl ps | grep $PODID | grep istio-proxy |  awk '{print $1}'`

kubectl -n idm-mark exec -it fortio-server -c  istio-proxy  -- bash -c 'ls -l /proc/`pgrep envoy`/ns/pid'
lrwxrwxrwx 1 istio-proxy istio-proxy 0 Mar 31 09:18 /proc/15/ns/pid -> 'pid:[4026532799]'
pgrep envoy | tee /dev/fd/2 | xargs -L 1 -I '{}' sudo ls -l /proc/'{}'/ns 

export PID=4215
export SCRIPT_HOME=`pwd`
export bpftrace_image=cndt-bcc-ub

docker run -it --rm --init  --privileged --name bpftrace -h bpftrace \
    --pid host \
    --net host \
    -e SCRIPT_HOME=$SCRIPT_HOME \
    -e PID=$PID \
    -e ENVOY_PID=$PID \
    -e BT=trace-envoy-filter-router.bt \
    -v /etc/localtime:/etc/localtime:ro \
    -v /sys:/sys:rw \
    -v /usr/src:/usr/src:rw \
    -v /lib/modules:/lib/modules:ro \
    -v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \
    $bpftrace_image \
    ${SCRIPT_HOME}/warp-bt.sh $PID
*/


/*
1. BPF Map 说明:

1.1. range tid map
- @watchedWakeRound[tid]=tid
       - sys_exit_accept4
       - FileEventImpl*assignEvents*
- @fdFired[tid]=$fd;
       - uprobe: *FileEventImpl*assignEvents*
       - uretprobe: *FileEventImpl*assignEvents*

1.2. fd map
- @fd2sockpair[$fd]=@sockpair[tid]
       - sys_exit_accept4
       - sys_enter_close
- @fd2sockopt[$fd, $level, $optname, $optval_int] = 1
       - sys_enter_setsockopt

- @filterManagerImpl2fd[arg0] = @fdFired[tid] // filterManagerImpl to downstream fd
- @fd2filterManagerImpl[$fd] = arg0 
- @fd2filterManagerImpl2[$fd] = arg0
- @downFD2upFD[$downFD] = $fd; //downstream fd to upstream fd
       - sys_enter_epoll_ctl
- @upFD[$fd] = $fd; //upstream fd list
       - sys_enter_epoll_ctl

1.3. temp tid map
- @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
- @sockpair_exist[tid]=1
- @watchedIo[tid]=fd

*/

#include 
#include 
#include 
#include 
// #include 

// The event argument describes the object linked to the file descriptor fd.  The struct epoll_event is defined as:
// struct epoll_data_t {
//     int          fd;
// };

// struct epoll_event {
//     //a bit mask composed by ORing together zero or more of the following available event types
//     uint32_t     events;      /* Epoll events */

//     struct epoll_data_t data;        /* User data variable */
// };



BEGIN
{
    printf("Tracing Envoy. Hit Ctrl-C to end.\n");
       printf("#define EPOLL_CTL_ADD 1\n #define EPOLL_CTL_DEL 2\n #define EPOLL_CTL_MOD 3\n");

       @epoll_ctl_op[1]="EPOLL_CTL_ADD";
       @epoll_ctl_op[2]="EPOLL_CTL_DEL";
       @epoll_ctl_op[3]="EPOLL_CTL_MOD";


}

/*
获取 accept 连接时的本地和对端地址。并标记本轮 wakeup 需要监控
 */
kretprobe:inet_csk_accept
/pid==$1 /
{
    $sk = (struct sock *)retval;
    $inet_family = $sk->__sk_common.skc_family;

    if ($inet_family == AF_INET || $inet_family == AF_INET6) {
        // initialize variable type:
        $daddr = ntop(0);
        $saddr = ntop(0);
        if ($inet_family == AF_INET) {
            $daddr = ntop($sk->__sk_common.skc_daddr);
            $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
        } else {
            printf("not support IPv6.\n");
            return;
        }
        
        $lport = $sk->__sk_common.skc_num;

              // printf("accept(), port=%d\n", $lport);

              //only watch listen port 15006
        if( 15006 != $lport ) { 
            return;
        }

        $dport = $sk->__sk_common.skc_dport;
        $qlen  = $sk->sk_ack_backlog;
        $qmax  = $sk->sk_max_ack_backlog;

        // Destination port is big endian, it must be flipped
        $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);

              printf("OS handshaked TCP:\n");
        time("%H:%M:%S ");
        printf("%-6d %-14s ", pid, comm);
        printf("%-39s %-5d %-39s %-5d ", $daddr, $dport, $saddr,
            $lport);
        printf("%d/%d\n", $qlen, $qmax);

        @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
        @sockpair_exist[tid]=1;

              @watchedWakeRound[tid]=tid;
    }
}

/**
 * 获取新连接的 FD 。并标记本轮 libevent 回调需要监控
 */
tracepoint:syscalls:sys_exit_accept4
/pid==$1 && @sockpair_exist[tid] /
{
    $fd = args->ret;
    if( $fd < 0 ) {
        return;
    }
    printf("sys_exit_accept4 fd=%d\n", $fd);
    @fd2sockpair[$fd]=@sockpair[tid];

       @fdFired[tid]=$fd;

    delete(@sockpair[tid]);
    delete(@sockpair_exist[tid]);
       printf("%s \n", ustack());
}

/**
 * 记录 FD 的 sockopt
 */
tracepoint:syscalls:sys_enter_setsockopt
/pid==$1/
{
       // socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92     
       $level = args->level;
       $fd = args->fd;

       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
              $optname = args->optname;
              $optval = args->optval;
              $optval_int = *$optval;
              $optlen = args->optlen;
              // printf("\n########## setsockopt() ##########\n");
              printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);
              @fd2sockopt[$fd, $level, $optname, $optval_int] = 1;
       }
}



/*
记录 epoll 监听的 FD

cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_ctl/format
 */
tracepoint:syscalls:sys_enter_epoll_ctl
/pid==$1/
{
       $fd=args->fd;
       $epollEvents = (uint32) (args->event->events);
       $op = args->op;
       if( @fd2sockpair[$fd].0 ) {

              printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                     elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);


              if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                     if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                            printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                     }
                     if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                            printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                     }
                     if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                            printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                     }
                     printf("%s\n", ustack(30));
              }
              
       } else {
              $downFD = @fdFired[tid];
              if( $downFD ) {//in downstream fd event callback, register upstream event trigger
                     @upFD[$fd] = $fd;
                     @downFD2upFD[$downFD] = $fd;

                     printf("***** elapsed=%d: tid=%d,comm=%s: register upstream event trigger:sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                            elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);


                     if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                            if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                            }
                            if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                            }
                            if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                            }
                            printf("%s\n", ustack(60));
                     }                     
              }
       }
}

/*
结束本轮的 wakeRound/runnableRound，并等待下一轮
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_wait/format
 */
tracepoint:syscalls:sys_enter_epoll_wait
/pid==$1/
{

       if( @watchedWakeRound[tid] ) {
              $runnableStartTime=@tid2Waketime[tid];
              if( $runnableStartTime ) {
                     $runableDuaration = elapsed - $runnableStartTime;
                     printf("\n******* WAKE-ROUND:END Summary *******\n");
                     printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait, runableDuaration=%d, tid2epollNrFdReady=%d\n", 
                            elapsed, tid, comm, $runableDuaration, @tid2epollNrFdReady[tid]);
                     $tid_last_epoll_wait_args = @last_epoll_wait_args[tid];
                     if( $tid_last_epoll_wait_args.0  ) {
                            printf("*** last_epoll_wait_args: epfd=%d, events=%d, maxevents=%d, timeout=%d \n", 
                                   $tid_last_epoll_wait_args.0, $tid_last_epoll_wait_args.1, $tid_last_epoll_wait_args.2, $tid_last_epoll_wait_args.3);
                     }
                     printf("***************************\n\n");
              }
       }

       delete(@tid2Waketime[tid]);
       delete(@watchedWakeRound[tid]);
       delete(@tid2epollNrFdReady[tid]);

       //read in next sys_enter_epoll_wait
       @last_epoll_wait_args[tid]=(args->epfd, args->events, args->maxevents, args->timeout);
}

/*
开始本轮 wakeRound/runnableRound
cat /sys/kernel/debug/tracing/events/syscalls/sys_exit_epoll_wait/format
 */
tracepoint:syscalls:sys_exit_epoll_wait
/pid==$1/
{
       // printf("\n***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait\n", elapsed, tid, comm);

       // printf("epfd: 0x%08lx, events: 0x%08lx, maxevents: 0x%08lx, timeout: 0x%08lx \n", 
       //        ((args->epfd)), ((args->events)), ((args->maxevents)), ((args->timeout)) );

       @tid2Waketime[tid]=elapsed;
       @tid2epollNrFdReady[tid]=args->ret;
}

/*
记录本轮 epoll 事件触发的 FD 级回调开始

Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int fd, short events, void* fileEventImplThis)  !!!NOT!!: FileEventImpl::assignEvents()
C++11 Lambda expressions:
event_assign( &raw_event_, base, fd_, xxx, [](evutil_socket_t, short what, void* arg)-> void {}, this );

event_assign(): https://libevent.org/doc/event_8h.html#a3e49a8172e00ae82959dfe64684eda11
       event_assign (   struct event *  ev,
              struct event_base *   base,
              evutil_socket_t   fd,
              short     events,
              event_callback_fn     callback,
              void *    callback_arg 
       )    

       https://libevent.org/doc/event_8h.html#aed2307f3d9b38e07cc10c2607322d758
       typedef void(* event_callback_fn) (evutil_socket_t, short, void *)
              fd    An fd or signal
              events    One or more EV_* flags
              arg   A user-supplied argument.
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1/ 
{
       $fd = arg0;
       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
              $libevent_events = arg1;

              printf("\n***** elapsed=%d: tid=%d,comm=%s: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()\n", elapsed, tid, comm);
              printf("FileEventImpl*=%p, fd=%d, events=0x%x\n",arg2, $fd, $libevent_events);

              if( $libevent_events & (uint16)0x01 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_TIMEOUT\n");
              }
              if( $libevent_events & (uint16)0x02 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_READ\n");
              }
              if( $libevent_events & (uint16)0x04 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_WRITE\n");
              }
              if( $libevent_events & (uint16)0x20 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_ET\n");
              }
              if( $libevent_events & (uint16)0x80 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_CLOSED\n");
              }
              
              printf("%s\n", kstack);
              @fdFired[tid]=$fd;
              @watchedWakeRound[tid]=tid;
       }
}

/*
记录本轮 epoll 事件触发的 FD 级回调结束
*/
uretprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1 && @fdFired[tid] / 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: END:EventFired\n", elapsed, tid, comm);
       delete(@fdFired[tid]);
}

/*
埋点 socket 读
*/ 
tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_readv,tracepoint:syscalls:sys_enter_recv*
/pid == $1 && (@fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
{
       @watchedIo[tid]=args->fd;
}

tracepoint:syscalls:sys_exit_read,tracepoint:syscalls:sys_exit_readv,tracepoint:syscalls:sys_exit_recv*
/pid == $1 && @watchedIo[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_read, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid],args->ret);
       delete(@watchedIo[tid]);
       printf("%s\n", ustack(20));
}

/*
埋点 socket 写
*/
tracepoint:syscalls:sys_enter_write,tracepoint:syscalls:sys_enter_writev
/pid == $1 && ( @fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
{
       @watchedIo[tid]=args->fd;
}

tracepoint:syscalls:sys_exit_write,tracepoint:syscalls:sys_exit_writev
/pid == $1 && @watchedIo[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_write, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid], args->ret);
       delete(@watchedIo[tid]);
       printf("%s\n", ustack(20));
}


/*
打印 TlsInspector 的所有函数调用
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*TlsInspector*
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: TlsInspector*, probe=%s\n", elapsed, tid, comm, probe);
}

/*
打印连接确认协后的函数调用
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnectionSocketImpl*setRequestedApplicationProtocols*
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: ConnectionSocketImpl::setRequestedApplicationProtocols\n", elapsed, tid, comm);
       // printf("%s", ustack);
}

//SslSocket::SslSocket
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocketC2ENSt3__110shared_ptrINS_3Ssl7ContextEEENS2_12InitialStateERKNS5_IKNS_7Network22TransportSocketOptionsEEENS4_8functionIFNS5_INS6_10HandshakerEEENS4_10unique_ptrI6ssl_stN4bssl8internal7DeleterISK_EEEEiPNS6_18HandshakeCallbacksEEEE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// setTransportSocketCallbacks
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket27setTransportSocketCallbacksERNS_7Network24TransportSocketCallbacksE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}


// SslSocket::doRead
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket6doReadERNS_6Buffer8InstanceE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}


// SslSocket::doWrite
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket7doWriteERNS_6Buffer8InstanceEb
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::onSuccess
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_N5Envoy10Extensions16TransportSockets3Tls9SslSocket9onSuccessEP6ssl_st
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

/*
打印匹配到的 Network Fitler Chain 名字
void setFilterChainName(absl::string_view filter_chain_name)
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10StreamInfo14StreamInfoImpl18setFilterChainNameEN4absl11string_viewE 
/pid == $1 && @fdFired[tid]/ 
{ 
/**
[Assembly 2: Calling convention](https://cs61.seas.harvard.edu/site/2018/Asm2/)

1. A structure argument that fits in a single machine word (64 bits/8 bytes) is passed in a single register.
   
    Example: `struct small { char a1, a2; }`

2. A structure that fits in two to four machine words (16–32 bytes) is passed in sequential registers, as if it were multiple arguments.
   
    Example: `struct medium { long a1, a2; }`
*/
    $filterName = str(reg("si"));
    $filterNameLength = reg("dx");

    // printf("tid:%d: Got setFilterChainName=%s, lenght=%d\n %s \n", tid, $filterName, $filterNameLength, ustack ); 
    printf("comm:%s,tid:%d: Got setFilterChainName=%s, lenght=%d\n", comm, tid, $filterName, $filterNameLength ); 

    if( $filterNameLength > 0 ) {
        printf("%s\n", ustack(3));
    }
}

/*
埋点 FilterManagerImpl::add*Filter()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*add*Filter*,uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*FilterManagerImpl*
/pid == $1 && @fdFired[tid]/ 
{ 
       if( @upFD[@fdFired[tid]] ) {
              return;
       }

       $fd = @fdFired[tid];
       @filterManagerImpl2fd[arg0] = $fd;
       if( @fd2filterManagerImpl[$fd] ) {
              @fd2filterManagerImpl2[$fd] = arg0;
       } else {
              @fd2filterManagerImpl[$fd] = arg0;
       }
       printf("comm:%s,tid:%d: FilterManagerImpl.add*Filter|FilterManagerImpl::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);
       printf("%s\n", ustack(40));
}

/*
埋点 FilterManagerImpl::on*()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*on*
/pid == $1 && @fdFired[tid]/ 
{ 
       if( @upFD[@fdFired[tid]] ) {
              return;
       }

       $fd = @fdFired[tid];
       @filterManagerImpl2fd[arg0] = $fd;
       if( @fd2filterManagerImpl[$fd] ) {
              @fd2filterManagerImpl2[$fd] = arg0;
       } else {
              @fd2filterManagerImpl[$fd] = arg0;
       }
       printf("comm:%s,tid:%d: FilterManagerImpl.on*::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd); 
}

/*
埋点 ConnPoolImplBase::attachStreamToClient()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnPoolImplBase*attachStreamToClient*
/pid == $1 && @fdFired[tid]/ 
{ 
       printf("comm:%s,tid:%d: ConnPoolImplBase*attachStreamToClient: %s \n", comm, tid, probe); 
       printf("%s\n", ustack(40));
}

/*
埋点 socket connect upstream
*/
kprobe:tcp_connect
/pid == $1/
{
       $sk = ((struct sock *)arg0);
       $inet_family = $sk->__sk_common.skc_family;

       if ($inet_family == AF_INET || $inet_family == AF_INET6)
       {
              if ($inet_family == AF_INET)
              {
                     $daddr = ntop($sk->__sk_common.skc_daddr);
                     $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
              }
              else
              {
                     $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
                     $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
              }
              $lport = $sk->__sk_common.skc_num;
              $dport = $sk->__sk_common.skc_dport;

              // Destination port is big endian, it must be flipped
              $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);

              // only watch listen port 8080
              if (8080 != $dport)
              {
                     return;
              }

              printf("connecting to 8080:\n");
              time("%H:%M:%S ");
              printf("%-8d %-16s ", pid, comm);
              printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);
              printf("%s\n", ustack(40));
       }
}


/*
清理关闭的 FD 相关的 Map
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_close/format
*/
tracepoint:syscalls:sys_enter_close
/pid==$1/
{
    $fd = args->fd;
    if( $fd < 0 ) {
        return;
    }

       delete(@upFD[$fd]);

       if( @fd2sockpair[$fd].0 ) {
              printf("sys_enter_close fd=%d\n", $fd);
       }
       else {
              return;
       }
    delete(@fd2sockpair[$fd]);

       /////

       $filterManagerImpl = @fd2filterManagerImpl[$fd];
       if( $filterManagerImpl ) {
              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
              delete( @fd2filterManagerImpl[$fd] );
       }

       if( @filterManagerImpl2fd[$filterManagerImpl] ) {
              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
              delete(@filterManagerImpl2fd[$filterManagerImpl]);
       }


       $filterManagerImpl = @fd2filterManagerImpl2[$fd];
       if( $filterManagerImpl ) {
              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
              delete( @fd2filterManagerImpl2[$fd] );
       }

       if( @filterManagerImpl2fd[$filterManagerImpl] ) {
              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
              delete(@filterManagerImpl2fd[$filterManagerImpl]);
       }

       delete(@upFD[@downFD2upFD[$fd]]);
       delete(@downFD2upFD[$fd]);
}

END
{
       clear(@last_epoll_wait_args);
       clear(@tid2Waketime);
       clear(@tid2epollNrFdReady);
}

结尾

这 part，从 socket 事件驱动的角度，了解了 envoy 作为反向代理的主要流程。我自己觉得是学习到东西的，你呢？
写博客的初憧是记录自己。想不到也能通过博客认识到很多志同道合，对技术认真之人（不能叫同志？）。谢谢你们，我才能坚持写下去，也认为这样的写作有一定价值！

image.png

这是几年前拍的老照片了，共勉。