Skywalking TraceSegmentReportServiceHandler NullPointerException 错误

公司的skywalking 进去里面什么都没有，拓扑 、追踪都没有了。
docker logs 查看一下，哇 一堆 半天还没完。
然后去 /var/lib/docker 里面找到对应的container，tail -f 一下对应的log文件
发现一直在报NPE（NullPointerException ）错误。如下：

{"log":"2022-09-27 08:14:56,711 org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler 75 [grpcServerPool-1-thread-15] ERROR [] - [9.0.0-89644fe (20220217220813)] null\n","stream":"stdout","time":"2022-09-27T08:14:56.711214818Z"}

往上翻有堆栈信息

{"log":"2022-09-27 08:12:11,340 org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler 75 [grpcServerPool-1-thread-14] ERROR [] - [9.0.0-89644fe (20220217220813)] null\n","stream":"stdout","time":"2022-09-27T08:12:11.340318793Z"}
{"log":"java.lang.NullPointerException: null\n","stream":"stdout","time":"2022-09-27T08:12:11.340336261Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.core.source.EndpointRelation.prepare(EndpointRelation.java:101) ~[server-core-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340340368Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.core.analysis.DispatcherManager.forward(DispatcherManager.java:59) ~[server-core-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340343716Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.core.source.SourceReceiverImpl.receive(SourceReceiverImpl.java:36) ~[server-core-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340347085Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.EndpointDepFromCrossThreadAnalysisListener.lambda$build$0(EndpointDepFromCrossThreadAnalysisListener.java:130) ~[agent-analyzer-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340360438Z"}
{"log":"\u0009at java.util.ArrayList.forEach(Unknown Source) ~[?:?]\n","stream":"stdout","time":"2022-09-27T08:12:11.340365604Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.EndpointDepFromCrossThreadAnalysisListener.build(EndpointDepFromCrossThreadAnalysisListener.java:124) ~[agent-analyzer-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340369577Z"}
{"log":"\u0009at java.util.ArrayList.forEach(Unknown Source) ~[?:?]\n","stream":"stdout","time":"2022-09-27T08:12:11.340373956Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.TraceAnalyzer.notifyListenerToBuild(TraceAnalyzer.java:75) ~[agent-analyzer-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340381544Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.TraceAnalyzer.doAnalysis(TraceAnalyzer.java:71) ~[agent-analyzer-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.34038456Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.SegmentParserServiceImpl.send(SegmentParserServiceImpl.java:40) ~[agent-analyzer-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340387556Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:72) [skywalking-trace-receiver-plugin-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340390469Z"}
{"log":"\u0009at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:63) [skywalking-trace-receiver-plugin-9.0.0.jar:9.0.0]\n","stream":"stdout","time":"2022-09-27T08:12:11.340395078Z"}
{"log":"\u0009at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262) [grpc-stub-1.43.2.jar:1.43.2]\n","stream":"stdout","time":"2022-09-27T08:12:11.340399023Z"}
{"log":"\u0009at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:318) [grpc-core-1.43.2.jar:1.43.2]\n","stream":"stdout","time":"2022-09-27T08:12:11.340402095Z"}
{"log":"\u0009at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:301) [grpc-core-1.43.2.jar:1.43.2]\n","stream":"stdout","time":"2022-09-27T08:12:11.340404899Z"}
{"log":"\u0009at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834) [grpc-core-1.43.2.jar:1.43.2]\n","stream":"stdout","time":"2022-09-27T08:12:11.340407679Z"}
{"log":"\u0009at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.43.2.jar:1.43.2]\n","stream":"stdout","time":"2022-09-27T08:12:11.340410546Z"}
{"log":"\u0009at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.43.2.jar:1.43.2]\n","stream":"stdout","time":"2022-09-27T08:12:11.340413445Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]\n","stream":"stdout","time":"2022-09-27T08:12:11.340416303Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]\n","stream":"stdout","time":"2022-09-27T08:12:11.34041897Z"}
{"log":"\u0009at java.lang.Thread.run(Unknown Source) [?:?]\n","stream":"stdout","time":"2022-09-27T08:12:11.340421694Z"}

日志被格式化了，很难看，但能找到类似java.util.ArrayList.forEach 的信息
百度无果
github 找一找issue
搜索TraceSegmentReportServiceHandler
找到一条相关信息
https://github.com/apache/skywalking/issues/9297

表示这是一个bug，看不懂里面分享的源码（在APM领域还是新新手，什么都不懂，话说skywalking的开发者号称华裔，也没个官方的中文文档，英文文档感觉介绍的要么很深，要不很浅，没有一个APM入门的新手需要的，issue里面还不准别人提交中文issue），但问题应该是出在9.0.0 and 9.1.0版本。所以两个解决办法，要么升级 要么降级
当然升级了！
查询docker hub，升级到9.2.0，一番操作猛如虎，然后··报了一堆错误，大概如下：

There is insufficient memory for the Java Runtime Environment to continue.
---------------  S U M M A R Y ------------

Command Line: -Xms2G org.apache.skywalking.oap.server.starter.OAPServerStartUp

Host: Intel(R) Xeon(R) Gold 6278C CPU @ 2.60GHz, 4 cores, 7G, Ubuntu 22.04.1 LTS
Time: Tue Sep 27 03:32:44 2022 UTC elapsed time: 0.002954 seconds (0d 0h 0m 0s)

内存小了？ 不会吧··我分了2048m啊，继续搜索
https://github.com/apache/skywalking/issues/9349
虽然被作者关闭了issue，但下面有个好心人给出了链接
https://stackoverflow.com/questions/72841549/container-fails-to-start-insufficient-memory-for-the-java-runtime-environment-t
是docker的锅
docker info一下，

Server Version: 20.10.9

果然，好 ，升级docker 先，可是··

yum clean all && yum makecache
# 出现了下面的错误
Errors during downloading metadata for repository 'epel':
  - Status code: 404 for http://archives.fedoraproject.org/pub/archive/epel/8/Everything/x86_64/repodata/repomd.xml (IP: 38.145.60.23)
Error: Failed to download metadata for repo 'epel': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
# 8.2 不支持了

好，那就换源

mv /etc/yum.repos.d /etc/yum.repos.d.bak
mkdir -p /etc/yum.repos.d
curl https://mirrors.aliyun.com/repo/Centos-vault-8.5.2111.repo > /etc/yum.repos.d/Centos-vault-8.5.2111.repo
curl https://mirrors.aliyun.com/repo/epel-archive-8.repo > /etc/yum.repos.d/epel-archive-8.repo
yum clean all && yum makecache

然后列出可安装的版本

# 列出可安装版本
yum list docker-ce --showduplicates | sort -r
Last metadata expiration check: 0:27:27 ago on Wed 28 Sep 2022 08:09:36 AM CST.
Installed Packages
docker-ce.x86_64               3:20.10.9-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.9-3.el8                 @docker-ce-stable
docker-ce.x86_64               3:20.10.8-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.7-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.6-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.5-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.4-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.3-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.2-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.18-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.17-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.16-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.15-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.14-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.1-3.el8                 docker-ce-stable
docker-ce.x86_64               3:20.10.13-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.12-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.11-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.10-3.el8                docker-ce-stable
docker-ce.x86_64               3:20.10.0-3.el8                 docker-ce-stable
docker-ce.x86_64               3:19.03.15-3.el8                docker-ce-stable
docker-ce.x86_64               3:19.03.14-3.el8                docker-ce-stable
docker-ce.x86_64               3:19.03.13-3.el8                docker-ce-stable

最高就是20.10.9 ？ 不会吧
换个机器，人家都18了啊
查询 Linux 版本

lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 8.2.2004 (Core)
Release:        8.2.2004
Codename:       Core

咋整，这个时候，我只能选择用原来的docker-compose，奇迹的是，竟然好了，拓扑、跟踪都有了。
但后台还是会报NPE错误，有没有大神指点一下？

2022年9月28日更新：
看错了，yum list 是有18 的，只不过在下面，我就说嘛，我在docker官网都看到有rpm包了。
https://download.docker.com/linux/centos/8/x86_64/stable/Packages/
好，下面就是重装docker。