Kibana无法启动 kibana_task_manager search_phase_execution_exception 问题解决

今天服务器重启以后，发现无法启动Kibana服务了，在网上也是把错误搜索了半天，尝试了那些命令，但是执行删除命令时居然报错了。后来，仔细观察后，发现是删除方式不太正确，需要改一下删除命令才能正确执行，在这里就记录一下解决办法。

环境信息：

ES版本：8.1.2
Kibana版本：8.1.2

问题现象

• 启动时异常信息如下：（主要看那几行 ERROR 类型的）

[2023-10-30T18:05:45.132+08:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
[2023-10-30T18:05:45.565+08:00][INFO ][http.server.Preboot] http server running at http://192.168.24.215:5601
[2023-10-30T18:05:46.012+08:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-10-30T18:05:46.021+08:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
[2023-10-30T18:05:46.076+08:00][INFO ][root] Holding setup until preboot stage is completed.
[2023-10-30T18:05:46.129+08:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-10-30T18:05:46.896+08:00][INFO ][plugins-system.standard] Setting up [112] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,sharedUX,share,embeddable,uiActionsEnhanced,screenshotMode,screenshotting,banners,telemetry,newsfeed,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,lists,fileUpload,ingestPipelines,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,savedObjectsManagement,console,controls,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,maps,dashboardEnhanced,expressionTagcloud,expressionPie,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,dataViewFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,dataViewManagement]
[2023-10-30T18:05:46.934+08:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 1acef6b9-4d10-41ef-a740-f83d68d1a73c
[2023-10-30T18:05:47.282+08:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-10-30T18:05:47.283+08:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-10-30T18:05:47.321+08:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-10-30T18:05:47.322+08:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-10-30T18:05:47.345+08:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-10-30T18:05:47.366+08:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-10-30T18:05:47.387+08:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-10-30T18:05:47.409+08:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-10-30T18:05:47.435+08:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-10-30T18:05:48.379+08:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-10-30T18:05:49.630+08:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-10-30T18:05:49.630+08:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-10-30T18:05:50.673+08:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 986ms.
[2023-10-30T18:05:51.616+08:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 943ms.
[2023-10-30T18:05:52.158+08:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 542ms.
[2023-10-30T18:05:52.163+08:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 2471ms.
[2023-10-30T18:05:52.467+08:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 309ms.
[2023-10-30T18:05:52.844+08:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 377ms.
[2023-10-30T18:05:52.879+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 1 in 2 seconds.
[2023-10-30T18:05:52.880+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 716ms.
[2023-10-30T18:05:53.918+08:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 1074ms.
[2023-10-30T18:05:53.919+08:00][INFO ][savedobjects-service] [.kibana] Migration completed after 4232ms
[2023-10-30T18:05:55.140+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 2 in 4 seconds.
[2023-10-30T18:05:55.141+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 2261ms.
[2023-10-30T18:05:55.384+08:00][INFO ][plugins.screenshotting.chromium] Browser executable: /kibana-8.1.2/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-10-30T18:05:59.177+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 3 in 8 seconds.
[2023-10-30T18:05:59.178+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 4037ms.
[2023-10-30T18:06:07.214+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 4 in 16 seconds.
[2023-10-30T18:06:07.214+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 8036ms.
[2023-10-30T18:06:23.249+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 5 in 32 seconds.
[2023-10-30T18:06:23.249+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 16036ms.
[2023-10-30T18:06:55.288+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 6 in 64 seconds.
[2023-10-30T18:06:55.290+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 32039ms.
[2023-10-30T18:07:59.695+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 7 in 64 seconds.
[2023-10-30T18:07:59.696+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64407ms.
[2023-10-30T18:09:03.910+08:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 8 in 64 seconds.
[2023-10-30T18:09:03.911+08:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64215ms.

排查与解决

1. 查看当前ES aliases 中是否存在 .kibana 开头的数据

curl http://localhost:9200/_cat/aliases | grep kibana

结果示例：

.kibana                    .kibana_8.1.2_001                      - - - -
.kibana_8.1.2              .kibana_8.1.2_001                      - - - -
.kibana_task_manager       .kibana_task_manager_8.1.2_001         - - - -
.kibana_task_manager_8.1.2 .kibana_task_manager_8.1.2_001         - - - -

有的博文写的直接用这个命令来删除 aliases ，其实是会报错的。

curl -X DELETE http://localhost:9200/.kibana*

错误信息一般这个样子：

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [.kibana] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [.kibana] matches an alias, specify the corresponding concrete indices instead."},"status":400}

其实上面已经解释的比较明确了，需要我们删除具体的索引名称，而不能直接删除别名。也就是上述结果的第2列 (其实也就两个)。
所以，我们要把命令改为：

curl -X DELETE http://localhost:9200/.kibana_task_manager_8.1.2_001

curl -X DELETE http://localhost:9200/.kibana_8.1.2_001

执行后，都返回 {"acknowledged":true} 就代表删除成功了，然后我们重新启动 kibana 就可以正常启动了。

OK，本文就到这里了，如有错误，欢迎评论指正。别忘了点赞、收藏哦！Thanks♪(･ω･)ﾉ