kafka管理工具之kafka-ui的环境搭建笔记

    由于项目需要kafka支持认证功能,就把kafka升级到3.2.0了。之前一直使用的kafka tools(现在叫Offset Explorer,个人使用免费,商用付费),开了认证之后就不好用了,卡的很,一点也不丝滑了,于是只好重新发掘新的工具,然后就发现了这款名为kafka-ui(github地址)的管理工具。它是一款web的工具,开源,并且官网提供了docker镜像,安装部署很方便。

以下动图是其官网展示的UI图片,功能非常强大:

    如果对该项目没有修改的需求,那么可以直接部署,官网提供了docker或者直接通过jar运行的部署模式。详情请见官网,这里不做赘述。下面主要介绍在本地build源码的过程,在搭建环境的时候,折腾了2周的时间,遇到了不少的坑,当然这其中大多问题都是与本地环境以及网络环境有关。现将问题整理于此,以作备忘。

  1. 安装jdk17+,该项目仅支持jdk17及以上的版本,所以这里需要注意设置JAVA_HOME的环境变量等设置。build时可能遇到如下错误:

[INFO] — maven-compiler-plugin:3.10.1:compile (default-compile) @ kafka-ui-contract —
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 362 source files to C:\workspace\kafka-ui-new\kafka-ui-contract\target\classes
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] kafka-ui 0.0.1-SNAPSHOT … SUCCESS [ 0.042 s]
[INFO] kafka-ui-contract 0.0.1-SNAPSHOT … FAILURE [01:23 min]
[INFO] kafka-ui-serde-api 1.0.0 … SKIPPED
[INFO] kafka-ui-api 0.0.1-SNAPSHOT … SKIPPED
[INFO] kafka-ui-e2e-checks 0.0.1-SNAPSHOT … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:24 min
[INFO] Finished at: 2023-09-01T17:11:22+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project kafka-ui-contract: Fatal error compiling: invalid flag: --release -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :kafka-ui-contract
PS C:\workspace\kafka-ui-new>

重点关注这一句:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project kafka-ui-contract: Fatal error compiling: invalid flag: --release -> [Help 1]

输出以上信息就是由于我当前环境变量的JAVA版本是1.8,kafka-ui需要java 17+。所以,需要修改JAVA_HOME的环境变量,由于我本机没有系统管理员权限,不能直接去修改环境变量,因此我这里采用了通过命令设置的方式,打开Windows PowerShell,输入以下命令:
PS C:\workspace\kafka-ui-new> Set-Item -Path Env:JAVA_HOME -Value "C:\jdks\openjdk-20.0.2_windows-x64_bin\jdk-20.0.2\"

  1. 安装nvw,下载nvw,免安装版本,然后设置环境变量,如果不设置环境变量,后面build的时候,会提示找不到命令。nvw中已经包含了nodejs,不需要再单独安装。以下是环境变量的参考:
    nvm环境变量
    记住,除了上面的设置,还需要将NVM_HOME以及NVM_SYMLINK加入到Path路劲下,否则不会生效。Path中包含如下两项:
    Path环境变量
    另外,如果电脑中之前安装过nodejs,为了避免不必要的错误,建议先卸载已安装的nodejs,然后通过nvw来安装nodejs的各版本。

  2. 升级pom.xml中的软件版本,由于前面安装的nvm中的node=v18.17.1,pnpm=8.7.1,所以我们需要修改kafka-ui/pom.xml文件中的这两个地方:


<node.version>v18.17.1node.version>
<pnpm.version>v8.7.1pnpm.version>
  1. 设置maven的代理,如果当前网络需要代理才能访问外网,那么需要对maven的代理进行设置。找到当前maven所使用的settings文件,然后增加代理:
    <proxy>
      <id>testid>
      <active>trueactive>
      <protocol>httpprotocol>
      <username>username>
      <password>password>
      <host>10.10.10.10host>
      <port>8000port>
      <nonProxyHosts>localhost|127.0.0.1nonProxyHosts>
    proxy>
  1. 如果build时,出现了nodejs内存溢出的问题,错误如下:

<— Last few GCs —>
<— JS stacktrace —>
#
# Fatal javascript OOM in GC during deserialization
#

解决办法:
查看是否设置了NODE_OPTIONS环境变量

PS C:\workspace\kafka-ui-new> echo $env:NODE_OPTIONS

如果输出为空,则说明没有设置环境变量,则按照如下方式进行设置:

PS C:\workspace\kafka-ui-new> Set-Item -Path Env:NODE_OPTIONS -Value "max-old-space-size=4096"    

再次查看NODE_OPTIONS的环境变量是否设置成功:

PS C:\workspace\kafka-ui-new> echo $env:NODE_OPTIONS                   
PS C:\workspace\kafka-ui-new> --max-old-space-size=4096
  1. 如果maven中设置代理,在build项目的前端相关的源码时,可能还会出现如下错误:

[INFO] > [email protected] gen:sources C:\workspace\kafka-ui-new\kafka-ui-react-app
[INFO] > rimraf src/generated-sources && openapi-generator-cli generate “–https-proxy=http://10.10.10.10:8000” “–proxy=http://10.10.10.10:8000” “–noproxy=localhost,127.0.0.1”
[INFO]
[INFO] Download 5.3.0 …
[INFO] Downloaded 5.3.0
[INFO] [error] Found unexpected parameters: [–https-proxy=http://10.10.10.10:8000, --proxy=http://10.10.10.10:8000, --noproxy=localhost,127.0.0.1]
[INFO]
[INFO] See ‘openapi-generator-cli help’ for usage.
[INFO] 鈥塃LIFECYCLE鈥? Command failed with exit code 1.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] kafka-ui 0.0.1-SNAPSHOT … SUCCESS [ 0.029 s]
[INFO] kafka-ui-contract 0.0.1-SNAPSHOT … FAILURE [05:02 min]
[INFO] kafka-ui-serde-api 1.0.0 … SKIPPED
[INFO] kafka-ui-api 0.0.1-SNAPSHOT … SKIPPED
[INFO] kafka-ui-e2e-checks 0.0.1-SNAPSHOT … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:03 min
[INFO] Finished at: 2023-09-01T16:36:46+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:pnpm (pnpm gen:sources) on project kafka-ui-contract: Failed to run task: ‘pnpm gen:sources --https-proxy=http://10.10.10.10:8000 --proxy=http://10.10.10.10:8000 --noproxy=localhost,127.0.0.1’ failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

输出的日志,仅需关注这行就可以了:

[INFO] [error] Found unexpected parameters: [–https-proxy=http://10.10.10.10:8000, --proxy=http://10.10.10.10:8000, --noproxy=localhost,127.0.0.1]

上面的日志说的很清楚,代理相关的那几个参数是有问题的。所以,执行rimraf src/generated-sources && openapi-generator-cli generate时,后面不能跟代理。

解决办法如下,进入kafka-ui-contract目录,修改pom.xml文件:

<execution>
	<id>pnpm gen:sourcesid>
	<goals>
		<goal>pnpmgoal>
	goals>
	<configuration>
		<pnpmInheritsProxyConfigFromMaven>falsepnpmInheritsProxyConfigFromMaven> // 增加这一行,表示pnpm不继承maven配置文件中的代理
		<arguments>gen:sourcesarguments>
	configuration>
execution>
  1. 还是上面提到的问题,虽然设置了执行pnpm时,不继承maven中的代理,但是还是无法下载generator-cli插件,从日志可中看到,在安装=插件时,一直卡在downloading环节,这个在前面的文章中也有记录,这里就不再继续探讨了。解决办法就是加上私有仓库。在kafka-ui-react-app/openapitools.json中需要修改的配置如下:
{
  "$schema": "node_modules/@openapitools/openapi-generator-cli/config.schema.json",
  "spaces": 2,
  "generator-cli": {
    "version": "5.3.0",
    "generators": {
		......
    },
    "repository": {
      "downloadUrl": "http://22.22.22.22:8081/nexus/content/groups/public/${groupId}/${artifactId}/${versionName}/${artifactId}-${versionName}.jar"
    }
  }
}
  1. nodejs版本问题,如果遇到如下错误:

[INFO] — frontend-maven-plugin:1.12.1:install-node-and-pnpm (install node and pnpm) @ kafka-ui-contract —
[INFO] Node v17.9.1 is already installed.
[INFO] PNPM 8.6.12 was installed, but we need version v8.6.12
[INFO] Installing pnpm version v8.6.12
[WARNING] Failed to delete existing PNPM installation.
[INFO] Unpacking C:\Users\.m2\repository\com\github\eirslett\pnpm\8.6.12\pnpm-8.6.12.tar.gz into C:\workspace\kafka-ui-new\kafka-ui-react-app\node\node_modules
[INFO] Installed pnpm locally.
[INFO]
[INFO] — frontend-maven-plugin:1.12.1:pnpm (pnpm install) @ kafka-ui-contract —
[INFO] Running ‘pnpm install’ in C:\workspace\kafka-ui-new\kafka-ui-react-app
[INFO] node:internal/modules/cjs/loader:936
[INFO] throw err;
[INFO] ^
[INFO]
[INFO] Error: Cannot find module ‘C:\workspace\kafka-ui-new\kafka-ui-react-app\node\node_modules\pnpm\bin\pnpm.js’
[INFO] at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
[INFO] at Function.Module._load (node:internal/modules/cjs/loader:778:27)
[INFO] at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
[INFO] at node:internal/main/run_main_module:17:47 {
[INFO] code: ‘MODULE_NOT_FOUND’,
[INFO] requireStack: []
[INFO] }
[INFO]
[INFO] Node.js v17.9.1
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] kafka-ui 0.0.1-SNAPSHOT … SUCCESS [ 0.038 s]
[INFO] kafka-ui-contract 0.0.1-SNAPSHOT … FAILURE [01:12 min]
[INFO] kafka-ui-serde-api 1.0.0 … SKIPPED
[INFO] kafka-ui-api 0.0.1-SNAPSHOT … SKIPPED
[INFO] kafka-ui-e2e-checks 0.0.1-SNAPSHOT … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:15 min
[INFO] Finished at: 2023-09-01T16:41:10+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:pnpm (pnpm install) on project kafka-ui-contract: Failed to run task: ‘pnpm install’ failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :kafka-ui-contract
PS C:\workspace\kafka-ui-new>

上面的日志,说是找不到pnpm的文件,可能是由于我本地安装的pnpm的版本不一致导致的,所以先查看pnpm版本的信息:

PS C:\workspace\kafka-ui-new> pnpm -v
ERROR: This version of pnpm requires at least Node.js v16.14
The current version of Node.js is v11.15.0

卸载pnpm:

PS C:\workspace\kafka-ui-new> npm uninstall -g pnpm
removed 1 package in 0.788s

查看是否卸载成功:

PS C:\workspace\kafka-ui-new> pnpm -v
pnpm : The term 'pnpm' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ pnpm -v
+ ~~~~
    + CategoryInfo          : ObjectNotFound: (pnpm:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
  1. 提示nodejs版本不是32位的。

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:pnpm (pnpm install) on project kafka-ui-contract: Failed to run task: ‘pnpm install --https-proxy=http://1.1.1.1:8000 --proxy=http://1.1.1.1:8000 --noproxy=localhost,127.0.0.1’ failed. java.io.IOException: Cannot run program “C:\workspace\kafka-ui-2\kafka-ui-react-app\node\node.exe” (in directory “C:\workspace\kafka-ui-2\kafka-ui-react-app”): CreateProcess error=193, %1 is not a valid Win32 application -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :kafka-ui-contract

这个问题的解决办法就是去官网下一个32位的版本的免安装包,然后把里面的node.exe放到上面日志打印的路径中去替换原有的exe,比如上面提到node.exe在C:\workspace\kafka-ui-2\kafka-ui-react-app\node目录下。

至此,整个项目应该能build成功了,但在run前端时,可能出现如下的问题:

  1. 设置后台api,运行前端之前,需要在env.local中配置:
VITE_DEV_PROXY= http://127.0.0.1:8080 # your API server

不要使用localhost,如果使用localhost,运行时会报: [vite] http proxy error at xxx 的错误。

  1. 提示找不到Windows PowerShell,详细错误信息如下:
PS C:\workspace\kafka-ui-2\kafka-ui-react-app> pnpm dev

> [email protected] dev C:\workspace\kafka-ui-2\kafka-ui-react-app
> vite

(!) the `splitVendorChunk` plugin doesn't have any effect when using the object form of `build.rollupOptions.output.manualChunks`. Consider using the function form instead.

  VITE v4.4.9  ready in 527 ms

  ➜  Local:   http://localhost:3000/
  ➜  Network: use --host to expose
  ➜  press h to show help
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: spawn undefined\System32\WindowsPowerShell\v1.0\powershell ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:476:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:289:12)
    at onErrorNT (node:internal/child_process:476:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn undefined\\System32\\WindowsPowerShell\\v1.0\\powershell',
  path: 'undefined\\System32\\WindowsPowerShell\\v1.0\\powershell',
  spawnargs: [
    '-NoProfile',
    '-NonInteractive',
    '–ExecutionPolicy',
    'Bypass',
    '-EncodedCommand',
    'UwB0AGEAcgB0ACAAIgBoAHQAdABwADoALwAvAGwAbwBjAGEAbABoAG8AcwB0ADoAMwAwADAAMAAvACIA'
  ]
}

Node.js v18.17.1
ELIFECYCLE  Command failed with exit code 1.

网上搜索的答案是这里会自动打开浏览器,至于如何解决,没有找到合适的答案,但可以设置配置文件,使其不要自动打开浏览器,算是变相解决问题。
在/kafka-ui-react-app下的.env或者env.local文件中,加入如下变量:

BROWSER=none

如果还是报一样的错误,则升级vite至新版本,修改/kafka-ui-react-app/package.json文件:

"vite": "^4.4.5",

    目前遇到的就是这些问题了,如果你有遇到相同的问题,或者有更好的解决办法,欢迎补充,感谢!

你可能感兴趣的:(kafka,kafka-ui,kafka)