【问题】部署filebeat-7.X,然后卸载重新部署6.X,提示初始化registry失败
[root@k8s-32 helm-charts-6.8.9]# helm install fb -n logging ./filebeat/ NAME: fb LAST DEPLOYED: Tue Jun 2 23:02:18 2020 NAMESPACE: logging STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. Watch all containers come up. $ kubectl get pods --namespace=logging -l app=fb-filebeat -w [root@k8s-32 helm-charts-6.8.9]# kubectl get pod -n logging NAME READY STATUS RESTARTS AGE fb-filebeat-tr75f 0/1 CrashLoopBackOff 1 8s [root@k8s-32 helm-charts-6.8.9]# kubectl -n logging logs -f fb-filebeat-tr75f 2020-06-03T03:02:42.826Z INFO instance/beat.go:611 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] 2020-06-03T03:02:42.826Z INFO instance/beat.go:618 Beat UUID: 893233e6-25c6-4673-9a51-62645bb2c7da 2020-06-03T03:02:42.826Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed 2020-06-03T03:02:42.826Z INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/ filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "893233e6-25c6-4673-9a51-62645bb2c7da"}}} 2020-06-03T03:02:42.826Z INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "1d55b4bd9dbf106a4ad4bc34fe9ee425d922363b", "libbeat": "6.7.1", "time": "2019-04-02T15: 01:15.000Z", "version": "6.7.1"}}} 2020-06-03T03:02:42.826Z INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}} 2020-06-03T03:02:42.827Z INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-28T07:53:19Z","containerized":true,"name":"fb-filebea t-tr75f","ip":["127.0.0.1/8","10.11.255.129/32"],"kernel_version":"3.10.0-1062.18.1.el7.x86_64","mac":["5a:05:16:d2:91:59"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)"," major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}} 2020-06-03T03:02:42.828Z INFO [beat] instance/beat.go:976 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setui d","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_c hroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"boundin g":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/u sr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-06-03T03:02:42.730Z"}}} 2020-06-03T03:02:42.828Z INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.7.1 2020-06-03T03:02:42.828Z INFO elasticsearch/client.go:164 Elasticsearch url: http://10.88.77.32:9200 2020-06-03T03:02:42.828Z INFO [publisher] pipeline/module.go:110 Beat name: fb-filebeat-tr75f 2020-06-03T03:02:42.829Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s 2020-06-03T03:02:42.829Z INFO instance/beat.go:402 filebeat start running. 2020-06-03T03:02:42.829Z WARN [cfgwarn] api/server.go:34 EXPERIMENTAL: Metrics endpoint is enabled. 2020-06-03T03:02:42.829Z INFO api/server.go:38 Starting stats endpoint 2020-06-03T03:02:42.829Z ERROR beater/filebeat.go:311 Could not init registrar: Registry file path must be a file. /usr/share/filebeat/data/registry is a directory. 2020-06-03T03:02:42.829Z INFO api/server.go:49 Metrics endpoint listening on: localhost:5066 2020-06-03T03:02:42.832Z INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":19}},"total":{"ticks":30,"time":{"ms":4 2},"value":30},"user":{"ticks":20,"time":{"ms":23}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"481342d6-cf94-4302-a580-fb5118cb95da","uptime":{"ms":13}},"memstats":{"gc_ next":4194304,"memory_alloc":2977176,"memory_total":4508840,"rss":14995456}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"p ipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"load":{"1":0.5,"15":0.4,"5":0.42,"norm":{"1":0.125,"15":0.1,"5":0.105}}}}}} 2020-06-03T03:02:42.832Z INFO [monitoring] log/log.go:153 Uptime: 15.581345ms 2020-06-03T03:02:42.832Z INFO [monitoring] log/log.go:130 Stopping metrics logging. 2020-06-03T03:02:42.832Z INFO instance/beat.go:412 filebeat stopped. 2020-06-03T03:02:42.832Z ERROR instance/beat.go:906 Exiting: Registry file path must be a file. /usr/share/filebeat/data/registry is a directory. Exiting: Registry file path must be a file. /usr/share/filebeat/data/registry is a directory.
【分析】Filebeat会将自己处理日志文件的进度信息写入到registry文件中,以保证filebeat在重启之后能够接着处理未处理过的数据,而无需从头开始。
但是在filebeat-7.X之后,registry是以文件夹方式存在,里面存储了两份文件 { data.json meta.json }。
所以你先部署filebeat-7.X,将会创建registry文件夹,然后你再部署filebeat-6.X时将无法创建registry文件。
【解决】切换到对应节点,删除filebeat文件夹即可(rm -rf /var/lib/fb-filebeat-logging-data)。
注意:/var/lib 是默认日志采集目录,如果你更改了,请切换至对应目录删除。