kubelet启动之启动oomWatcher模块

概述

基于kubernetes v1.18.6

oomWatcher是一个kubelet子模块,在kubelet启动流程内启动。通过与cAdvisor模块交互采集主机oom进程信息,启动阶段如下图(红色部分)

boot-oomWatcher.png

oomWatcher是做什么的?

oomWatcher记录系统oom并记录到节点的event

工作流程解析

oomWatcher模块在启动的时候会fork出两个goroutine,一个作为oom事件生产者,一个作为oom事件消费者。两个goroutine采用channel进行数据传输。

oomWatcher-transfer.png

源码实现

// Start watches for system oom's and records an event for every system oom encountered.
func (ow *realWatcher) Start(ref *v1.ObjectReference) error {
    outStream := make(chan *oomparser.OomInstance, 10)

    // cAdvisor模块 从/dev/kmsg读取数据解析出oom容器信息:宿主机pid、时间戳、容器名称、容器内进程名称
    go ow.oomStreamer.StreamOoms(outStream)
    
    go func() {
        defer runtime.HandleCrash()

        for event := range outStream {
            if event.ContainerName == recordEventContainerName {
                klog.V(1).Infof("Got sys oom event: %v", event)
                eventMsg := "System OOM encountered"
                if event.ProcessName != "" && event.Pid != 0 {
                    eventMsg = fmt.Sprintf("%s, victim process: %s, pid: %d", eventMsg, event.ProcessName, event.Pid)
                }
                ow.recorder.Eventf(ref, v1.EventTypeWarning, systemOOMEvent, eventMsg)
            }
        }
        klog.Errorf("Unexpectedly stopped receiving OOM notifications")
    }()
    return nil
}

数据源采集

cAdvisor模块从/dev/kmsg读取数据解析出oom容器信息:pidoom时间戳、容器名称、容器内进程名称等。其中字段值如下:

  • 优先级: 6
  • 序列号: 4029
  • 时间戳: 2021308979
  • 信息: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
$ cat /dev/kmsg
...
6,4029,2021308979,-;[  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
...

数据处理

  1. 数据过滤

过滤出oom信息,过滤规则:正则匹配包含invoked oom-killer:字符串的行。

随后利用正则匹配+字符串分割生成对应对象

4,3994,2021308860,-;java invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=999
  1. 数据解析
  • 获取进程pid: 正则匹配+字符串分割
  • 获取容器名称: 正则匹配+字符串分割
// StreamOoms writes to a provided a stream of OomInstance objects representing
// OOM events that are found in the logs.
// It will block and should be called from a goroutine.
func (self *OomParser) StreamOoms(outStream chan<- *OomInstance) {
    // 通道内写入/dev/kmsg内解析的数据:、序列号、时间戳、信息。如:
    // - 优先级: 6
    // - 序列号: 4029
    // - 时间戳: 2021308979
    // - 信息:  [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
    // 6,4029,2021308979,-;[  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
    kmsgEntries := self.parser.Parse()
    defer self.parser.Close()

    for msg := range kmsgEntries {
        // 正则查找oom进程实例,正则规则为:包含`invoked oom-killer:`字符串
        // 4,3994,2021308860,-;java invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=999
        in_oom_kernel_log := checkIfStartOfOomMessages(msg.Message)
        if in_oom_kernel_log {
            oomCurrentInstance := &OomInstance{
                ContainerName:       "/",
                VictimContainerName: "/",
                TimeOfDeath:         msg.Timestamp,
            }
            for msg := range kmsgEntries {
                // 获取容器名称(docker下貌似为空)
                err := getContainerName(msg.Message, oomCurrentInstance)
                if err != nil {
                    klog.Errorf("%v", err)
                }
                // 获取进程id
                // 3,13796,359453210036,-;Memory cgroup out of memory: Killed process 61746 (nginx) total-vm:10668kB, anon-rss:892kB, file-rss:12kB, shmem-rss:0kB, UID:0 pgtables:56kB oom_score_adj:999
                finished, err := getProcessNamePid(msg.Message, oomCurrentInstance)
                if err != nil {
                    klog.Errorf("%v", err)
                }
                if finished {
                    oomCurrentInstance.TimeOfDeath = msg.Timestamp
                    break
                }
            }
            outStream <- oomCurrentInstance
        }
    }
    // Should not happen
    klog.Errorf("exiting analyzeLines. OOM events will not be reported.")
}

数据源消费

cAdvisor模块生产出的oom进程信息,最终被记录至节点的event,可通过以下方式获取相关信息。

$ kubectl describe node 
...
        Events:
      Type     Reason     Age    From            Message
      ----     ------     ----   ----            -------
      Warning  SystemOOM  13m    kubelet, node1  System OOM encountered, victim process: nginx, pid: 60867
      Warning  SystemOOM  13m    kubelet, node1  System OOM encountered, victim process: nginx, pid: 61438
      Warning  SystemOOM  13m    kubelet, node1  System OOM encountered, victim process: nginx, pid: 61746
      Warning  SystemOOM  12m    kubelet, node1  System OOM encountered, victim process: runc:[2:INIT], pid: 71899
      Warning  SystemOOM  12m    kubelet, node1  System OOM encountered, victim process: nginx, pid: 82162
      Warning  SystemOOM  11m    kubelet, node1  System OOM encountered, victim process: nginx, pid: 96195
      Warning  SystemOOM  10m    kubelet, node1  System OOM encountered, victim process: runc:[2:INIT], pid: 12707
      Warning  SystemOOM  7m33s  kubelet, node1  System OOM encountered, victim process: nginx, pid: 87657
      Warning  SystemOOM  2m32s  kubelet, node1  System OOM encountered, victim process: runc:[2:INIT], pid: 98756
...

你可能感兴趣的:(kubelet启动之启动oomWatcher模块)