限制 kubernetes 跑的业务容器所起的进程数

现象描述

发现业务容器，不停地起线程，现象如下:

[root@master ~]# kubectl logs appserv-2336744132-qhd2d -n xiaojiang
... 
WARNING: Bean creation exception on non-lazy FactoryBean type check: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testService': Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
Jan 29, 2018 10:25:40 AM org.springframework.beans.factory.support.DefaultListableBeanFactory getTypeForFactoryBean
WARNING: Bean creation exception on non-lazy FactoryBean type check: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testService': Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
Jan 29, 2018 10:25:40 AM org.springframework.beans.factory.support.DefaultListableBeanFactory getTypeForFactoryBean
WARNING: Bean creation exception on non-lazy FactoryBean type check: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testService': Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
Jan 29, 2018 10:25:40 AM org.springframework.beans.factory.support.DefaultListableBeanFactory getTypeForFactoryBean
WARNING: Bean creation exception on non-lazy FactoryBean type check: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testService': Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread

这种情况下，会把所在的节点资源耗光，从而触发各种问题的产生

处理方法

限制docker 容器所起的进程数，执行以下命令就可以解决此类问题

#!/bin/bash

MAX_PIDS=1000
CGROUP_PIDS=/sys/fs/cgroup/pids
SYSTEM_SLICE=/sys/fs/cgroup/pids/system.slice/

DOCKER=/sys/fs/cgroup/pids/docker/*
KUBEPODS=/sys/fs/cgroup/pids/kubepods

echo "set @ `date`">/tmp/pids_setting.latest

if [ ! -d $CGROUP_PIDS ];then
    mkdir -p $CGROUP_PIDS
    mount -t cgroup -o pids none $CGROUP_PIDS
fi

for i in `find $DOCKER -type d`;
do
    echo $MAX_PIDS > $i/pids.max
done

for i in `find $KUBEPODS -name "pod*" -type d`;
do
    echo $MAX_PIDS > $i/pids.max
done

更多精彩分享博客:
http://www.lijiaocn.com/tags/kubernetes.html