《OpenShift / RHEL / DevSecOps 汇总目录》
说明:本文已经在OpenShift 4.8 环境中验证
$ CONTROL_NODE=$(oc get nodes -l node-role.kubernetes.io/master= -o jsonpath='{.items[0].metadata.name}')
$ oc adm node-logs $CONTROL_NODE --path=/
audit/
btmp
chrony/
containers/
crio/
etcd/
glusterfs/
journal/
kube-apiserver/
lastlog
oauth-apiserver/
openshift-apiserver/
openvswitch/
pods/
private/
qemu-ga/
samba/
sssd/
wtmp
$ oc adm node-logs --role=master --path=openshift-apiserver/
ip-10-0-135-39.us-east-2.compute.internal audit.log
ip-10-0-173-152.us-east-2.compute.internal audit.log
ip-10-0-173-213.us-east-2.compute.internal audit.log
$ oc adm node-logs --role=master --path=kube-apiserver/
$ oc adm node-logs --role=master --path=oauth-apiserver/
$ oc adm node-logs $CONTROL_NODE --path=journal | wc -l
$ oc adm node-logs $CONTROL_NODE -u kubelet | wc -l
可以通过以下命令将集群节点的重要日志(etcd、kube-apiserver、openshift-apiserver、oauth-apiserver)下载到本地。
$ oc adm must-gather -- /usr/bin/gather_audit_logs
$ ll must-gather.local.xxxxxxx/quay-io-openshift-release-dev-ocp-xxxxxxx/audit_logs/
total 24
drwxr-xr-x. 2 lab-user users 230 Mar 5 01:06 etcd
-rw-r--r--. 1 lab-user users 194 Mar 5 01:06 etcd.audit_logs_listing
drwxr-xr-x. 2 lab-user users 4096 Mar 5 01:07 kube-apiserver
-rw-r--r--. 1 lab-user users 3025 Mar 5 01:06 kube-apiserver.audit_logs_listing
drwxr-xr-x. 4 lab-user users 92 Mar 5 01:06 monitoring
drwxr-xr-x. 2 lab-user users 194 Mar 5 01:07 oauth-apiserver
-rw-r--r--. 1 lab-user users 158 Mar 5 01:06 oauth-apiserver.audit_logs_listing
drwxr-xr-x. 2 lab-user users 194 Mar 5 01:07 openshift-apiserver
-rw-r--r--. 1 lab-user users 158 Mar 5 01:06 openshift-apiserver.audit_logs_listing
审计日志会记录访问 OpenShift API 服务、Kubernetes API 服务和 OAuth API 服务的请求。其中 OpenShift API 审计日志按时间排序记录用户、管理员或其他系统组件访问OpenShift API 服务的活动。审计日志的内容分散在不同 Node 上的 /var/log/openshift-apiserver/audit.log 文件中,内容为 JSON 格式。
以下是 OpenShift API 审计日志的主要内容:
其中根据对用户认证结果,以上 user 可以所属以下三类组:
Virtual 组 | 描述 |
---|---|
system:authenticated | 关联到所有认证过的用户 |
system:authenticated:oauth | 关联到所有认证过并带有OAuth访问令牌的用户 |
system:unauthenticated | 关联到所有未经认证的用户 |
OpenShift 提供以下4中预定义的审计策略配置,不同的策略记录的审计日志内容范围有差别。
审计策略名称 | 描述 |
---|---|
Default | 仅记录读取和写入请求的日志元数据 ;除了 OAuth 访问令牌请求外,不记录请求正文。这是默认策略。 |
WriteRequestBodies | 仅除了记录所有请求的元数据外,同时记录对 API 服务器的写入请求(create、update、patch)的具体数据(body)。这个配置集的资源开销比 Default 配置集大。 |
AllRequestBodies | 除了记录所有请求的元数据外,对 API 服务器的每个读写请求(get、list、create、update、patch)都进行日志记录。这个配置集的资源开销最大。 |
None | 关闭审计日志。 |
$ oc adm node-logs $CONTROL_NODE --path=openshift-apiserver/audit.log | tail -1 | jq '{requestReceivedTimestamp,verb,username: .user.username,usergroups: .user.groups,requestObject}'
{
"requestReceivedTimestamp": "2022-03-06T02:27:37.343244Z",
"kind": "Event",
"verb": "watch",
"username": "system:serviceaccount:openshift-ingress:router",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-ingress",
"system:authenticated"
],
"requestObject": null
}
缺省情况,API 审计日志不记录requestObject或 responseObject中的内容。
$ oc get apiserver cluster -oyaml
apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
。。。
spec:
audit:
profile: Default
$ oc patch apiserver/cluster -p '{"spec":{"audit":{"profile":"WriteRequestBodies"}}}' --type merge
$ oc get clusteroperator openshift-apiserver kube-apiserver authentication
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.10.1 True False False 100m
kube-apiserver 4.10.1 True False False 7h31m
openshift-apiserver 4.10.1 True False False 5h19m
$ oc adm node-logs --role=master --path=openshift-apiserver/audit.log | awk 'sub($1,"")' | jq 'select(.objectRef.name == "my-project1") | {requestReceivedTimestamp,verb,username: .user.username,usergroups: .user.groups,requestObject}'
$ oc new-project my-project1
$ oc adm node-logs --role=master --path=openshift-apiserver/audit.log | awk 'sub($1,"")' | jq 'select(.objectRef.name == "my-project1") | {requestReceivedTimestamp,verb,username: .user.username,usergroups: .user.groups,requestObject}'
{
"requestReceivedTimestamp": "2022-03-06T03:21:59.175551Z",
"verb": "get",
"username": "system:serviceaccount:openshift-apiserver:openshift-apiserver-sa",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-apiserver",
"system:authenticated"
],
"requestObject": null
}
{
"requestReceivedTimestamp": "2022-03-06T03:21:59.192883Z",
"verb": "create",
"username": "system:serviceaccount:openshift-apiserver:openshift-apiserver-sa",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-apiserver",
"system:authenticated"
],
"requestObject": {
"kind": "Project",
"apiVersion": "project.openshift.io/v1",
"metadata": {
"name": "my-project1",
"creationTimestamp": null,
"annotations": {
"openshift.io/description": "",
"openshift.io/display-name": "",
"openshift.io/requester": "opentlc-mgr"
}
},
"spec": {},
"status": {}
}
}
{
"requestReceivedTimestamp": "2022-03-06T03:21:59.356254Z",
"verb": "get",
"username": "system:serviceaccount:openshift-apiserver:openshift-apiserver-sa",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-apiserver",
"system:authenticated"
],
"requestObject": null
}
{
"requestReceivedTimestamp": "2022-03-06T03:21:59.167383Z",
"verb": "create",
"username": "opentlc-mgr",
"usergroups": [
"system:masters",
"system:authenticated"
],
"requestObject": {
"kind": "ProjectRequest",
"apiVersion": "project.openshift.io/v1",
"metadata": {
"name": "my-project1",
"creationTimestamp": null
}
}
}
下面操作将修改API审计日志配置:
. 缺省使用 Default 日志配置,即不记录所有操作的 Request 部分。
. 当用户所属组为 “system:authenticated:oauth”,则使用 Default 日志配置,即不记录所有操作的 Request 部分。
. 当用户所属组为 “system:authenticated”,则使用 WriteRequestBodies 日志配置,即只记录Write操作的 Request 部分。
$ oc edit apiserver cluster
apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
...
spec:
audit:
customRules:
- group: system:authenticated:oauth
profile: Default
- group: system:authenticated
profile: WriteRequestBodies
profile: Default
$ oc get clusteroperator openshift-apiserver kube-apiserver authentication
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
openshift-apiserver 4.10.1 True False False 5h19m
kube-apiserver 4.10.1 True False False 7h32m
authentication 4.10.1 True False False 101m
$ oc new-project my-project2
$ oc adm node-logs --role=master --path=openshift-apiserver/audit.log | awk 'sub($1,"")' | jq 'select(.objectRef.name == "my-project2") | {requestReceivedTimestamp,verb,username: .user.username,usergroups: .user.groups,requestObject}'
{
"requestReceivedTimestamp": "2022-03-06T08:00:23.084097Z",
"verb": "get",
"username": "system:serviceaccount:openshift-apiserver:openshift-apiserver-sa",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-apiserver",
"system:authenticated"
],
"requestObject": null
}
{
"requestReceivedTimestamp": "2022-03-06T08:00:23.133258Z",
"verb": "create",
"username": "system:serviceaccount:openshift-apiserver:openshift-apiserver-sa",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-apiserver",
"system:authenticated"
],
"requestObject": {
"kind": "Project",
"apiVersion": "project.openshift.io/v1",
"metadata": {
"name": "my-project2",
"creationTimestamp": null,
"annotations": {
"openshift.io/description": "",
"openshift.io/display-name": "",
"openshift.io/requester": "opentlc-mgr"
}
},
"spec": {},
"status": {}
}
}
{
"requestReceivedTimestamp": "2022-03-06T08:00:23.378818Z",
"verb": "get",
"username": "system:serviceaccount:openshift-apiserver:openshift-apiserver-sa",
"usergroups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-apiserver",
"system:authenticated"
],
"requestObject": null
}
{
"requestReceivedTimestamp": "2022-03-06T08:00:23.073977Z",
"verb": "create",
"username": "opentlc-mgr",
"usergroups": [
"system:authenticated:oauth",
"system:authenticated"
],
"requestObject": null
}
https://learning.redhat.com/mod/scorm/player.php?a=2175¤torg=&scoid=4951&sesskey=bmCgDdE67n&display=popup&mode=normal
https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.9/html-single/security_and_compliance/index#audit-log-view