AWS EKS上使用OpenSearch部署EFK日志分析收集系统

一、为Fluent Bit配置IRSA

1. 创建OIDC身份提供商

   eksctl utils associate-iam-oidc-provider \
       --cluster {clustername} \
       --approve
   

2. 创建IAM角色和策略

   mkdir ~/environment/logging/
   
   export ES_DOMAIN_NAME="eks-logging"
   
   cat <<EoF > ~/environment/logging/fluent-bit-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "es:ESHttp*"
               ],
               "Resource": "arn:aws:es:${AWS_REGION}:${ACCOUNT_ID}:domain/${ES_DOMAIN_NAME}",
               "Effect": "Allow"
           }
       ]
   }
   EoF
   
   aws iam create-policy   \
     --policy-name fluent-bit-policy \
     --policy-document file://~/environment/logging/fluent-bit-policy.json
   

   创建角色

   kubectl create namespace logging
   
   eksctl create iamserviceaccount \
       --name fluent-bit \
       --namespace logging \
       --cluster {clustername} \
       --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/fluent-bit-policy" \
       --approve \
       --override-existing-serviceaccounts
   

二、配置Amazon OpenSearch访问

${ES_DOMAIN_NAME}为OpenSearch的名称

# We need to retrieve the Fluent Bit Role ARN
export FLUENTBIT_ROLE=$(eksctl get iamserviceaccount --cluster {clustername} --namespace logging -o json | jq '.[].status.roleARN' -r)

# Get the Amazon OpenSearch Endpoint
export ES_ENDPOINT=$(aws opensearch describe-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")

# Update the Elasticsearch internal database
curl -sS -u "${ES_DOMAIN_USER}:${ES_DOMAIN_PASSWORD}" \
    -X PATCH \
    https://${ES_ENDPOINT}/_opendistro/_security/api/rolesmapping/all_access?pretty \
    -H 'Content-Type: application/json' \
    -d'
[
  {
    "op": "add", "path": "/backend_roles", "value": ["'${FLUENTBIT_ROLE}'"]
  }
]
'

三、部署Fluent Bit

cd ~/environment/logging

# get the Amazon OpenSearch Endpoint
export ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")

curl -Ss https://www.eksworkshop.com/intermediate/230_logging/deploy.files/fluentbit.yaml \
    | envsubst > ~/environment/logging/fluentbit.yaml
    
# deploy
kubectl apply -f ~/environment/logging/fluentbit.yaml

# watch
kubectl --namespace=logging get pods

四、在OpenSearch查看仪表盘

创建Dashboards

添加数据

添加索引

添加*fluent-bit*作为索引模式，然后单击下一步

选择**@timestamp作为时间过滤器字段名称并通过单击创建索引模式**关闭配置窗口