【aws】ECS上构筑基于Concourse的CI/CD方案

Amazon Elastic Container Service (Amazon ECS) 是一项高度可扩展的快速容器管理服务,它可轻松运行、停止和管理群集上的容器。您的容器在任务定义中定义,用于运行服务中的单个任务或服务。在此上下文中,服务是一种配置,使您能够同时在集群中运行和维护指定数量的任务。您可以在由 AWS Fargate 管理的无服务器基础设施上运行您的任务和服务。或者,要更好地控制您的基础设施,您可以在管理的 Amazon EC2 实例集群上运行您的任务和服务。

Concourse CI 是一款 CI/CD 工具,它的魅力在于极简设计,被广泛应用于 Cloud Foundry 各个模块的 CI/CD。Concourse CI 官方提供了标准的 Docker 镜像,可以通过AWS ECS容器服务部署一套 Concourse CI 应用。

本次构筑Concourse CI的后端存储,选用的postgres数据库,利用AWS Fargate部署容器版的postgres。另外需要部署Concourse CI的管理Web和worker组件。其中Web也是利用AWS Fargate部署Serverless的容器服务,Worker需要开启特权privileged,而AWS Fargate并不支持privileged,所以选用EC2集群执行Worker容器。

前置条件:

  1. 需要创建一个名称为concourse-data的efs,作为postgres, web以及worker容器的存储卷
    【aws】ECS上构筑基于Concourse的CI/CD方案_第1张图片2. 配置ecsTaskExecutionRole,开放给容器执行必要的权限,比如SystemManager的访问权限等。

构筑步骤:

  1. 定义postgres任务
{
     
    "ipcMode": null,
    "executionRoleArn": "xxxx/ecsTaskExecutionRole",
    "containerDefinitions": [
        {
     
            "dnsSearchDomains": null,
            "environmentFiles": null,
            "logConfiguration": {
     
                "logDriver": "awslogs",
                "secretOptions": null,
                "options": {
     
                    "awslogs-group": "/ecs/concourse-db",
                    "awslogs-region": "cn-north-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "entryPoint": null,
            "portMappings": [],
            "command": null,
            "linuxParameters": null,
            "cpu": 0,
            "environment": [
                {
     
                    "name": "PGDATA",
                    "value": "/database"
                },
                {
     
                    "name": "POSTGRES_PASSWORD",
                    "value": ""
                },
                {
     
                    "name": "POSTGRES_DB",
                    "value": "concourse"
                }
            ],
            "resourceRequirements": null,
            "ulimits": null,
            "dnsServers": null,
            "mountPoints": [
                {
     
                    "readOnly": null,
                    "containerPath": "/database",
                    "sourceVolume": "concourse-db"
                }
            ],
            "workingDirectory": null,
            "secrets": null,
            "dockerSecurityOptions": null,
            "memory": null,
            "memoryReservation": null,
            "volumesFrom": [],
            "stopTimeout": null,
            "image": "postgres:12.2",
            "startTimeout": null,
            "firelensConfiguration": null,
            "dependsOn": null,
            "disableNetworking": null,
            "interactive": null,
            "healthCheck": null,
            "essential": true,
            "links": null,
            "hostname": null,
            "extraHosts": null,
            "pseudoTerminal": null,
            "user": null,
            "readonlyRootFilesystem": null,
            "dockerLabels": null,
            "systemControls": null,
            "privileged": null,
            "name": "concourse-db"
        }
    ],
    "memory": "512",
    "taskRoleArn": "xxxx/ecsTaskExecutionRole",
    "family": "concourse-db",
    "pidMode": null,
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "networkMode": "awsvpc",
    "cpu": "256",
    "inferenceAccelerators": null,
    "proxyConfiguration": null,
    "volumes": [
        {
     
            "efsVolumeConfiguration": {
     
                "transitEncryptionPort": null,
                "fileSystemId": "fs-12835e8f",
                "authorizationConfig": {
     
                    "iam": "DISABLED",
                    "accessPointId": null
                },
                "transitEncryption": "DISABLED",
                "rootDirectory": "/data/concourese/database"
            },
            "name": "concourse-db",
            "host": null,
            "dockerVolumeConfiguration": null
        }
    ],
    "tags": []
}
  1. 定义Web任务
{
     
    "ipcMode": null,
    "executionRoleArn": "xxxx/ecsTaskExecutionRole",
    "containerDefinitions": [
        {
     
            "dnsSearchDomains": null,
            "environmentFiles": null,
            "logConfiguration": {
     
                "logDriver": "awslogs",
                "secretOptions": null,
                "options": {
     
                    "awslogs-group": "/ecs/concourse-web",
                    "awslogs-region": "cn-north-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "entryPoint": null,
            "portMappings": [
                {
     
                    "hostPort": 443,
                    "protocol": "tcp",
                    "containerPort": 443
                }
            ],
            "command": [
                "web"
            ],
            "linuxParameters": null,
            "cpu": 0,
            "environment": [
                {
     
                    "name": "CONCOURSE_AWS_SSM_PIPELINE_SECRET_TEMPLATE",
                    "value": "/concourse/{
     {.Team}}/{
     {.Pipeline}}/{
     {.Secret}}"
                },
                {
     
                    "name": "CONCOURSE_ADD_LOCAL_USER",
                    "value": "admin:xxxx,platform:xxxx"
                },
                {
     
                    "name": "CONCOURSE_POSTGRES_HOST",
                    "value": "concourse-db.local"
                },
                {
     
                    "name": "CONCOURSE_LOG_LEVEL",
                    "value": "debug"
                },
                {
     
                    "name": "CONCOURSE_TLS_CERT",
                    "value": "/concourse-keys/server.crt"
                },
                {
     
                    "name": "CONCOURSE_AWS_SSM_TEAM_SECRET_TEMPLATE",
                    "value": "/concourse/{
     {.Team}}/{
     {.Secret}}"
                },
                {
     
                    "name": "CONCOURSE_TLS_KEY",
                    "value": "/concourse-keys/server.key"
                },
                {
     
                    "name": "CONCOURSE_AWS_SSM_SECRET_KEY",
                    "value": "Y/"
                },
                {
     
                    "name": "CONCOURSE_POSTGRES_PASSWORD",
                    "value": ""
                },
                {
     
                    "name": "CONCOURSE_POSTGRES_DATABASE",
                    "value": "concourse"
                },
                {
     
                    "name": "CONCOURSE_AWS_SSM_REGION",
                    "value": "cn-north-1"
                },
                {
     
                    "name": "CONCOURSE_TLS_BIND_PORT",
                    "value": "443"
                },
                {
     
                    "name": "CONCOURSE_MAIN_TEAM_LOCAL_USER",
                    "value": "admin"
                },
                {
     
                    "name": "CONCOURSE_AWS_SSM_ACCESS_KEY",
                    "value": ""
                },
                {
     
                    "name": "CONCOURSE_EXTERNAL_URL",
                    "value": "https://xxxx:8443"
                },
                {
     
                    "name": "CONCOURSE_POSTGRES_USER",
                    "value": "concourse@postgres"
                }
            ],
            "resourceRequirements": null,
            "ulimits": null,
            "dnsServers": null,
            "mountPoints": [
                {
     
                    "readOnly": null,
                    "containerPath": "/concourse-keys",
                    "sourceVolume": "concourse-keys"
                }
            ],
            "workingDirectory": null,
            "secrets": null,
            "dockerSecurityOptions": null,
            "memory": null,
            "memoryReservation": null,
            "volumesFrom": [],
            "stopTimeout": null,
            "image": "voss2018/concourse:6.5.1.1",
            "startTimeout": null,
            "firelensConfiguration": null,
            "dependsOn": null,
            "disableNetworking": null,
            "interactive": null,
            "healthCheck": null,
            "essential": true,
            "links": null,
            "hostname": null,
            "extraHosts": null,
            "pseudoTerminal": null,
            "user": null,
            "readonlyRootFilesystem": null,
            "dockerLabels": null,
            "systemControls": null,
            "privileged": null,
            "name": "concourse-web"
        }
    ],
    "memory": "2048",
    "taskRoleArn": "arn:aws-cn:iam::348769610664:role/ecsTaskExecutionRole",
    "family": "concourse-web",
    "pidMode": null,
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "networkMode": "awsvpc",
    "cpu": "1024",
    "inferenceAccelerators": null,
    "proxyConfiguration": null,
    "volumes": [
        {
     
            "efsVolumeConfiguration": {
     
                "transitEncryptionPort": null,
                "fileSystemId": "fs-12835e8f",
                "authorizationConfig": {
     
                    "iam": "DISABLED",
                    "accessPointId": null
                },
                "transitEncryption": "DISABLED",
                "rootDirectory": "/data/concourese/web"
            },
            "name": "concourse-keys",
            "host": null,
            "dockerVolumeConfiguration": null
        }
    ],
    "tags": []
}
  1. 定义worker任务
{
     
    "ipcMode": null,
    "executionRoleArn": "xxxx/ecsTaskExecutionRole",
    "containerDefinitions": [
        {
     
            "dnsSearchDomains": null,
            "environmentFiles": null,
            "logConfiguration": {
     
                "logDriver": "awslogs",
                "secretOptions": null,
                "options": {
     
                    "awslogs-group": "/ecs/concourse-worker",
                    "awslogs-region": "cn-north-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "entryPoint": null,
            "portMappings": [],
            "command": [
                "worker"
            ],
            "linuxParameters": null,
            "cpu": 0,
            "environment": [
                {
     
                    "name": "CONCOURSE_TSA_HOST",
                    "value": "concourse-web.local:2222"
                }
            ],
            "resourceRequirements": null,
            "ulimits": null,
            "dnsServers": null,
            "mountPoints": [
                {
     
                    "readOnly": null,
                    "containerPath": "/concourse-keys",
                    "sourceVolume": "concourse-keys"
                }
            ],
            "workingDirectory": null,
            "secrets": null,
            "dockerSecurityOptions": null,
            "memory": null,
            "memoryReservation": null,
            "volumesFrom": [],
            "stopTimeout": null,
            "image": "voss2018/concourse:6.5.1.1",
            "startTimeout": null,
            "firelensConfiguration": null,
            "dependsOn": null,
            "disableNetworking": null,
            "interactive": null,
            "healthCheck": null,
            "essential": true,
            "links": null,
            "hostname": null,
            "extraHosts": null,
            "pseudoTerminal": null,
            "user": null,
            "readonlyRootFilesystem": null,
            "dockerLabels": null,
            "systemControls": null,
            "privileged": true,
            "name": "concourse-worker"
        }
    ],
    "memory": "4096",
    "taskRoleArn": "xxxx/ecsTaskExecutionRole",
    "family": "concourse-worker",
    "pidMode": null,
    "requiresCompatibilities": [
        "EC2"
    ],
    "networkMode": "awsvpc",
    "cpu": "1024",
    "inferenceAccelerators": null,
    "proxyConfiguration": null,
    "volumes": [
        {
     
            "efsVolumeConfiguration": {
     
                "transitEncryptionPort": null,
                "fileSystemId": "fs-12835e8f",
                "authorizationConfig": {
     
                    "iam": "DISABLED",
                    "accessPointId": null
                },
                "transitEncryption": "DISABLED",
                "rootDirectory": "/data/concourese/worker"
            },
            "name": "concourse-keys",
            "host": null,
            "dockerVolumeConfiguration": null
        }
    ],
    "placementConstraints": [],
    "tags": []
}
  1. 创建cluster platform,利用下面的cluster模板创建,并且添加一台EC2实例,以及配置必要的VPC,安全组
      EC2 Linux + 联网
         要创建的资源:
             集群
             VPC
             子网
    带 Linux AMI 的 Auto Scaling 组
  1. 在Platform cluster中创建service:concourse-db
    【aws】ECS上构筑基于Concourse的CI/CD方案_第2张图片
  2. 在platform cluster中创建concourse-web service
    【aws】ECS上构筑基于Concourse的CI/CD方案_第3张图片
  3. 在platform cluster中创建concourse-worker service
    【aws】ECS上构筑基于Concourse的CI/CD方案_第4张图片
  4. 确认三个service是否正常Running
    【aws】ECS上构筑基于Concourse的CI/CD方案_第5张图片
  5. 配置负载均衡,登录concourse web
    在这里插入图片描述【aws】ECS上构筑基于Concourse的CI/CD方案_第6张图片

你可能感兴趣的:(aws,kubernetes,kubernetes,docker,ci/cd,aws)