IaC基础设施即代码：Terraform 创建ACK集群 与部署应用
目录
一、实验
1.环境
2.Terraform 创建网络资源
3. 阿里云给RAM添加权限
4.Terraform 创建 ACK集群
5.在ACK集群中部署应用
6.销毁资源
二、问题
1.Terraform 验证失败
2.Terraform申请资源失败
一、实验
1.环境
（1）主机
表1-1 主机
| 主机 | 系统 | 软件 | 工具 | 备注 |
| jia | Windows |
Terraform 1.6.6 | VS Code、 PowerShell、 Chocolatey |
2.Terraform 创建网络资源
（1）查看项目
（2）网络配置文件
network.tf
//VPC 专有网络
resource "alicloud_vpc" "vpc" {
vpc_name = "k8s_vpc"
cidr_block = "172.16.0.0/12"
}
//switch 交换机
resource "alicloud_vswitch" "vsw" {
vpc_id = alicloud_vpc.vpc.id
cidr_block = "172.16.0.0/16"
zone_id = "cn-hangzhou-j"
}
(3) 版本配置文件
versions.tf
terraform {
required_providers {
alicloud = {
source = "aliyun/alicloud"
version = "1.214.1"
}
}
}
# Configure the Alicloud Provider 默认供应商
provider "alicloud" {
access_key = var.access_key
secret_key = var.secret_key
region = "cn-hangzhou"
}
（4）变量配置文件
variables.tf
variable "access_key" {
description = "access_key"
}
variable "secret_key" {
description = "secret_key"
}
(5) 密钥配置文件
terraform.tfvars
（6）初始化
terraform init
（7）格式化代码
terraform fmt
（8）验证代码
terraform validate
（9）计划与预览
terraform plan
（10）申请资源
terraform apply
（11）登录阿里云系统查看VPC
VPC已新增1个 （cn-hangzhou）
交换机已新增1个 （cn-hangzhou）
3. 阿里云给RAM添加权限
（1）AliyunCSFullAcess
（2）AliyunApiGatewayFullAcess
（3）NATGatewayFullAcess
4.Terraform 创建 ACK集群
（1）查看alicloud provider 示例
Terraform Registry
托管版K8S 示例
……
resource "alicloud_cs_managed_kubernetes" "k8s" {
name = var.name
cluster_spec = "ack.pro.small"
# version can not be defined in variables.tf.
version = "1.26.3-aliyun.1"
worker_vswitch_ids = length(var.vswitch_ids) > 0 ? split(",", join(",", var.vswitch_ids)) : length(var.vswitch_cidrs) < 1 ? [] : split(",", join(",", alicloud_vswitch.vswitches.*.id))
pod_vswitch_ids = length(var.terway_vswitch_ids) > 0 ? split(",", join(",", var.terway_vswitch_ids)) : length(var.terway_vswitch_cidrs) < 1 ? [] : split(",", join(",", alicloud_vswitch.terway_vswitches.*.id))
new_nat_gateway = true
node_cidr_mask = var.node_cidr_mask
proxy_mode = var.proxy_mode
service_cidr = var.service_cidr
dynamic "addons" {
for_each = var.cluster_addons
content {
name = lookup(addons.value, "name", var.cluster_addons)
config = lookup(addons.value, "config", var.cluster_addons)
}
}
}
(2) 修改主配置文件
main.tf
locals {
cluster_version = "1.26.3-aliyun.1"
service_cidr = "192.168.0.0/16"
pod_cidr = "10.212.0.0/16"
}
resource "alicloud_cs_managed_kubernetes" "k8s" {
name = var.cluster_name
version = local.cluster_version
cluster_spec = "ack.standard"
worker_vswitch_ids = [alicloud_vswitch.vsw.id]
new_nat_gateway = true
pod_cidr = local.service_cidr
service_cidr = local.pod_cidr
load_balancer_spec = "slb.s1.small"
slb_internet_enabled = true
dynamic "addons" {
for_each = var.cluster_addons
content {
name = lookup(addons.value, "name", var.cluster_addons)
config = lookup(addons.value, "config", var.cluster_addons)
}
}
}
resource "alicloud_cs_kubernetes_node_pool" "default" {
name = var.nodepool_name
cluster_id = alicloud_cs_managed_kubernetes.k8s.id
vswitch_ids = [alicloud_vswitch.vsw.id]
instance_types = ["ecs.g6.xlarge"]
system_disk_category = "cloud_efficiency"
system_disk_size = 40
desired_size = 1
password = "Admin@123"
runtime_name = "containerd"
runtime_version = "1.6.20"
}
(3) 修改变量配置文件
variables.tf
variable "access_key" {
description = "access_key"
}
variable "secret_key" {
description = "secret_key"
}
variable "cluster_name" {
default = "k8s_cluster_01"
}
variable "nodepool_name" {
default = "k8s-nodepool"
}
variable "cluster_addons" {
type = list(object({
name = string
config = string
}))
default = [
{
"name" = "flannel",
"config" = "",
},
{
"name" = "csi-plugin",
"config" = "",
},
{
"name" = "csi-provisioner",
"config" = "",
},
{
"name" = "logtail-ds",
"config" = "{'IngressDashboardEnabled':'true'}",
},
{
"name" = "nginx-ingress-controller",
"config" = "{'IngressSlbNetworkType':'internet'}",
},
{
"name" = "arms-prometheus",
"config" = "",
},
{
"name" = "ack-node-problem-detector",
"config" = "{'sls_project_name':''}",
}
]
}
(4) 验证代码
terraform validate
(5) 计划与预览
terraform plan
(6)申请资源
terraform apply
yes ,用时大约6分钟
(7) 登录阿里云系统查看ACK集群
初始化中
运行中
（8）查看节点池
节点池
伸缩活动
（9）查看命名空间
（10）查看网络
服务 service
5.在ACK集群中部署应用
（1）查看目录
（2）Terraform模板(docker)
Terraform Registry
USE PROVIDER
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.25.2"
}
}
}
provider "kubernetes" {
# Configuration options
}
(3)下载软件包
https://github.com/hashicorp/terraform-provider-kubernetes/releases
（3）修改K8S集群配置文件
阿里云系统查看连接集群信息
复制上面的连接集群信息到clustera.config
（5）修改主配置文件
provider "kubernetes" {
# Configuration options
config_path = "../config/clustera.config"
config_context = "kubernetes-admin-c718a5ce282f94d539ee5ce1986370194"
alias = "clustera"
insecure = true
}
resource "kubernetes_namespace" "jenkins" {
provider = kubernetes.clustera
metadata {
name = "devops"
}
}
（6）修改版本配置文件
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.25.2"
}
}
}
provider "kubernetes" {
# Configuration options
}
（7）修改输出配置文件
output "service_name" {
value = kubernetes_service_v1.jenkins.metadata[0].name
}
（8）修改服务配置文件
jenkins.tf
resource "kubernetes_deployment_v1" "jenkins" {
provider = kubernetes.clustera
metadata {
name = "jenkins"
labels = {
app = "jenkins"
}
namespace = kubernetes_namespace.jenkins.id
}
spec {
replicas = 1
selector {
match_labels = {
app = "jenkins"
}
}
template {
metadata {
labels = {
app = "jenkins"
}
}
spec {
container {
image = "jenkins/jenkins:latest"
name = "jenkins"
image_pull_policy = "IfNotPresent"
port {
container_port = 8080
}
resources {
limits = {
cpu = "1000m"
memory = "4096Mi"
}
requests = {
cpu = "250m"
memory = "1024Mi"
}
}
# liveness_probe {
# http_get {
# path = "/"
# port = 8080
# }
# initial_delay_seconds = 30
# period_seconds = 3
# }
}
}
}
}
}
resource "kubernetes_service_v1" "jenkins" {
provider = kubernetes.clustera
metadata {
name = "jenkins-service"
namespace = kubernetes_namespace.jenkins.id
}
spec {
selector = {
app = kubernetes_deployment_v1.jenkins.metadata[0].labels.app
}
port {
port = 8080
target_port = 8080
}
type = "ClusterIP"
}
}
resource "kubernetes_ingress_v1" "jenkins_ingress" {
provider = kubernetes.clustera
metadata {
name = "jenkins-ingress"
namespace = kubernetes_namespace.jenkins.id
}
spec {
rule {
host = "jenkins.maojing.site"
http {
path {
backend {
service {
name = kubernetes_service_v1.jenkins.metadata[0].name
port {
number = 8080
}
}
}
path_type = "Prefix"
path = "/"
}
}
}
}
}
(9)初始化
terraform init
(10)格式化代码
terraform fmt
(11)验证代码
terraform validate
(12)计划与预览
terraform plan
(13) 申请资源
terraform apply
yes , 4个资源将被添加
（14）登录阿里云系统查看
命名空间新增1个 devops
工作负载（无状态deployment）新增1个jenkins
进入jenkins，状态为running
服务service
service关联路由
（15）修改输出配置文件
outputs.tf,添加如下代码
output "ingress_ip" {
value = kubernetes_ingress_v1.jenkins_ingress.status[0].load_balancer[0].ingress[0].ip
}
（16）计划与预览
terraform plan成功拿到ingress的ip
（17）添加DNS配置文件
dns.tf
# DNS
resource "alicloud_dns_record" "record" {
name = "maojing.site"
host_record = "jenkins"
type = "A"
value = kubernetes_ingress_v1.jenkins_ingress.status[0].load_balancer[0].ingress[0].ip
}
(18) 添加变量配置文件
variables.tf
variable "access_key" {
description = "access_key"
}
variable "secret_key" {
description = "secret_key"
}
(19) 修改版本配置文件
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.25.2"
}
alicloud = {
source = "aliyun/alicloud"
version = "1.214.1"
}
}
}
provider "kubernetes" {
# Configuration options
}
# Configure the Alicloud Provider 默认供应商
provider "alicloud" {
access_key = var.access_key
secret_key = var.secret_key
region = "cn-hangzhou"
}
（20）初始化
terraform init
（21）格式化代码
terraform fmt
（22）验证代码
terraform validate
（23）计划与预览
terraform plan
（24）申请资源
terraform apply
yes
（25）阿里云系统查看
域名解析已新增
（26）dig测试DNS
dig jenkins.maojing.site
(27) 浏览器测试
显示Jenkins安装界面
(28) 查看集群监控
6.销毁资源
（1）销毁服务资源
terraform destroy
yes
（2）登录阿里云系统
DNS解析已删除
devops命名空间已删除
（3）销毁集群资源
terraform destroy
yes ，用时大约5分钟
(4)登录阿里云系统查看集群
删除中
已删除
二、问题
1.Terraform 验证失败
（1）报错
╷
│ Error: "availability_zone": [REMOVED] Field 'availability_zone' has been removed from provider version 1.212.0.
│
│ with alicloud_cs_managed_kubernetes.k8s,
│ on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│ 7: resource "alicloud_cs_managed_kubernetes" "k8s" {
│ Error: "availability_zone": [REMOVED] Field 'availability_zone' has been removed from provider version 1.212.0.
│
│ with alicloud_cs_managed_kubernetes.k8s,
│ on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│ 7: resource "alicloud_cs_managed_kubernetes" "k8s" {
│ Error: "runtime": [REMOVED] Field 'runtime' has been removed from provider version 1.212.0. Please use resource 'alicloud_cs_kubernetes_node_pool' to manage cluster nodes, by using field 'runtime_name' and 'runtime_version' to replace it.
│
│ with alicloud_cs_managed_kubernetes.k8s,
│ on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│ 7: resource "alicloud_cs_managed_kubernetes" "k8s" {
（2）原因分析
Terraform Registry
从1.212版本开始，部分关键地段被移除，推荐使用alicloud_cs_kubernetes_node_pool 管理工作节点。
From version 1.212.0, runtime,enable_ssh,rds_instances,exclude_autoscaler_nodes,worker_number,worker_instance_types,password,key_name,kms_encrypted_password,kms_encryption_context,worker_instance_charge_type,worker_period,worker_period_unit,worker_auto_renew,worker_auto_renew_period,worker_disk_category,worker_disk_size,worker_data_disks,node_name_mode,node_port_range,os_type,platform,image_id,cpu_policy,user_data,taints,worker_disk_performance_level,worker_disk_snapshot_policy_id,install_cloud_monitor,kube_config,availability_zone are removed. Please use resource alicloud_cs_kubernetes_node_pool to manage your cluster worker nodes.
（3）解决方法
修改配置文件。
2.Terraform申请资源失败
（1）报错
Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_cs_kubernetes.go:1230: Resource c28e6d5ac0cf64922a476e6963f1239b8 DescribeNatGateways Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
│ SDK.ServerError
│ ErrorCode: Forbidden.RAM
│ Recommend: https://api.aliyun.com/troubleshoot?q=Forbidden.RAM&product=Vpc&requestId=0254494A-FE5F-51C9-96DA-394123C37E13
│ RequestId: 0254494A-FE5F-51C9-96DA-394123C37E13
│ Message: User not authorized to operate on the specified resource, or this API doesn't support RAM.
│ RespHeaders: map[Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[*] Connection:[keep-alive] Content-Length:[568] Content-Type:[application/json;charset=utf-8] Date:[Tue, 23 Jan 2024 05:11:28 GMT] Keep-Alive:[timeout=25] X-Acs-Request-Id:[0254494A-FE5F-51C9-96DA-394123C37E13] X-Acs-Trace-Id:[740d51a284c42eb37e67556a9d62faa6]]
│ AccessDeniedDetail: map[AuthPrincipalDisplayName:205814005146961779 AuthPrincipalOwnerId:1889388625243280 AuthPrincipalType:SubUser EncodedDiagnosticMessage:AQEAAAAAZa9KgzAyNTQ0OTRBLUZFNUYtNTFDOS05NkRBLTM5NDEyM0MzN0UxMw==]
│
│ with alicloud_cs_managed_kubernetes.k8s,
│ on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│ 7: resource "alicloud_cs_managed_kubernetes" "k8s" {
（2）原因分析
RAM缺少NATGatewayFullAcess权限
（3）解决方法
RAM添加NATGatewayFullAcess权限。
重新申请资源
yes,先删除旧的实例
开始创建新实例