본문 바로가기
DevOps

[T1014-실습] Terraform 으로 AWS EKS 배포하기

by 서어켜엉 2024. 7. 24.
해당 내용은 cloudNet@ 팀의 가시다 님이 진행하는 테라폼 스터디 T101 4기에서 실습한 내용을 정리한 것입니다.

 

1. Amazon EKS Blueprints for Terraform

실습에 사용된 코드는 link 에서 제공하는 Blueprint 기반으로 만들어진 것을 사용했다.

해당 Blueprint를 사용할 때 몇 가지 고려사항을 알면 사용하기 편리하다.

권장사항

  • 그대로 사용하도록 만들어진 것이 아니다.
  • 해당 Blueprint에 사용되는 패턴과 스니펫은 Terraform 모듈로 사용하도록 설계되지 않았다.
  • 일반적으로 Local 블록을 통해 내용을 변경하고, 특정 정보가 필요한 경우(R53 호스트영역 ID 등)만 variables 블록을 사용한다.
  • consumption model 과의 혼란을 줄이기 위해 variables 와 output 은 최대한 노출하지 않았다.
  • AWS EKS bluprints for Terraform 지원을 위해 제작한 모듈 정보
    • terraform-aws-eks-blueprint-addon
    • terraform-aws-eks-plueprint-addons
    • terraform-aws-eks-blueprints-teams
  • 기타 프로젝트
    1. GitOps 
      • terraform-aws-eks-ack-addons
      • crossplane-on-eks
    2. Data on EKS
      • data-on-eks
      • terraform-aws-eks-data-addons
    3. Observability Accelerator
      • terraform-aws-observability-accelerator
    4. karpenter-bluprints

주의사항

  • 권장 사례나 테라폼 구성 지침을 제공하지 않는다.
  • 최대한 빠르고 쉽게 목표 패턴 실행에 목적이 있다.
  • 안정적인 배포 보장을 위해 기존 VPC (default VPC) 대신 신규 VPC 를 배포한다.

 

2. Karpenter on EKS Fargate - Link

 

Karpenter on EKS Fargate - Amazon EKS Blueprints for Terraform

Karpenter on EKS Fargate This pattern demonstrates how to provision Karpenter on a serverless cluster (serverless data plane) using Fargate Profiles. Deploy See here for the prerequisites and steps to deploy this pattern. Validate Test by listing the nodes

aws-ia.github.io

이 패턴은 Fargate Profiles를 사용하여 서버리스 클러스터(서버리스 데이터 플레인)에 Karpenter를 프로비저닝하는 방법을 보여줍니다.

사전준비 : awscli(관리자수전-자격증명), terraform, kubectl, helm - link

 

Getting Started - Amazon EKS Blueprints for Terraform

Getting Started This getting started guide will help you deploy your first pattern using EKS Blueprints. Prerequisites Ensure that you have installed the following tools locally: Deploy For consuming EKS Blueprints, please see the Consumption section. For

aws-ia.github.io

 

코드준비 - Repository

 

GitHub - aws-ia/terraform-aws-eks-blueprints: Configure and deploy complete EKS clusters.

Configure and deploy complete EKS clusters. Contribute to aws-ia/terraform-aws-eks-blueprints development by creating an account on GitHub.

github.com

git clone https://github.com/aws-ia/terraform-aws-eks-blueprints
cd terraform-aws-eks-blueprints/patterns/karpenter
tree

 

  • versions.tf
더보기
terraform {
  required_version = ">= 1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.34"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.9"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
  }

  # ##  Used for end-to-end testing on project; update to suit your needs
  # backend "s3" {
  #   bucket = "terraform-ssp-github-actions-state"
  #   region = "us-west-2"
  #   key    = "e2e/karpenter/terraform.tfstate"
  # }
}

 

  • main.tf - local 블록 수정
더보기
provider "aws" {
  region = local.region
}

# Required for public ECR where Karpenter artifacts are hosted
provider "aws" {
  region = "us-east-1"
  alias  = "virginia"
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1" 
    command     = "aws"
    # This requires the awscli to be installed locally where Terraform is executed
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      # This requires the awscli to be installed locally where Terraform is executed
      args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}

data "aws_ecrpublic_authorization_token" "token" {
  provider = aws.virginia
}

data "aws_availability_zones" "available" {}

locals {
  name   = "t1014-${basename(path.cwd)}"  ## 수정
  region = "ap-northeast-2"  ## 서울리전

  vpc_cidr = "10.10.0.0/16"
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    Blueprint  = local.name
    GithubRepo = "github.com/aws-ia/terraform-aws-eks-blueprints"
  }
}

################################################################################
# Cluster
################################################################################

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.11"

  cluster_name                   = local.name
  cluster_version                = "1.30"
  cluster_endpoint_public_access = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Fargate profiles use the cluster primary security group so these are not utilized
  create_cluster_security_group = false
  create_node_security_group    = false

  enable_cluster_creator_admin_permissions = true

  fargate_profiles = {
    karpenter = {
      selectors = [
        { namespace = "karpenter" }
      ]
    }
    kube_system = {
      name = "kube-system"
      selectors = [
        { namespace = "kube-system" }
      ]
    }
  }

  tags = merge(local.tags, {
    # NOTE - if creating multiple security groups with this module, only tag the
    # security group that Karpenter should utilize with the following tag
    # (i.e. - at most, only one security group should have this tag in your account)
    "karpenter.sh/discovery" = local.name
  })
}

################################################################################
# EKS Blueprints Addons
################################################################################

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.16"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  # We want to wait for the Fargate profiles to be deployed first
  create_delay_dependencies = [for prof in module.eks.fargate_profiles : prof.fargate_profile_arn]

  eks_addons = {
    coredns = {
      configuration_values = jsonencode({
        computeType = "Fargate"
        # Ensure that the we fully utilize the minimum amount of resources that are supplied by
        # Fargate https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html
        # Fargate adds 256 MB to each pod's memory reservation for the required Kubernetes
        # components (kubelet, kube-proxy, and containerd). Fargate rounds up to the following
        # compute configuration that most closely matches the sum of vCPU and memory requests in
        # order to ensure pods always have the resources that they need to run.
        resources = {
          limits = {
            cpu = "0.25"
            # We are targeting the smallest Task size of 512Mb, so we subtract 256Mb from the
            # request/limit to ensure we can fit within that task
            memory = "256M"
          }
          requests = {
            cpu = "0.25"
            # We are targeting the smallest Task size of 512Mb, so we subtract 256Mb from the
            # request/limit to ensure we can fit within that task
            memory = "256M"
          }
        }
      })
    }
    vpc-cni    = {}
    kube-proxy = {}
  }

  enable_karpenter = true

  karpenter = {
    repository_username = data.aws_ecrpublic_authorization_token.token.user_name
    repository_password = data.aws_ecrpublic_authorization_token.token.password
  }

  karpenter_node = {
    # Use static name so that it matches what is defined in `karpenter.yaml` example manifest
    iam_role_use_name_prefix = false
  }

  tags = local.tags
}

resource "aws_eks_access_entry" "karpenter_node_access_entry" {
  cluster_name      = module.eks.cluster_name
  principal_arn     = module.eks_blueprints_addons.karpenter.node_iam_role_arn
  kubernetes_groups = []
  type              = "EC2_LINUX"
}

################################################################################
# Supporting Resources
################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = local.name
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
    # Tags subnets for Karpenter auto-discovery
    "karpenter.sh/discovery" = local.name
  }

  tags = local.tags
}

 

  • outputs.tf
output "configure_kubectl" {
  description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig"
  value       = "aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name}"
}
  • (참고) slice Function - Docs
# slice extracts some consecutive elements from within a list.
# startindex is inclusive, while endindex is exclusive. 
# This function returns an error if either index is outside the bounds of valid indices for the given list.
slice(list, startindex, endindex)

# Examples
> slice(["a", "b", "c", "d"], 1, 3)
[
  "b",
  "c",
]

 

INIT

terraform init
tree .terraform
cat .terraform/modules/modules.json | jq
tree .terraform/providers/registry.terraform.io/hashicorp -L 2

 

VPC 배포 : 3분 소요

terraform apply -target="module.vpc" -auto-approve

aws ec2 describe-vpcs --filter 'Name=isDefault,Values=false' --output yaml
echo "data.aws_availability_zones.available" | terraform console

VPCID=<각자 자신의 VPC ID>
aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPCID" | jq

# public 서브넷과 private 서브넷 CIDR 확인
## private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
## public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
terraform state show 'module.vpc.aws_subnet.public[0]'
terraform state show 'module.vpc.aws_subnet.private[0]'

VPC 정보

EKS 배포 : 13분 소요

더보기
terraform apply -target="module.eks" -auto-approve

# EKS 자격증명
## aws eks --region <REGION> update-kubeconfig --name <CLUSTER_NAME> --alias <CLUSTER_NAME>
aws eks --region ap-northeast-2 update-kubeconfig --name t1014-karpenter
cat ~/.kube/config

# (참고) context name 변경
kubectl config rename-context "arn:aws:eks:ap-northeast-2:$(aws sts get-caller-identity --query 'Account' --output text):cluster/t101-karpenter" "T101-Lab"

# k8s 노드, 파드 정보 확인
kubectl cluster-info
kubectl get node
kubectl get pod -A
EKS 생성중

 

# EKS 자격증명
## aws eks --region <REGION> update-kubeconfig --name <CLUSTER_NAME> --alias <CLUSTER_NAME>
aws eks --region ap-northeast-2 update-kubeconfig --name t1014-karpenter
cat ~/.kube/config

# k8s 클러스터 정보 확인
kubectl cluster-info

# 각자 자신의 IAM User 의 access entry 생성
ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
MYIAMUSER=<각자 자신의 IAM User>
MYIAMUSER=terraform-study

echo $ACCOUNT_ID $MYIAMUSER
aws eks create-access-entry --cluster-name t1014-karpenter --principal-arn arn:aws:iam::${ACCOUNT_ID}:user/${MYIAMUSER}
aws eks list-access-entries --cluster-name t1014-karpenter

# 각자 자신의 IAM User에 AmazonEKSClusterAdminPolicy 연동
aws eks associate-access-policy --cluster-name t1014-karpenter --principal-arn arn:aws:iam::${ACCOUNT_ID}:user/${MYIAMUSER} \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy --access-scope type=cluster

aws eks list-associated-access-policies --cluster-name t1014-karpenter --principal-arn arn:aws:iam::${ACCOUNT_ID}:user/${MYIAMUSER} | jq
aws eks describe-access-entry --cluster-name t1014-karpenter --principal-arn arn:aws:iam::${ACCOUNT_ID}:user/${MYIAMUSER} | jq
 
# (참고) context name 변경
kubectl config rename-context "arn:aws:eks:ap-northeast-2:$(aws sts get-caller-identity --query 'Account' --output text):cluster/t1014-karpenter" "T101-Lab"

# k8s 클러스터, 노드, 파드 정보 확인
kubectl cluster-info
kubectl get node
kubectl get nodes -L node.kubernetes.io/instance-type -L topology.kubernetes.io/zone

kubectl get pod -A

 

같은 access entry 리소스를 생성하려고 할 때 발생하는 에러
aws eks delete-access-entry --cluster-name t1014-karpenter --principal-arn arn:aws:iam::${ACCOUNT_ID}:user/${MYIAMUSER}

 위 커맨드 실행 후 다시 시도한다.

 

 

kube-ops-view 설치 : 노드의 파드 상태 정보를 웹 페이지에서 실시간으로 출력 - Link

 

kube-ops-view 1.2.2 · funkypenguin/geek-cookbook

A read-only system dashboard for multiple K8s clusters

artifacthub.io

# helm 배포
helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set env.TZ="Asia/Seoul" --namespace kube-system

# 포트 포워딩
kubectl port-forward deployment/kube-ops-view -n kube-system 8080:8080

# 접속 주소 확인 : 각각 1배, 1.5배, 3배 크기
echo -e "KUBE-OPS-VIEW URL = http://localhost:8080"
echo -e "KUBE-OPS-VIEW URL = http://localhost:8080/#scale=1.5"
echo -e "KUBE-OPS-VIEW URL = http://localhost:8080/#scale=3"

 

리소스 삭제

# kube-ops-view 삭제
helm uninstall kube-ops-view -n kube-system

# 삭제 : vpc 삭제가 잘 안될 경우 aws 콘솔에서 vpc 수동 삭제 -> vnic 등 남아 있을 경우 해당 vnic 강제 삭제 : 9분 소요
terraform destroy -auto-approve

# VPC 삭제 확인
aws ec2 describe-vpcs --filter 'Name=isDefault,Values=false' --output yaml

# kubeconfig 삭제
rm -rf ~/.kube/config

'DevOps' 카테고리의 다른 글

[T1014-실습] OpenTofu 1.7.0  (0) 2024.08.02
[T1014-이론] OpenTofu 소개  (0) 2024.07.31
[T1014-이론] EKS? Kubernetes? Karpenter? Helm? ArgoCD?  (1) 2024.07.24
[T1014-실습] Terraform Runner  (1) 2024.07.14
[T1014-이론] 6장 Module  (0) 2024.07.10