Mastering AWS EKS Cluster Upgrades: Best Practices, Prerequisites, and Key Considerations

Upgrading an Amazon Elastic Kubernetes Service (EKS) cluster ensures you stay current with Kubernetes features, security patches, and API deprecations. However, the process requires careful planning to prevent service disruptions. This detailed blog will guide you through the prerequisites, key considerations, and best practices to ensure a smooth upgrade.

Key Points to Remember Before Starting

Maximum of 5 Available IPs per Node:
Ensure each node in the cluster has at least 5 available IP addresses in its subnet. A shortage of IPs during the upgrade can cause nodes to fail in joining the cluster.
Downgrade Is Not Supported:
Kubernetes upgrades on AWS EKS are irreversible. Once a cluster is upgraded to a new version, you cannot roll back to a previous version.
Check Kubernetes Release Notes:
Always review the Kubernetes release notes for deprecations, feature changes, and breaking updates in the target version.
Lower Version Upgrades Are Not Possible:
AWS EKS does not allow upgrading to a lower Kubernetes version. If you need a lower version, you must deploy a new cluster and migrate workloads manually.
Sequential Upgrades Only:
EKS only supports upgrading one minor version at a time. For example, if your cluster is on version 1.25, you must upgrade to 1.26 before moving to 1.27.

Prerequisites for AWS EKS Cluster Upgrades

1. Check Cluster Version Compatibility

Use the AWS CLI to determine your current Kubernetes version:

  aws eks describe-cluster --name <cluster-name> --query cluster.version

Confirm the target version is supported by AWS EKS. Sequential upgrades are mandatory; skipping versions is not allowed.

2. Backup the Cluster

Export all resources and configurations to YAML:

  kubectl get all --all-namespaces -o yaml > cluster-backup.yaml

Use tools like Velero for automated snapshots, including PVCs and critical data.

3. Validate Subnet IP Availability

Ensure each node’s subnet has at least 5 available IP addresses:

  aws ec2 describe-subnets --subnet-ids <subnet-id> --query "Subnets[].AvailableIpAddressCount"

4. Review Application Compatibility

Identify and update deprecated APIs:

  kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found

Test workloads on a staging cluster with the target Kubernetes version.

5. Update Kubernetes Tools

Ensure kubectl, Helm, and other tools match the target version:

  curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  chmod +x kubectl
  sudo mv kubectl /usr/local/bin/
  kubectl version --client

6. Upgrade Node Groups

Managed Node Groups:

  aws eks update-nodegroup-version --cluster-name <cluster-name> --nodegroup-name <nodegroup-name>

Self-Managed Nodes:
- Update to the latest AMI for your region from EKS Optimized AMIs.
- Replace older nodes with updated ones using a rolling update strategy.

7. Check IAM Permissions

Verify that your IAM role includes permissions like:
- eks:UpdateClusterVersion
- eks:UpdateNodegroupVersion

8. Update Critical Add-ons

CoreDNS:

  kubectl apply -f https://github.com/coredns/deployment/kubernetes/coredns.yaml

kube-proxy:

  aws eks update-addon --cluster-name <cluster-name> --addon-name kube-proxy

VPC CNI Plugin:

  aws eks update-addon --cluster-name <cluster-name> --addon-name vpc-cni

9. Plan for Downtime

Inform stakeholders and schedule upgrades during off-peak hours. Some workloads may experience brief disruptions.

Steps to Perform the Upgrade

Upgrade the Control Plane

Use the AWS CLI to upgrade the EKS control plane:

  aws eks update-cluster-version --name <cluster-name> --kubernetes-version <target-version>

Upgrade Node Groups