Loading search…

esc to close

All blogs

Three walls I hit building a DevSecOps pipeline on EKS

Pod limits on t3.small, ArgoCD’s 256KB annotation ceiling, and Kyverno failing to verify signed images from ECR. What broke and how I fixed it.

3 min read
On this page
  1. The setup
  2. Wall 1: pods stuck in Pending on a half-empty cluster
  3. Wall 2: ArgoCD refusing to sync Kyverno
  4. Wall 3: Kyverno blocking my own signed images
  5. What I’d tell past me

I built a cloud-native DevSecOps pipeline to simulate how a real organisation might govern and secure workloads at scale, on a student budget. EKS on spot instances, a GitHub Actions pipeline full of scanners, ArgoCD for GitOps, Kyverno for admission control.

The architecture diagram looked clean. The first week did not. Here are the three walls that cost me the most time.

The setup

For context, the moving parts:

Layer Tools
Infra Terraform, EKS (3× t3.small spot), ECR, VPC, SSM
CI GitHub Actions: Gitleaks, Semgrep, Trivy, Cosign, SBOM
GitOps/policy ArgoCD app-of-apps, Kyverno
Visibility OWASP ZAP → DefectDojo → Jira

Wall 1: pods stuck in Pending on a half-empty cluster

Plenty of CPU and memory free, yet new pods sat in Pending. The culprit was the AWS VPC CNI: every pod gets a real VPC IP address, and the number of IPs a node can hold is capped by its instance type’s ENI limits.

For a t3.small, that works out to:

max pods = ENIs × (IPs per ENI − 1) + 2
         = 3 × (4 − 1) + 2
         = 11

Eleven pods per node. And aws-node, kube-proxy, CoreDNS and the monitoring agents eat a good chunk of that before your workloads show up.

The fix: enable prefix delegation on the VPC CNI so each ENI slot hands out a /28 prefix instead of a single IP, then raise the node’s max-pods to match.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
Tip Setting the env var isn’t enough on its own. The kubelet still enforces the old max-pods. Update it in your node group / launch template too, then roll the nodes.

Wall 2: ArgoCD refusing to sync Kyverno

ArgoCD happily synced everything except Kyverno, which failed with:

metadata.annotations: Too long: must have at most 262144 bytes

By default ArgoCD uses client-side apply, which stores the entire manifest in the kubectl.kubernetes.io/last-applied-configuration annotation. Kyverno’s CRDs are huge, and Kubernetes caps total annotation size at 256KB.

The fix: switch that application to server-side apply, which doesn’t need the annotation.

spec:
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

Wall 3: Kyverno blocking my own signed images

Every image was signed with Cosign in CI, and a Kyverno verifyImages policy made sure nothing unsigned ran in the cluster. It worked perfectly. It blocked everything, including the images I’d just signed.

Kyverno needs to pull the signature from ECR to verify it, and its admission controller had no permission to read from the registry. So verification failed, the pod was rejected, ArgoCD retried, and round we went.

The fix: give Kyverno’s service account read-only ECR access (IRSA / EKS Pod Identity) so it can authenticate to the registry when fetching signatures.

What I’d tell past me

  1. Read the instance type’s pod limit before choosing the cheapest node.
  2. Default to ServerSideApply=true for anything that ships big CRDs.
  3. When a security control blocks everything, check whether the control itself can see what it’s checking.

The full code is on GitHub: the pipeline repo and the GitOps repo. I’ve since started moving the whole thing onto a k3s homelab. That’s a post for another day.