Stop deploying :latest: immutable image tags with Git SHAs

· 4 min read

containersecrkubernetesgithub-actionsgitops

“What version is running in production?” should take five seconds to answer. On a lot of the clusters we inherit, it takes an afternoon: the Deployment says api:latest, the registry says latest was pushed three times this week, and nobody is sure which of those pushes the pods pulled.

Mutable tags are the root cause. Here’s how we remove them.

Pipeline diagram: a git commit is built once by GitHub Actions using an OIDC role and pushed to Amazon ECR tagged with the commit SHA; ECR tag immutability rejects any re-push to the same tag. Promotion copies the same SHA tag from envs/staging/values.yaml to envs/prod/values.yaml in the GitOps repo, ArgoCD syncs each environment and pulls the image by SHA, and rollback is a git revert of the promotion commit.
The tag is the provenance. Promotion and rollback are Git commits.

The problem with :latest (and :v2, and :prod)

A tag is a pointer. Unless the registry forbids it, anyone can move that pointer to a different image. That breaks three things:

Semantic version tags are better but still mutable unless you enforce otherwise, and they’re usually assigned by a human after the build, which is one more step that can drift.

The rule: one commit, one tag, forever

Every image gets tagged with the full commit SHA that produced it:

123456789012.dkr.ecr.us-east-1.amazonaws.com/api:3f9c2e1a7b...

Properties we get for free:

You can add a human-friendly tag on top (v1.14.0), but deployments reference the SHA.

Make the registry enforce it

Convention isn’t enough; someone will re-run a job with a hand-edited tag. On ECR, turn on tag immutability:

resource "aws_ecr_repository" "api" {
  name                 = "api"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

Now a push to an existing tag fails. That failure is the point: it means the pipeline tried to change history.

Pair it with a lifecycle policy so the registry doesn’t grow forever, and keep enough history to roll back:

{
  "rules": [{
    "rulePriority": 1,
    "description": "Keep last 50 images",
    "selection": {
      "tagStatus": "any",
      "countType": "imageCountMoreThan",
      "countNumber": 50
    },
    "action": { "type": "expire" }
  }]
}

Build once, in CI

# .github/workflows/build.yml
name: build
on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1

      - id: ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push
        env:
          IMAGE: ${{ steps.ecr.outputs.registry }}/api:${{ github.sha }}
        run: |
          docker build -t "$IMAGE" .
          docker push "$IMAGE"

No long-lived AWS keys (GitHub OIDC assumes a role), and the tag is github.sha. There is no step where a person chooses a tag.

Promote the image, not the code

The same image moves through every environment. Promotion is a change to a values file in the GitOps repo, not a rebuild:

# envs/staging/api/values.yaml
image:
  repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/api
  tag: 3f9c2e1a7b...
# envs/prod/api/values.yaml — same tag, one PR later
image:
  repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/api
  tag: 3f9c2e1a7b...

ArgoCD syncs each environment from its folder. What you tested in staging is byte-for-byte what reaches prod, because it’s the same image.

Rebuilding per environment (“build with ENV=prod”) quietly breaks this: base images, dependency resolution and build flags can all differ between two builds of the same commit. Push environment differences into runtime config (ConfigMaps, Secrets, env vars), never into the image.

Pin by digest when it matters

A SHA tag on an immutable repository is already stable. For third-party images you don’t control, or for the highest-assurance paths, pin by content digest:

image: public.ecr.aws/nginx/nginx@sha256:<64-hex-digest>
# get it with: docker buildx imagetools inspect public.ecr.aws/nginx/nginx:1.27

A digest can’t be moved by anyone, including the upstream publisher.

Rollback is git revert

With this setup, rolling back prod is:

git revert <promotion-commit>
git push

ArgoCD sees the values file point back at the previous SHA and syncs. Nothing is rebuilt, nothing is re-tagged, and the rollback shows up in the Git history with an author and a reason.

Answering “what’s running?”

kubectl get deploy api -n prod \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
# …/api:3f9c2e1a7b...

git log -1 3f9c2e1a7b

Two commands, one answer, and it matches what the auditor sees in the repo.

What we’d do differently

We do this for US teams.Senior nearshore engineers, US hours, AWS Partner, Vanta Partner.

Book a 30-minute architecture review
Instagram Story image: Stop deploying :latest: immutable image tags with Git SHAs

Share to your Instagram Story

  1. Share or download this 9:16 image.
  2. Add it to your story in Instagram.
  3. Add a Link sticker and paste the post link. We copy it when you share or download.
Download image