Jobs

A Deployment runs forever. A Job runs until it finishes successfully, and then stops — a database migration, a batch import, a certificate renewal, a one-off script in the cluster.

Reference

• https://kubernetes.io/docs/concepts/workloads/controllers/job/

Terminology

• Job — Runs Pods to completion and counts successes. It is done when the required number of Pods have exited 0.
• restartPolicy — Never or OnFailure for a Job. Always is rejected: a container that never exits can never complete a Job.
• completions — How many successful Pod exits are needed. Defaults to 1.
• parallelism — How many Pods may run at once. Defaults to 1. Setting parallelism: 5, completions: 20 means “twenty units of work, five at a time”.
• backoffLimit — How many times a Pod may fail before the Job is marked failed. Defaults to 6, with exponential backoff between attempts.
• activeDeadlineSeconds — A hard wall-clock limit for the whole Job. When it expires, running Pods are terminated and the Job fails.
• completionMode: Indexed — Each Pod gets a JOB_COMPLETION_INDEX in its environment, so work can be partitioned deterministically.

The Shape of It

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
spec:
  backoffLimit: 3
  ttlSecondsAfterFinished: 600
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: docker.io/library/busybox
          command: ["sh", "-c", "echo migrating; sleep 5; echo done"]

ttlSecondsAfterFinished is worth adopting as a habit: without it, completed Jobs and their Pods accumulate forever, and a cluster full of dead Pods is harder to read.

Exercises

1. A Job that succeeds.

   kubectl create job hello --image=docker.io/library/busybox -- sh -c "echo working; sleep 3; echo done"
   kubectl get jobs,pods
   kubectl logs job/hello

   The Pod sticks around showing Completed — that is a finished Job, not a broken one.

2. A Job that fails and retries. Run something that exits non-zero and watch the backoff:

   kubectl create job failing --image=docker.io/library/busybox -- sh -c "echo attempt; exit 1"
   kubectl get pods -w        # attempts appear, spaced further apart
   kubectl describe job failing

   After backoffLimit attempts the Job is marked Failed and stops. kubectl describe job shows the sequence of failures and their reasons — this is usually where the answer is.

3. Parallelism. Run a Job with completions: 6 and parallelism: 2 and watch how the controller keeps two Pods running until six have succeeded. Then set parallelism: 6 and compare the wall-clock time.

4. Indexed work. Add completionMode: Indexed and have the container print $JOB_COMPLETION_INDEX. Each Pod now knows which shard of the work it owns, which is how you avoid having five workers fight over the same records.

Gotchas Worth Knowing

• A Job’s Pod template is immutable. Change the command and you must delete the Job (and, if you want a clean slate, its Pods) before recreating it.
• A Job that never completes is usually a container that never exits. If your command starts a background daemon, the Job waits forever. activeDeadlineSeconds is the safety net.
• restartPolicy: Always is rejected, and the error message when you forget is not obvious the first time.
• Deleting a Job deletes its Pods by default, along with their logs. If the output matters, ship it somewhere before cleaning up.

What to Take Away

• A Job is the run-to-completion counterpart to a Deployment, and the two differ mainly in what “done” means.
• completions, parallelism and backoffLimit are the three knobs that define the shape of the work.
• Set ttlSecondsAfterFinished so finished work does not pile up in the cluster.