Stop Kubelet Eviction Loops with Aggressive Image Garbage Collection

Your production cluster is stable until a large CI/CD pipeline pushes a 4GB container image to a node with 50GB of disk space. Suddenly, the node reports DiskPressure, and Kubelet begins evicting healthy pods to free up space, causing a cascading service outage. The root cause isn't the new image size, but the accumulation of unused images from previous deployments that Kubelet hasn't cleaned up because the default garbage collection thresholds are too conservative.

The 5-minute setup

By default, Kubelet only triggers image garbage collection when the image filesystem usage exceeds 80% of the available capacity. This is often too late to prevent DiskPressure on smaller nodes or those with high I/O latency. You can tighten this threshold to 60% to proactively clean up unused images before they impact node health.

1. Check current Kubelet configuration:

    # On a worker node
    sudo cat /var/lib/kubelet/config.yaml | grep -A 5 imageGCHighThresholdPercent

1. Update the Kubelet configuration:

Create or edit the Kubelet configuration file (usually /var/lib/kubelet/config.yaml or /etc/systemd/system/kubelet.service.d/10-kubeadm.conf depending on your setup). Add or modify the following keys:

    imageGCHighThresholdPercent: 60
    imageGCLowThresholdPercent: 40
    imageMaximumGCAge: 24h

1. Restart Kubelet to apply changes:

    sudo systemctl daemon-reload
    sudo systemctl restart kubelet

1. Verify the change:

    kubectl get nodes -o jsonpath='{.items[*].status.conditions[?(@.type=="DiskPressure")].status}'
    # Should show "False" for healthy nodes

Why it works

Kubelet maintains a local image store that can grow indefinitely if images are not explicitly removed. The imageGCHighThresholdPercent triggers garbage collection when the image filesystem usage exceeds the specified percentage of the node's disk capacity. By lowering this threshold from the default 80% to 60%, Kubelet starts removing unused images earlier, ensuring that there is always a buffer of free space available for new image pulls and temporary storage. The imageGCLowThresholdPercent defines the target usage level after garbage collection, ensuring that enough space is freed to prevent immediate re-triggering of the GC loop.

Pro Tip

Monitor image disk usage proactively with Prometheus. Add the following metric to your Grafana dashboard or alerting rules to catch potential issues before they cause DiskPressure:

# Alert if any node's image filesystem usage exceeds 70%
(
  node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} -
  node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
) /
node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} * 100 > 70

Also, consider using crictl images on worker nodes to manually inspect and remove old images if you need to free up space immediately without waiting for Kubelet's GC cycle:

# List images sorted by size (largest first)
sudo crictl images --digests | awk '{print $1, $4}' | sort -k2 -h -r | head -10

# Remove a specific unused image
sudo crictl rmi <IMAGE_ID>