Stop Production Crashes: Purge Stale Docker Images and Volumes Automatically

It’s 3:00 AM, and your CI/CD pipeline is failing with no space left on device. You SSH into the build server, and df -h shows the root filesystem is 98% full. The culprit? Hundreds of dangling images and anonymous volumes from aborted builds that have accumulated over months. You manually run docker system prune, but you know this is a band-aid. You need a sustainable, automated way to reclaim space before the next deployment cycle breaks.

The 5-minute setup

Create a systemd timer and service to run a safe, non-interactive cleanup job daily. This script targets only "dangling" resources (untagged images) and unused anonymous volumes, ensuring you don't delete tagged production images or named volumes that might be in use by stopped services.

First, create the cleanup script at /usr/local/bin/docker-cleanup.sh:

#!/bin/bash
set -euo pipefail

LOG_FILE="/var/log/docker-cleanup.log"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

log "Starting Docker cleanup..."

# 1. Remove dangling images (untagged, unused)
# -a flag removes all unused images, not just dangling. 
# However, to be safer for prod, we only remove dangling by default.
# If you have a lot of old tagged images, change to 'docker image prune -a'
docker image prune -f 2>&1 | tee -a "$LOG_FILE"

# 2. Remove stopped containers that are not part of a restart policy
# This is optional and more aggressive. Uncomment if safe for your env.
# docker container prune -f 2>&1 | tee -a "$LOG_FILE"

# 3. Remove unused anonymous volumes
# This is critical for disk space leakage from build steps.
docker volume prune -f 2>&1 | tee -a "$LOG_FILE"

# 4. System-wide prune (networks, etc.)
docker system prune -f 2>&1 | tee -a "$LOG_FILE"

log "Docker cleanup complete."

Make it executable:

sudo chmod +x /usr/local/bin/docker-cleanup.sh

Create the systemd service unit at /etc/systemd/system/docker-cleanup.service:

[Unit]
Description=Daily Docker Cleanup
After=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-cleanup.sh
User=root

Create the systemd timer unit at /etc/systemd/system/docker-cleanup.timer:

[Unit]
Description=Run Docker Cleanup Daily

[Timer]
OnCalendar=daily
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target

Enable and start the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now docker-cleanup.timer
sudo systemctl status docker-cleanup.timer

Why it works

Docker’s storage driver (typically overlay2) accumulates layers for images and data for volumes. When builds fail or containers are removed without -v flags, these resources often become "dangling" or "anonymous." docker system prune is the most comprehensive tool, but running it ad-hoc is error-prone. By scheduling it via a systemd timer with RandomizedDelaySec, you avoid thundering herd problems if you have multiple hosts, and Persistent=true ensures the job runs immediately if the host was off when the scheduled time passed. This prevents the slow, silent disk fill-up that leads to emergency outages.

Pro Tip

Monitor disk usage specifically for the Docker data root (usually /var/lib/docker) to catch leaks before they hit the root partition. Add this to your monitoring stack (e.g., Prometheus node_exporter or Datadog):

# One-liner to check Docker disk usage and alert if > 80%
df -h /var/lib/docker | tail -n 1 | awk '{print $5}' | sed 's/%//' | xargs -I {} sh -c 'if [ {} -gt 80 ]; then echo "ALERT: Docker disk usage at {}%"; fi'

Additionally, if you use BuildKit, ensure you are using --output type=docker correctly in your CI pipelines to avoid creating intermediate images that never get cleaned up. Check your build logs for warnings about "cache not pruned" to identify specific stages leaking space.