Stop Cron Jobs From Stomping on Each Other with `flock`

You schedule a nightly rsync backup at 2 AM. One Tuesday, the source volume is heavily fragmented and the transfer takes 4 hours instead of 1. At 3 AM, the next scheduled invocation fires, opens the same destination directory, and begins writing partial chunks into files the first process is still streaming. By morning you have a corrupted backup that passes ls -la but fails every restore test. This is one of the most common silent failures in cron-based automation — the job doesn't crash, it just quietly produces garbage.

The 5-minute setup

Wrap any cron entry with flock using a lock file in /var/lock:

# Crontab entry — the -n flag means "fail fast, don't wait"
0 2 * * * /usr/bin/flock -n /var/lock/nightly-backup.lock /usr/local/bin/nightly-backup.sh

Inside /usr/local/bin/nightly-backup.sh, the script runs normally. If the lock is already held, flock -n exits immediately with code 1 and the second invocation never starts. The cron line itself is the entire fix — no changes to the script logic required.

For systemd timers (if you've migrated), apply the same pattern in the ExecStart line:

# /etc/systemd/system/nightly-backup.service
[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /var/lock/nightly-backup.lock /usr/local/bin/nightly-backup.sh

If you'd rather have the second invocation wait instead of skipping (useful for jobs that must all run but shouldn't overlap), drop the -n and add a timeout:

# Wait up to 10 minutes for the lock, then give up
*/15 * * * * /usr/bin/flock -w 600 /var/lock/collect-metrics.lock /usr/local/bin/collect-metrics.sh

Why it works

flock uses the kernel's flock(2) syscall, which is advisory but atomic. The lock is tied to the file descriptor, not the PID, so it's automatically released if the process is killed by OOM, SIGKILL, or a container restart — no stale lock files to clean up. The -n flag makes the call non-blocking: if another process already holds an exclusive lock on that file, flock exits 1 immediately rather than sleeping. This gives you a zero-cost guardrail that costs nothing when the job finishes on time and saves you from a disaster when it doesn't.

Pro Tip

Add a MAILTO or logging wrapper so you actually know when a skip happens — a silently skipped backup is still a missed backup:

0 2 * * * /usr/bin/flock -n /var/lock/nightly-backup.lock /usr/local/bin/nightly-backup.sh || echo "$(date -Iseconds) SKIPPED: backup still running" >> /var/log/backup-skips.log

Then alert on that log with a one-liner in your monitoring stack (Prometheus node_exporter textfile collector, or a simple grep in a health check):

# Quick check from any monitoring agent
grep -c "SKIPPED" /var/log/backup-skips.log

A non-zero count in the last 24 hours means your job duration is creeping past its interval — time to either move the schedule or split the work.