Systemd Timers That Actually Fire — Stop Discovering Your Scheduled Tasks Have Been Silent for Weeks

The Problem

You set up a systemd timer to run your log cleanup job daily at 2 AM. Three weeks later, df -h shows the log partition at 94% — the cleanup job has been running with Persistent=true set to false (the default), and every time the server was in a brief maintenance window, a reboot, or the disk was briefly full, the timer simply skipped that run and moved on. There's no alert. There's no "missed execution" flag. The timer's OnCalendar=-- 02:00:00 fired, systemd tried to start the service, the service's ExecStartPost dependency (the log partition) wasn't ready, the service failed, and systemd marked the timer's last trigger as "done" — because from the timer's perspective, it did* trigger the service. The service failing is the service's problem, not the timer's.

This is one of the most common and most damaging patterns in production environments. Here's what actually goes wrong:

1. **Persistent=true is the default, but it only works if the system was powered on when the timer was supposed to fire.** If the server was in a planned maintenance window, rebooting, or (worse) the disk was full and the service failed to start, the timer's "persistent" flag does nothing — it only catches up if the system was off (powered down) and then came back. A brief reboot during a kernel update, or a systemctl stop for maintenance, can cause the timer to miss its window silently. The next run is 24 hours later.
2. OnCalendar vs OnBootSec/OnUnitActiveSec confusion. OnCalendar=-- 02:00:00 fires at 2 AM wall clock — but if the system was rebooted at 1:50 AM and the timer service takes 20 seconds to start, you get a run at 2:00:20 AM. That's fine. But if the system was rebooted at 2:05 AM, the next OnCalendar trigger is tomorrow at 2 AM. A 5-minute reboot gap becomes a 24-hour gap. OnUnitActiveSec=24h would have fired 24 hours after the last successful* activation — but only if the service was ever activated.
3. No monitoring of timer status. systemctl list-timers shows Last Trigger and Next Elapsed — but nobody is checking Last Trigger against the expected schedule. If the timer missed 3 runs in a row because the service kept failing, Last Trigger still shows a timestamp from 3 days ago. There's no "missed" counter. There's no alert. The timer looks healthy because it is triggering — the service is what's broken, and that's a different monitoring surface.
4. AccuracySec=1us is the default, but nobody sets it. The default AccuracySec is 1us (effectively immediate), which is fine for most jobs. But if you have multiple timers all targeting the same wall-clock time (e.g., backup at 2 AM, cleanup at 2 AM, report at 2 AM), they all fire within microseconds of each other, creating a thundering herd of I/O at 2 AM. The default is fine — but if you want to stagger them, you need to explicitly set AccuracySec=5min or similar, and nobody does.
5. ConditionPathExists / ConditionDirectoryNotEmpty guards silently suppress runs. If your cleanup service has ConditionPathExists=/var/log/myapp/.log, and the log directory is empty (because the app hasn't written any logs yet, or the logs were already cleaned up), the service exits 0 without doing anything. The timer shows "triggered," the service shows "succeeded," and the log partition fills up because the app is now writing to a different* directory that the condition doesn't cover. The condition guard is a silent no-op.

The Solution

**Step 1: Use OnUnitActiveSec for recurring jobs — it fires relative to the last successful activation, not wall clock**

# /etc/systemd/system/log-cleanup.timer
# Instead of: OnCalendar=*-*-* 02:00:00
# Use: OnUnitActiveSec=24h
# This means: "run 24 hours after the last successful run of log-cleanup.service"
# If the service fails, the timer will retry on the next activation cycle
# (governed by OnBootSec as the initial anchor)

[Unit]
Description=Run log cleanup every 24 hours

[Timer]
# Anchor to boot time (initial run happens 5 min after boot)
OnBootSec=5min
# Subsequent runs: 24h after the last activation
OnUnitActiveSec=24h
# Fallback: if the unit was never activated, fire at 2 AM wall clock
# (this gives you a known time-of-day anchor for the first run)
OnCalendar=*-*-* 02:00:00
# Allow up to 5 minutes of scheduling jitter (stagger with other timers)
AccuracySec=5min
# Catch up if the system was off (e.g., maintenance window, reboot)
Persistent=true
# Randomized delay of up to 30s to avoid thundering herd with other timers
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

**Step 2: Add a ConditionPathExists guard that actually matches your real log paths — and verify it**

# /etc/systemd/system/log-cleanup.service
[Unit]
Description=Clean up application logs older than 7 days
# ── CRITICAL: This condition must match your ACTUAL log path ──
# If your app writes to /var/log/myapp/*.log, NOT /var/log/myapp/
ConditionPathExists=/var/log/myapp

[Service]
Type=oneshot
# Run the cleanup (adjust paths to match your actual setup)
ExecStart=/usr/local/bin/log-cleanup.sh
# If the condition fails, the service exits 0 — make that VISIBLE
# by logging a message when the condition is not met
ExecStartPre=/usr/local/bin/log-cleanup-condition-check.sh

#!/bin/bash
# /usr/local/bin/log-cleanup-condition-check.sh
# Runs before log-cleanup.sh — logs a message if the log dir is empty
LOG_DIR="/var/log/myapp"
if [ ! -d "$LOG_DIR" ] || [ -z "$(ls -A "$LOG_DIR" 2>/dev/null)" ]; then
    logger -t log-cleanup "WARNING: $LOG_DIR is empty or missing — skipping cleanup"
    exit 0
fi
exit 0

**Step 3: Create a monitoring script that detects missed timer executions**

#!/bin/bash
# /usr/local/bin/check-timer-health.sh — run via a separate systemd timer (hourly)
# Detects if a systemd timer has missed its expected runs.

set -euo pipefail
TIMER_NAME="log-cleanup.timer"
EXPECTED_INTERVAL_MINUTES=1440  # 24 hours
ALERT_THRESHOLD=60              # Alert if last run was >60 min past expected
LOG="/var/log/timer-health.log"

log() { echo "[$(date -Is)] $*" | tee -a "$LOG"; }

# ── Get the last trigger time from systemctl ──
LAST_TRIGGER=$(systemctl show "$TIMER_NAME" --property=LastTriggerUSec \
    --value 2>/dev/null)

if [ -z "$LAST_TRIGGER" ] || [ "$LAST_TRIGGER" = "n/a" ] || [ "$LAST_TRIGGER" = "0" ]; then
    log "ALERT: $TIMER_NAME has never been triggered (LastTriggerUSec is unset)"
    exit 1
fi

# LastTriggerUSec is in microseconds since epoch
LAST_TRIGGER_EPOCH=$(( LAST_TRIGGER / 1000000 ))
NOW_EPOCH=$(date +%s)
SECONDS_SINCE=$(( NOW_EPOCH - LAST_TRIGGER_EPOCH ))
MINUTES_SINCE=$(( SECONDS_SINCE / 60 ))

# ── Expected: last trigger should be within (interval + alert_threshold) minutes ──
EXPECTED_MAX=$(( EXPECTED_INTERVAL_MINUTES + ALERT_THRESHOLD ))

if [ "$MINUTES_SINCE" -gt "$EXPECTED_MAX" ]; then
    log "ALERT: $TIMER_NAME last triggered ${MINUTES_SINCE} min ago (expected: every ~${EXPECTED_INTERVAL_MINUTES} min, threshold: ${EXPECTED_MAX} min)"
    # Also check if the service is in a failed state
    SERVICE_STATE=$(systemctl show "log-cleanup.service" --property=ActiveState --value)
    if [ "$SERVICE_STATE" = "failed" ]; then
        log "  → Service is FAILED: $(systemctl show 'log-cleanup.service' --property=Result --value)"
    fi
    exit 1
fi

log "OK: $TIMER_NAME last triggered ${MINUTES_SINCE} min ago (healthy)"
exit 0

Step 4: Wire the health check to a separate timer with alerting

# /etc/systemd/system/timer-health.timer
[Unit]
Description=Check systemd timer health every hour

[Timer]
OnCalendar=hourly
AccuracySec=1min
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/timer-health.service
[Unit]
Description=Check systemd timer health
Requires=timer-health.timer

[Service]
Type=oneshot
ExecStart=/usr/local/bin/check-timer-health.sh
# Alert on failure (integrate with your alerting: PagerDuty, Slack, etc.)
# If check-timer-health.sh exits non-zero, this service fails → alert

Step 5: Stagger multiple timers to avoid the 2 AM thundering herd

# If you have multiple daily jobs at 2 AM, stagger them:

# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=*-*-* 02:00:00
AccuracySec=5min
RandomizedDelaySec=60

# /etc/systemd/system/log-cleanup.timer
[Timer]
OnCalendar=*-*-* 02:05:00
AccuracySec=5min
RandomizedDelaySec=60

# /etc/systemd/system/report-generation.timer
[Timer]
OnCalendar=*-*-* 02:10:00
AccuracySec=5min
RandomizedDelaySec=60

# Now your jobs run at ~2:00, ~2:05, ~2:10 (±5 min jitter + up to 60s random delay)
# Instead of all three hammering the disk at 02:00:00.000

Step 6: Enable LogExtraFields so every timer run is traceable in journald

# Add to your [Service] section:
[Service]
Type=oneshot
ExecStart=/usr/local/bin/log-cleanup.sh
# Every log line from this service gets a structured field:
LogExtraFields=TIMER_NAME=log-cleanup
LogExtraFields=JOB_ID=24h-cleanup
# Now you can query: journalctl -f _SYSTEMD_UNIT=log-cleanup.service TIMER_NAME=log-cleanup

Why This Works

Each piece addresses a specific failure mode:

• OnUnitActiveSec + OnBootSec + OnCalendar (all three) gives you a belt-and-suspenders schedule. OnBootSec anchors the first run to boot time (so you always get a run within 5 minutes of boot). OnUnitActiveSec ensures recurring runs are relative to the last activation (not wall clock), so a failed run doesn't silently skip the next one. OnCalendar provides a wall-clock fallback in case the unit was never activated (e.g., freshly installed timer). Persistent=true catches up if the system was powered off.
• The health check script is the single most important addition. systemctl list-timers shows you the timer's perspective — it triggered, it's happy. But the service may have been failing for weeks. The health check looks at LastTriggerUSec and compares it against the expected interval, and also checks the service's ActiveState to catch the "timer triggered but service failed" case. This is the monitoring that catches the silent no-op.
• ConditionPathExists with a real path check — the ExecStartPre script logs a warning when the condition is about to suppress the run. Without this, an empty log directory means the service "succeeds" without doing anything, and you never know. The logger call makes it visible in journald.
• Staggering with RandomizedDelaySec prevents the 2 AM thundering herd. When 5 timers all fire at 02:00:00.000, you get a burst of I/O that can saturate the disk and slow down all the jobs (and anything else running). A 5-minute AccuracySec + up to 60 seconds of RandomizedDelaySec spreads the load.
• LogExtraFields makes every timer run traceable in journald. When something goes wrong at 3 AM, journalctl -f TIMER_NAME=log-cleanup shows you exactly what the cleanup job logged, without having to grep through a wall of other log lines.

Pro Tip

The "timer health" one-liner — add this to your monitoring:

# ── One command: check all timers, flag any that are stale ──
systemctl list-timers --all --no-pager --output=full | \
  awk 'NR>2 && $1 != "-" {
      split($1, t, "-");
      timer = t[1] "-" t[2] "-" t[3] "-" t[4];
      next;
  } {
      # Parse "Last Trigger" column (col 2) and "Next Elapsed" (col 3)
      if ($2 != "-" && $2 !~ /ago$/) {
          # Extract minutes from "X min ago" or "X h Y min ago"
          if (match($0, /([0-9]+) min ago/, m)) mins = m[1];
          else if (match($0, /([0-9]+) h/, m)) mins = m[1]*60;
          else mins = 0;
          if (mins > 1500) printf "STALE: %-40s last triggered %d min ago\n", $1, mins;
      }
  }' | \
  while read line; do
      echo "⚠️  $line"
      logger -t timer-health "$line"
  done || true

# ── Simpler version: just flag timers whose Next Elapsed is in the past ──
systemctl list-timers --no-pager | awk '
  NR>2 && $3 ~ /^-/ {
      print "⚠️  TIMER OVERDUE: " $1 " (next was " $3 " ago)"
  }'

The golden timer rule: A timer that has never been verified is a timer you hope is working. systemctl list-timers shows you the timer's intentions, not the service's results. The health check is the verification. Everything else is just scheduling.

"The first time you discover your scheduled task has been silent for three weeks is the first time you find out it was never really scheduled. The health check moves that discovery from the outage to the dashboard. You'd rather get a 2 AM alert about a missed timer than a 9 AM alert about a full disk."