Stop Infinite Boot Loops: Detecting Circular Dependencies in Systemd

You reboot a production server to apply a kernel update, but it never comes back up. The console spits out a rapid-fire stream of "Job app.service/start timed out" errors before the system eventually drops into emergency mode. You SSH in, and systemctl status app.service shows it as activating (auto-restart), but the logs reveal it’s waiting for a database service that, in turn, is waiting for the app to signal readiness. You have a circular dependency.

The 5-minute setup

Systemd does not have a built-in systemctl check-cycles command. To find these issues before they break your next reboot, use the systemd-analyze tool combined with grep to map out the dependency graph.

First, ensure you are looking at the actual runtime state, not just the unit files on disk. Run the following command to list all units that currently have a dependency cycle. Note that this command is available in modern systemd versions (v240+).

# List all units involved in circular dependencies
systemd-analyze verify --recursive-errors=true /etc/systemd/system/ 2>&1 | grep -i "circular\|dependency"

If that returns nothing, the issue might be dynamic or runtime-specific. A more reliable method to spot potential cycles during development or maintenance is to trace the After= and Requires= directives manually for critical services. However, the most robust "gotcha" detector is to check for units that are in a waiting state for longer than expected during boot.

Here is a script to identify units stuck in a "waiting for dependencies" state for more than 30 seconds during a boot sequence:

#!/bin/bash
# Find units that are stuck waiting for dependencies

# Get all units currently in 'activating' state
STUCK_UNITS=$(systemctl list-units --type=service --state=activating --no-legend --no-pager | awk '{print $1}')

if [ -z "$STUCK_UNITS" ]; then
    echo "No units currently stuck in activating state."
    exit 0
fi

echo "Checking dependencies for stuck units:"
for UNIT in $STUCK_UNITS; do
    # Get the dependencies of the stuck unit
    DEPS=$(systemctl show "$UNIT" --property=After --value | tr ' ' '\n' | sort -u)
    
    # Check if any of the dependencies are also in an activating/failed state
    for DEP in $DEPS; do
        DEP_STATE=$(systemctl is-active "$DEP" 2>/dev/null)
        if [[ "$DEP_STATE" == "activating" || "$DEP_STATE" == "failed" ]]; then
            echo "CIRCULAR RISK: $UNIT is waiting for $DEP (state: $DEP_STATE)"
        fi
    done
done

To fix a confirmed cycle, you must break the link. Usually, one side of the dependency is soft (i.e., the service doesn't strictly need the other to be fully up, just to start). Replace Requires= with Wants= or After= without Requires.

Example fix in /etc/systemd/system/app.service:

[Service]
# Change this:
# Requires=db.service
# After=db.service

# To this:
Wants=db.service
After=db.service

Then run:

systemctl daemon-reload
systemctl restart app.service

Why it works

Systemd resolves dependencies by building a Directed Acyclic Graph (DAG). If a cycle exists, the DAG is no longer acyclic, and the scheduler cannot determine a valid start order. By replacing Requires= (which implies a hard dependency and failure if the target fails) with Wants= (a soft dependency), you allow the service to start even if its dependency isn't ready yet. The After= directive ensures the order of starting is correct, but Wants= prevents the hard lock that causes the timeout.

Pro Tip

Use systemd-analyze critical-chain to see how long each service took to start. If you see a service taking 2 minutes to start, but its dependencies only took 5 seconds, it’s likely blocked waiting for a network resource or a circular dependency that didn’t fully deadlock but caused a timeout. Run journalctl -u <service> -b --no-pager | grep -i "waiting\|timeout" to confirm.