Stop Your Server from Crashing Because /var/log/journal Is Full

You’re on call for a critical production node when the disk usage alarm fires. You SSH in and find /var/log/journal has grown to 50GB, pushing the root filesystem to 100% capacity. The system is unstable, services are failing to start, and you’re frantically trying to delete logs while the system is already struggling to write to disk. This happens because systemd-journald defaults to using up to 4GB of disk space (or 10% of the filesystem) if not explicitly capped, and on high-traffic servers, this limit is easily exceeded during log spikes.

The 5-minute setup

First, check your current journal disk usage and configuration. Run these commands to see how much space the journal is currently consuming and what the current limits are:

# Check current journal disk usage
journalctl --disk-usage

# Check the current configuration
cat /etc/systemd/journald.conf

You will likely see that SystemMaxUse= or SystemMaxFileSize= is either commented out or set to a default value. To permanently fix this, edit the journal configuration file:

sudo nano /etc/systemd/journald.conf

Under the [Journal] section, uncomment or add the following lines. Set SystemMaxUse= to a reasonable cap (e.g., 1GB) and SystemMaxFileSize= to limit individual journal files:

[Journal]
# Limit the total size of journal files on disk to 1GB
SystemMaxUse=1G
# Limit each individual journal file to 100MB
SystemMaxFileSize=100M
# Retain logs for 7 days
MaxRetentionSec=7day

After saving the file, you need to restart systemd-journald to apply the changes. Be aware that restarting the journal service will cause a brief interruption in log collection, but it is necessary for the new limits to take effect:

sudo systemctl restart systemd-journald

Verify that the new limits are active by checking the disk usage again. You should see that the journal has been automatically vacuumed down to the new limit:

journalctl --disk-usage

If the disk usage is still high, you can manually force a vacuum to clean up old logs immediately:

sudo journalctl --vacuum-size=1G

Why it works

systemd-journald is responsible for collecting and storing system logs. By default, it has a built-in logic to prevent the journal from consuming too much disk space, but the default limits (4GB or 10% of the filesystem) can be too generous for servers with smaller root partitions or high log volumes. Setting SystemMaxUse= explicitly tells the journal daemon to enforce a hard cap on the total disk space used. When the limit is exceeded, systemd-journald automatically deletes the oldest journal files to bring the usage back under the cap. This prevents the root filesystem from filling up due to uncontrolled log growth, which is a common cause of system instability and failed service startups.

Pro Tip

If you are running multiple servers and want to standardize this configuration, create a drop-in configuration file instead of editing the main journald.conf file. This makes it easier to manage with configuration management tools like Ansible or Puppet. Create a file in /etc/systemd/journald.conf.d/ with a .conf extension:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo nano /etc/systemd/journald.conf.d/limit.conf

Add the following content to /etc/systemd/journald.conf.d/limit.conf:

[Journal]
SystemMaxUse=1G
SystemMaxFileSize=100M
MaxRetentionSec=7day

Then restart systemd-journald as before. This approach is cleaner because it doesn’t modify the main configuration file, making it easier to revert or update. Additionally, you can monitor the journal disk usage with a simple cron job or systemd timer to alert you if it approaches the limit:

# Add this to your crontab to check journal usage hourly
0 * * * * journalctl --disk-usage | grep -q "1.0G" && echo "Journal disk usage is high" | mail -s "Journal Alert" you@example.com