Stop Log Rot Chokepoints: Enable Socket Activation for Your High-Throughput Logger

You’re staring at a production incident where your application is timing out on write operations to a local syslog socket. The kernel log buffer is full, and systemd-journald is dropping messages because the traditional daemon process is struggling to keep up with the burst traffic from a DoS attempt. You need immediate relief without restarting the service or losing historical audit data.

The 5-minute setup

Instead of relying on a persistent daemon that buffers in memory or disk, switch your logging pipeline to use socket activation. This allows systemd to handle the initial connection and pass the file descriptor to the consumer only when data arrives, reducing context switching and allowing the kernel to manage backpressure more effectively.

First, ensure your logging consumer (e.g., a custom rsyslog instance or a specialized log forwarder) is configured to accept connections via a Unix socket. Create a socket unit file /etc/systemd/system/my-logger.socket:

[Unit]
Description=My Custom Logger Socket
Before=network.target

[Socket]
ListenStream=/run/my-logger.sock
SocketMode=0666
Accept=no
FreeBind=true

[Install]
WantedBy=sockets.target

Next, ensure your service unit /etc/systemd/system/my-logger.service references this socket. If you are using a standard daemon like rsyslogd, you might need to adjust its configuration to listen on this specific socket rather than the default /dev/log. For a custom application, ensure it calls bind() and listen() on /run/my-logger.sock.

Reload the daemon and enable the socket:

sudo systemctl daemon-reload
sudo systemctl enable --now my-logger.socket
sudo systemctl status my-logger.socket

To verify that the socket is active and the service is being started on demand, check the status. You should see Active: active (listening) for the socket. When a client writes to /run/my-logger.sock, systemd will automatically start my-logger.service if it isn’t already running, or simply pass the connection if it is.

Why it works

Traditional daemons often maintain a constant presence in memory, handling idle states inefficiently. Socket activation offloads the listening burden to the kernel, which is highly optimized for managing file descriptors. More importantly, it decouples the availability of the service from the persistence of the process. If the consumer crashes, the socket remains active. The next write attempt will trigger a restart of the service, ensuring that no logs are lost due to a transient daemon failure. This also allows for "on-demand" processing, which can be crucial for scaling out logging workers in a containerized environment where resources are strictly limited.

Pro Tip

Use systemd-analyze to verify that the socket activation is actually triggering the service correctly and to check for any latency introduced by the cold start. Run this command to see the time taken to start the service after a socket trigger:

systemd-analyze blame | grep my-logger

If you see high latency on cold starts (e.g., >200ms), consider adding KeepAlive=true to the [Socket] section of your socket unit file. This keeps the service running after the first connection, preventing the overhead of restarting the process for every single log burst, while still benefiting from the kernel-managed backpressure and crash resilience.