What to know
- Identify the events and investigation window you need.
- Measure actual history before choosing a storage budget.
- Preserve incident evidence before retention or cleanup removes it.
Define the evidence you need
This guide targets Linux hosts using systemd-journald. Start with an incident question: can you reconstruct failed sign-ins, service restarts and unexpected errors? Our suggested retention worksheet names the services, review owner, access permissions and required investigation window. Journal entries can contain account names and application details, so give reviewers only the access they need.
Inspect history and disk use
The journalctl reference documents history and storage inspection. These commands read the available data; replace ssh.service with the actual SSH unit on your distribution:
sudo journalctl --disk-usage
sudo journalctl --list-boots
sudo journalctl -u ssh.service --since "24 hours ago" --no-pagerCheck whether earlier boots are represented and whether the expected authentication events appear. Lack of an event may mean it is recorded elsewhere, not that the event never happened. Keep the investigation timeline separate from the machine’s local display timezone.
Choose persistent storage and a budget
The journal service documentation distinguishes volatile storage under /run from persistent history under /var/log/journal. After checking your installed version and free space, a local drop-in such as /etc/systemd/journald.conf.d/60-retention.conf can contain:
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1GThese sizes are examples, not a guaranteed number of retention days. The configuration reference explains how space limits interact and why retained size can differ from the configured target. Estimate event volume and choose values appropriate to the host.
Apply and verify carefully
Save the previous configuration. During the agreed maintenance window, restart the journal service using the documented restart operation rather than separately stopping and starting it, then flush to persistent storage. Check service health, new records and disk use afterward. Do not reboot a production machine solely to test this guide; inspect previous-boot availability after the next planned reboot.
Keep retention distinct from incident preservation
Space limits eventually remove old history. During an investigation, preserve relevant evidence to protected storage before any cleanup. Do not run vacuum commands as a reflex when disk space rises. Decide separately whether you need a remote collector, because local logs alone may not survive host loss or administrator compromise. Pair this routine with the restore exercise and SSH log-consumer checks. Record that collection, review and recovery all work.
Sources & further reading
Primary sources supporting this article. The practical recommendations are Kiava’s general guidance and should be adapted to your project.
