Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Why is my systemd journal not persistent across boots?

My systemd journal, as shown by journalctl on Fedora 21, is not persistent. The logs only go back to the previous system boot, and not before.

I have set the following options /etc/systemd/journald.conf

Storage=persistent
SystemMaxUse=250M

Despite this, journald's logs always start at the previous boot time:

$ journalctl --until=yesterday
-- Logs begin at Thu 2015-03-19 09:14:50 GMT, end at Thu 2015-03-19 09:30:02 GMT. --
$journalctl | head
-- Logs begin at Thu 2015-03-19 09:14:50 GMT, end at Thu 2015-03-19 09:30:02 GMT. --
Mar 19 09:14:50 www systemd-journal[1343]: Runtime journal is using 8.0M (max allowed 99.6M, trying to leave 149.5M free of 988.8M available → current limit 99.6M).
Mar 19 09:14:50 www systemd-journal[1343]: Runtime journal is using 8.0M (max allowed 99.6M, trying to leave 149.5M free of 988.8M available → current limit 99.6M).
Mar 19 09:14:50 www kernel: PAT configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC  
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpuset
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpu
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpuacct
Mar 19 09:14:50 www kernel: Linux version 3.19.1-x86_64-linode53 (maker@build.linode.com) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Tue Mar 10 15:30:28 EDT 2015
Mar 19 09:14:50 www kernel: Command line: root=/dev/xvda xencons=tty console=tty1 console=hvc0 nosep nodevfs ramdisk_size=32768 ip_conntrack.hashsize=8192 nf_conntrack.hashsize=8192 ro  devtmpfs.mount=1 
Mar 19 09:14:50 www kernel: ACPI in unprivileged domain disabled

Interestingly, the first two log entries talk about the “Runtime journal”, which is not persistent across boots. When I restart systemd-journald, it finally mentions a “Permanent Journal”:

$ sudo systemctl restart systemd-journald
$ journalctl | tail
Mar 19 09:36:23 www systemd[1]: Stopping Journal Service...
Mar 19 09:36:23 www systemd-journal[1343]: Journal stopped
Mar 19 09:36:23 www systemd-journal[2026]: Permanent journal is using 24.0M (max allowed 250.0M, trying to leave 1.9G free of 11.8G available → current limit 250.0M).
Mar 19 09:36:23 www systemd-journald[1343]: Received SIGTERM from PID 1 (systemd).
Mar 19 09:36:23 www systemd[1]: Starting Journal Service...
Mar 19 09:36:23 www systemd-journal[2026]: Journal started
Mar 19 09:36:23 www systemd[1]: Started Journal Service.
Mar 19 09:36:23 www sudo[2021]: pam_unix(sudo:session): session closed for user root
Mar 19 09:36:23 www systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 19 09:36:23 www systemd[1]: Started Flush Journal to Persistent Storage.

However, when I reboot the system nothing persists, and the journal start again at boot time. It's really driving me nuts. When I find the cause of this problem I intend to file a bug report (unless it's my own stupidity). Why aren't my logs persistent?

Why is my systemd journal not persistent across boots?

My systemd journal, as shown by journalctl on Fedora 21, is not persistent. The logs only go back to the previous system boot, and not before.

I have set the following options /etc/systemd/journald.conf

Storage=persistent
SystemMaxUse=250M

Despite this, journald's logs always start at the previous boot time:

$ journalctl --until=yesterday
-- Logs begin at Thu 2015-03-19 09:14:50 GMT, end at Thu 2015-03-19 09:30:02 GMT. --
$journalctl | head
-- Logs begin at Thu 2015-03-19 09:14:50 GMT, end at Thu 2015-03-19 09:30:02 GMT. --
Mar 19 09:14:50 www systemd-journal[1343]: Runtime journal is using 8.0M (max allowed 99.6M, trying to leave 149.5M free of 988.8M available → current limit 99.6M).
Mar 19 09:14:50 www systemd-journal[1343]: Runtime journal is using 8.0M (max allowed 99.6M, trying to leave 149.5M free of 988.8M available → current limit 99.6M).
Mar 19 09:14:50 www kernel: PAT configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC  
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpuset
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpu
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpuacct
Mar 19 09:14:50 www kernel: Linux version 3.19.1-x86_64-linode53 (maker@build.linode.com) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Tue Mar 10 15:30:28 EDT 2015
Mar 19 09:14:50 www kernel: Command line: root=/dev/xvda xencons=tty console=tty1 console=hvc0 nosep nodevfs ramdisk_size=32768 ip_conntrack.hashsize=8192 nf_conntrack.hashsize=8192 ro  devtmpfs.mount=1 
Mar 19 09:14:50 www kernel: ACPI in unprivileged domain disabled

Interestingly, the first two log entries talk about the “Runtime journal”, which is not persistent across boots. When I restart systemd-journald, it finally mentions a “Permanent Journal”:

$ sudo systemctl restart systemd-journald
$ journalctl | tail
Mar 19 09:36:23 www systemd[1]: Stopping Journal Service...
Mar 19 09:36:23 www systemd-journal[1343]: Journal stopped
Mar 19 09:36:23 www systemd-journal[2026]: Permanent journal is using 24.0M (max allowed 250.0M, trying to leave 1.9G free of 11.8G available → current limit 250.0M).
Mar 19 09:36:23 www systemd-journald[1343]: Received SIGTERM from PID 1 (systemd).
Mar 19 09:36:23 www systemd[1]: Starting Journal Service...
Mar 19 09:36:23 www systemd-journal[2026]: Journal started
Mar 19 09:36:23 www systemd[1]: Started Journal Service.
Mar 19 09:36:23 www sudo[2021]: pam_unix(sudo:session): session closed for user root
Mar 19 09:36:23 www systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 19 09:36:23 www systemd[1]: Started Flush Journal to Persistent Storage.

However, when I reboot the system nothing persists, and the journal start starts again at boot time. It's really driving me nuts. When I find the cause of this problem I intend to file a bug report (unless it's my own stupidity). Why aren't my logs persistent?