Ask Your Question
1

Why is my systemd journal not persistent across boots?

asked 2015-03-19 04:51:16 -0500

jjv gravatar image

updated 2015-03-19 04:56:05 -0500

My systemd journal, as shown by journalctl on Fedora 21, is not persistent. The logs only go back to the previous system boot, and not before.

I have set the following options /etc/systemd/journald.conf

Storage=persistent
SystemMaxUse=250M

Despite this, journald's logs always start at the previous boot time:

$ journalctl --until=yesterday
-- Logs begin at Thu 2015-03-19 09:14:50 GMT, end at Thu 2015-03-19 09:30:02 GMT. --
$journalctl | head
-- Logs begin at Thu 2015-03-19 09:14:50 GMT, end at Thu 2015-03-19 09:30:02 GMT. --
Mar 19 09:14:50 www systemd-journal[1343]: Runtime journal is using 8.0M (max allowed 99.6M, trying to leave 149.5M free of 988.8M available → current limit 99.6M).
Mar 19 09:14:50 www systemd-journal[1343]: Runtime journal is using 8.0M (max allowed 99.6M, trying to leave 149.5M free of 988.8M available → current limit 99.6M).
Mar 19 09:14:50 www kernel: PAT configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC  
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpuset
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpu
Mar 19 09:14:50 www kernel: Initializing cgroup subsys cpuacct
Mar 19 09:14:50 www kernel: Linux version 3.19.1-x86_64-linode53 (maker@build.linode.com) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Tue Mar 10 15:30:28 EDT 2015
Mar 19 09:14:50 www kernel: Command line: root=/dev/xvda xencons=tty console=tty1 console=hvc0 nosep nodevfs ramdisk_size=32768 ip_conntrack.hashsize=8192 nf_conntrack.hashsize=8192 ro  devtmpfs.mount=1 
Mar 19 09:14:50 www kernel: ACPI in unprivileged domain disabled

Interestingly, the first two log entries talk about the “Runtime journal”, which is not persistent across boots. When I restart systemd-journald, it finally mentions a “Permanent Journal”:

$ sudo systemctl restart systemd-journald
$ journalctl | tail
Mar 19 09:36:23 www systemd[1]: Stopping Journal Service...
Mar 19 09:36:23 www systemd-journal[1343]: Journal stopped
Mar 19 09:36:23 www systemd-journal[2026]: Permanent journal is using 24.0M (max allowed 250.0M, trying to leave 1.9G free of 11.8G available → current limit 250.0M).
Mar 19 09:36:23 www systemd-journald[1343]: Received SIGTERM from PID 1 (systemd).
Mar 19 09:36:23 www systemd[1]: Starting Journal Service...
Mar 19 09:36:23 www systemd-journal[2026]: Journal started
Mar 19 09:36:23 www systemd[1]: Started Journal Service.
Mar 19 09:36:23 www sudo[2021]: pam_unix(sudo:session): session closed for user root
Mar 19 09:36:23 www systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 19 09:36:23 www systemd[1]: Started Flush Journal to Persistent Storage.

However, when I reboot the system nothing persists, and the journal starts again at boot time. It's really driving me nuts. When I find the cause of this problem I intend to file a bug report (unless it ... (more)

edit retag flag offensive close merge delete

Comments

To the administrators: I received the following error when posting this question: "Error 503 Backend fetch failed, Guru Meditation: XID: 2218786, Varnish cache server"

jjv gravatar imagejjv ( 2015-03-19 04:54:45 -0500 )edit

Does /var/log/journal exist?

randomuser gravatar imagerandomuser ( 2015-03-19 08:53:47 -0500 )edit

3 Answers

Sort by » oldest newest most voted
3

answered 2015-03-20 01:54:10 -0500

jjv gravatar image

updated 2015-03-21 04:52:02 -0500

I have found the cause of the problem! The reason is that the machine identifier in /etc/machine-id changes at every reboot. This starts a new logging directory under /var/log/journal. Old logs can be viewed with the following command:

journalctl --merge

UPDATE -- Here's the reason my machine-id kept changing on reboots:

My /etc/machine-id was volatile because it was mounted as a temporary filesystem (tmpfs), since Linode (correctly) zeroed out the contents of the real /etc/machine-id. I fixed the problem by booting into rescue mode and writing a definitive identifier to /etc/machine-id . I have more thoroughly described this Linode-specific problem and fix here: http://unix.stackexchange.com/a/191373/18718

edit flag offensive delete link more
1

answered 2017-11-16 05:38:42 -0500

Aleksandar Kostadinov gravatar image

In recent journald versions, the default value for Storage is auto. This means persistent log will be used if /var/log/journal exists and volatile otherwise. In my case the directory did not exist. Thought it can help somebody although author's issue was different.

btw persistent storage also doesn't guarantee persistent logs. If it fails to create /var/log/journal, then log will still be volatile.

edit flag offensive delete link more
0

answered 2015-03-19 09:05:29 -0500

Peter_ gravatar image

updated 2015-03-19 09:06:55 -0500

The "Runtime" messages are about journald working in initramfs before root is mounted. So nothing unusual in that.

A problem is that you seem to be running some kind of OS containers version of F21 on top of a Linode, Debian host OS (see the kernel messages). Perhaps "/var/log/journal/(id-number)/" isn't persistent ? Try to see if ordinary files in that place survive a reboot.

So check this out with Linode before filing a bug for F21.

edit flag offensive delete link more

Comments

Linode uses Xen with virtualized disks. The third disk, /dev/xvdc, is mounted on /var. I'm pretty sure it's persistent, because it also holds my database cluster!

jjv gravatar imagejjv ( 2015-03-19 09:31:50 -0500 )edit

By default journald stores the logs in "/run/log/journal/" which is volatile (doesn't survive reboot). If /var/log/journal/ exist it will attempt to store the journal there. When systemd and journald pivots from initramfs into the real rootfs, journald is restarted with a "SIGUSR1" signal, that makes it flush the volatile /run/ to /var/ in order to make the journal persistent across boots. It doesn't seem unlikely to me that the problem is there: either /var comes up way too late, or there is some kind of permission problem, or similar.

Peter_ gravatar imagePeter_ ( 2015-03-19 12:49:33 -0500 )edit

Question Tools

1 follower

Stats

Asked: 2015-03-19 04:51:16 -0500

Seen: 10,779 times

Last updated: Nov 16 '17