Jumping into journald
On many Linux systems, systemd-journald runs as a daemon at boot and collects your logs. You can access them through journalctl but it turns out journald is a lot more complicated then just sending something to a text file. I’ll look at two main things here: What kind of information is included in a journald entry and how these entries get from programs to journald.
While you’ll mostly see entries as a terse error message on one line, every time you send a message journald collects and stores a lot more information. For example, here’s an error from my audio server, pipewire. Note that some fields are reordered from the raw journalctl output
An entry consists of freeform variables with binary (though generally ASCII/US English) values. Values starting with an underscore are “trusted” and generated by journald while others are sent by the process along with the primary message. This helps provide context about what exact process failed and what state it was in during that failure. Unfortunately the official descriptions of what these fields mean can be a bit obtuse.
While working on my prototype for a system-journald replacement, rjournald I’ve discovered what many of these mean through context or reading the systemd code. You can categorize these into one of a few types
