Bug 40901: (QA follow-up) Keep daemon output in the traditional log files under systemd

Under koha-sysv the daemon(1) wrapper captures each daemon's stdout and
stderr to /var/log/koha/INSTANCE/NAME-output.log and NAME-error.log.
The systemd units sent that output to the journal instead, so
zebra-*.log, indexer-*.log, es-indexer-*.log, worker-*.log and
sip-error.log silently stopped being written on migrated hosts, and
log locations differed between the two init packages.

Use StandardOutput=append: and StandardError=append: on those six
units so the files are identical whichever package supervises the
daemons. The shipped logrotate stanza uses copytruncate, which is safe
with append-mode descriptors. The journal still records systemd's own
messages about each unit, which is what matters when one fails to
start, and README.Debian shows the drop-in for sites that prefer the
journal. Plack and the Z39.50 responder already wrote their own files.

The README, NEWS.Debian entry and koha-common(8) text that described
the journal-only behaviour are updated accordingly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>

debian/docs/koha-common.xml

@@ -31,7 +31,7 @@
       <varlistentry>
         <term><option>koha-systemd</option></term>
         <listitem>
-          <para>The default on systemd hosts. Every daemon of every instance is its own unit: <literal>koha-plack@INSTANCE.service</literal>, <literal>koha-zebra@INSTANCE.service</literal>, <literal>koha-indexer@INSTANCE.service</literal>, <literal>koha-es-indexer@INSTANCE.service</literal>, <literal>koha-worker@INSTANCE.service</literal>, <literal>koha-worker-long_tasks@INSTANCE.service</literal>, <literal>koha-sip@INSTANCE.service</literal> and <literal>koha-z3950@INSTANCE.service</literal>, grouped by <literal>koha@INSTANCE.target</literal> per instance and <literal>koha.target</literal> for the whole host. <literal>systemctl restart koha-common</literal> still restarts everything. The daemons' standard output and error go to the journal (<literal>journalctl -u koha-zebra@INSTANCE</literal>); see <filename>/usr/share/doc/koha-systemd/README.Debian</filename>.</para>
+          <para>The default on systemd hosts. Every daemon of every instance is its own unit: <literal>koha-plack@INSTANCE.service</literal>, <literal>koha-zebra@INSTANCE.service</literal>, <literal>koha-indexer@INSTANCE.service</literal>, <literal>koha-es-indexer@INSTANCE.service</literal>, <literal>koha-worker@INSTANCE.service</literal>, <literal>koha-worker-long_tasks@INSTANCE.service</literal>, <literal>koha-sip@INSTANCE.service</literal> and <literal>koha-z3950@INSTANCE.service</literal>, grouped by <literal>koha@INSTANCE.target</literal> per instance and <literal>koha.target</literal> for the whole host. <literal>systemctl restart koha-common</literal> still restarts everything. Each daemon's standard output and error are captured to the same <filename>/var/log/koha/INSTANCE/NAME-output.log</filename> and <filename>NAME-error.log</filename> files as with koha-sysv; systemd's own messages about a unit are in the journal (<literal>journalctl -u koha-zebra@INSTANCE</literal>). See <filename>/usr/share/doc/koha-systemd/README.Debian</filename>.</para>
         </listitem>
       </varlistentry>
       <varlistentry>

debian/koha-systemd.NEWS

@@ -15,12 +15,11 @@ koha (26.06.00~1) unstable; urgency=medium
     "systemctl restart koha-common" still restarts everything, and the
     koha-plack/koha-zebra/koha-worker/... helpers work as before.
 
-  * Daemon stdout/stderr goes to the journal, not to
+  * Log files are unchanged: each daemon's stdout/stderr still lands in
-    /var/log/koha/INSTANCE/NAME-output.log and NAME-error.log. Use
+    /var/log/koha/INSTANCE/NAME-output.log and NAME-error.log, and the
-    "journalctl -u koha-zebra@INSTANCE" (or -t INSTANCE-koha-zebra).
+    files Koha writes itself (plack.log, the log4perl files, z3950.log)
-    Logs written by Koha itself (plack.log, plack-error.log, the log4perl
+    are untouched. The journal holds systemd's own messages about each
-    files such as plack-opac-error.log, sip-output.log, z3950.log) are
+    unit ("journalctl -u koha-zebra@INSTANCE" shows why one failed).
-    unchanged.
 
   * The units are sandboxed (PrivateTmp, ProtectSystem=full, ProtectHome,
     NoNewPrivileges). Instances whose koha-conf.xml has an empty

debian/koha-systemd.README.Debian

@@ -88,29 +88,14 @@ unit after changing them.
 
 ## Logging
 
-This is the one visible difference from the init script. With koha-sysv the
+Log locations are the same with either init package. With koha-sysv the
-daemon(1) wrapper captured each daemon's standard output and error to
+daemon(1) wrapper captures each daemon's standard output and error to
-/var/log/koha/INSTANCE/NAME-output.log and NAME-error.log. With systemd
+/var/log/koha/INSTANCE/NAME-output.log and NAME-error.log; the systemd
-that output goes to the journal instead:
+units do the same through StandardOutput=append: and StandardError=append:,
-
+so zebra-*.log, indexer-*.log, es-indexer-*.log, worker-*.log and
-    journalctl -u koha-zebra@library.service
+sip-*.log keep being written where they always were and the shipped
-    journalctl -t library-koha-worker         # by SyslogIdentifier
+logrotate configuration keeps rotating them. Everything Koha writes itself
-    journalctl -u 'koha-*@library.service'    # everything for one instance
+is unaffected either way:
-
-So on a koha-systemd host these files are no longer written:
-
-    zebra-output.log, zebra-error.log
-    indexer-output.log, indexer-error.log
-    es-indexer-output.log, es-indexer-error.log
-    worker-output.log, worker-error.log (the worker's log4perl appender
-                                         writes to stderr, so its messages
-                                         are in the journal too)
-    sip-error.log                       (SIP's own log4perl output still
-                                         goes to sip-output.log)
-
-Everything the application writes itself is unchanged, because it is
-configured in /etc/koha/sites/INSTANCE/log4perl.conf or passed on the
-command line, not captured from stdout:
 
     plack.log, plack-error.log          (Starman --access-log/--error-log)
     plack-opac-error.log, plack-intranet-error.log, plack-api-error.log,
@@ -120,8 +105,20 @@ command line, not captured from stdout:
     opac-error.log, intranet-error.log, opac-access.log, ...
                                         (Apache, per vhost)
 
-Existing NAME-output.log/NAME-error.log files are left in place; logrotate
+The journal therefore only holds what systemd itself records about a unit
-keeps rotating whatever is there.
+(starts, stops, failures, restart-limit hits), which is where to look when
+a unit refuses to start:
+
+    journalctl -u koha-zebra@library.service
+    journalctl -u 'koha-*@library.service'    # everything for one instance
+
+If you would rather have a daemon's output in the journal, override the two
+directives with a drop-in:
+
+    systemctl edit koha-worker@library.service
+    [Service]
+    StandardOutput=journal
+    StandardError=journal
 
 ## Sandboxing
 

debian/systemd/koha-es-indexer@.service

@@ -13,6 +13,10 @@ Type=simple
 User=%i-koha
 Group=%i-koha
 SyslogIdentifier=%i-koha-es-indexer
+# Capture stdout/stderr to the same files the daemon(1) wrapper used under
+# SysV, so log locations are identical whichever init package is installed.
+StandardOutput=append:/var/log/koha/%i/es-indexer-output.log
+StandardError=append:/var/log/koha/%i/es-indexer-error.log
 WorkingDirectory=/var/lib/koha/%i
 Environment=KOHA_CONF=/etc/koha/sites/%i/koha-conf.xml
 Environment=KOHA_HOME=/usr/share/koha

debian/systemd/koha-indexer@.service

@@ -13,6 +13,10 @@ Type=simple
 User=%i-koha
 Group=%i-koha
 SyslogIdentifier=%i-koha-indexer
+# Capture stdout/stderr to the same files the daemon(1) wrapper used under
+# SysV, so log locations are identical whichever init package is installed.
+StandardOutput=append:/var/log/koha/%i/indexer-output.log
+StandardError=append:/var/log/koha/%i/indexer-error.log
 WorkingDirectory=/var/lib/koha/%i
 Environment=KOHA_CONF=/etc/koha/sites/%i/koha-conf.xml
 Environment=KOHA_HOME=/usr/share/koha

debian/systemd/koha-sip@.service

@@ -15,6 +15,10 @@ Type=simple
 User=%i-koha
 Group=%i-koha
 SyslogIdentifier=%i-koha-sip
+# Capture stdout/stderr to the same files the daemon(1) wrapper used under
+# SysV, so log locations are identical whichever init package is installed.
+StandardOutput=append:/var/log/koha/%i/sip-output.log
+StandardError=append:/var/log/koha/%i/sip-error.log
 WorkingDirectory=/var/lib/koha/%i
 Environment=KOHA_CONF=/etc/koha/sites/%i/koha-conf.xml
 Environment=KOHA_HOME=/usr/share/koha

debian/systemd/koha-worker-long_tasks@.service

@@ -13,6 +13,10 @@ Type=simple
 User=%i-koha
 Group=%i-koha
 SyslogIdentifier=%i-koha-worker-long_tasks
+# Capture stdout/stderr to the same files the daemon(1) wrapper used under
+# SysV, so log locations are identical whichever init package is installed.
+StandardOutput=append:/var/log/koha/%i/worker-output.log
+StandardError=append:/var/log/koha/%i/worker-error.log
 WorkingDirectory=/var/lib/koha/%i
 Environment=KOHA_CONF=/etc/koha/sites/%i/koha-conf.xml
 Environment=KOHA_HOME=/usr/share/koha

debian/systemd/koha-worker@.service

@@ -13,6 +13,10 @@ Type=simple
 User=%i-koha
 Group=%i-koha
 SyslogIdentifier=%i-koha-worker
+# Capture stdout/stderr to the same files the daemon(1) wrapper used under
+# SysV, so log locations are identical whichever init package is installed.
+StandardOutput=append:/var/log/koha/%i/worker-output.log
+StandardError=append:/var/log/koha/%i/worker-error.log
 WorkingDirectory=/var/lib/koha/%i
 Environment=KOHA_CONF=/etc/koha/sites/%i/koha-conf.xml
 Environment=KOHA_HOME=/usr/share/koha

debian/systemd/koha-zebra@.service

@@ -13,6 +13,10 @@ Type=forking
 User=%i-koha
 Group=%i-koha
 SyslogIdentifier=%i-koha-zebra
+# Capture stdout/stderr to the same files the daemon(1) wrapper used under
+# SysV, so log locations are identical whichever init package is installed.
+StandardOutput=append:/var/log/koha/%i/zebra-output.log
+StandardError=append:/var/log/koha/%i/zebra-error.log
 WorkingDirectory=/var/lib/koha/%i
 Environment=KOHA_CONF=/etc/koha/sites/%i/koha-conf.xml
 Environment=KOHA_HOME=/usr/share/koha