CS 480 M01 Lecture Notes 12 - Oct 3, 2022

Last Time

Ch 2. Booting and System Management Daemons Continued
systemd in detail continued
systemctl examples, subcommands, statuses
Targets ( multiuser, graphical, poweroff, rescue, ...) and mapping to init run levels
Dependencies among units, Local services, systemd logging ( journal , journald , journalctl /etc/systemd/journald.conf)
Reboot and shutdown procedures
Stratagems for a nonbooting system
10. Logging
Log Management : Collect , query, analyze, filter, monitor, retention, and expiration
syslog
systemd-journal - collects messages, indexed and compressed binary format, command-line interface for viewing and filtering
Can stand alone or coexist with syslog
Logging architecture for a site with centralized logging
Log Files and Locations
Systemd journal

10. Logging Continued

Systemd journal

systemd's mission to replace all other Linux subsystems ... => includes systemd-journald
Duplicates most of syslog functions but can run in tandem with syslog (configurable).
Binary format for the entries ( => need journalctl X syslog entries in txt ...) and all message attributes indexed - fast search
Collects and indexes messages from several sources:
• /dev/log socket - msgs from software submitting per syslog conventions
• /dev/kmsg - from Linux kernel (replacing former klogd)
• /run/systemd/journal/stdout UNIX socket for service msgs to STDOUT
• /run/systemd/journal/socket UNIX socket for msgs from software using systemd API
• kernel's auditd daemon messages
systemd-journal-remote utility (plus related -...-gateway and ...-upload) to sent to remote journal - not installed by default

Configuration

/etc/systemd/journald.conf contains commented out defaults; not intended for direct editing (per the textbook) plus /etc/systemd/journald.conf.d/*.conf
• Storage= volatile/persisten/auto/none
      most Linux systems come with auto but no /var/log/journal/ nor /etc/systemd/journald.conf.d directories ...
• Compress (for large entries before writing into file)
• Seal=yes/no - enables Forward Secure Sealing (FSS) to protect journal files
• RateLimitInterval and RateLimitBurst : if limit reached within interval=> drop until the interval is over
• SystemMaxUse, SystemKeepFree, SystemMaxFileSize, SystemMaxFiles, RuntimeMaxUse, RuntimeKeepFree, RuntimeMaxFileSize, RuntimeMaxFiles - limits on files stored
• ...
journalctl filtering options

journalctl --disk-usage
journalctl --list-boots
journalctl -u sshd -b -1
journalctl -u sshd -b 0
journalctl -u sshd --since=yesterday --until=today
journalctl --since "2022-10-03 08:00:00"  -u sshd
journalctl --since "2021-10-3 08:30:00"

Coexisting with syslog
Both syslog and systemd journal active by default on most Linux systems
journal is missing many features available in syslog (input plug-ins plus forward to different outputs) ...

Syslog

• 2 main functions:
  • liberate programmers from mechanics or writing log files
  • puts sysadmins in control of logging
• Architecture
  • messages - streams of events
  • rsyslog - event-stream processing engine.
  3 parts
  1. (r)syslogd - /etc/(r)syslog.conf plus /etc/rsyslog.d/*.conf, starts at boot time reads from /dev/log (systems w/o systemd)
     if /etc/rsyslog.conf modified then rsyslogd needs to be restarted (HUP just makes rsyslogd to close open files)
  2. openlog - library routines for syslog -> /dev/log (UNIX domain socket)
  3. logger - user level command to submit log entries
• messages: time stamp, hostname, process name and ID, payload
    text files => standard tools can be used (grep, less, cat, awk, ...)

Configuring rsyslog

• Global properties for the daemon itself: what modules to load, default msg format, ownership and permissions, ...
• Include directive - include additional files from a configuration directory /etc/rsyslog.d/*.conf
• Filters / Selectors - define how to sort and process messages (expression to select and action to process):
  1. Lines - original syslog format
  2. Legacy rsyslog directives starting with '$'
  3. RainerScript
Traditional syslog format
selector < Tab > action
selector = facility.severity
    mail.info /var/log/maillog
facility1,facility2,...,facilityN.level : selects the level from all facilities 1 through N
    auth,authpriv.info /var/log/authlog
; means OR (couple of selectors for the same action)
   
facility names , severity levels, priority level qualifiers, and actions


plus
~ - to discard message, and
^program;template - to format msg per template and send it to program
/etc/rsyslog.conf rsyslog.com/doc
processed line by line - order is significant

• Global properties for the daemon itself:
  • what modules to load:
    

    $ModLoad immark.so
    $ModLoad imuxsock.so
    $ModLoad imklog.so
    ...

  • default msg format, ownership and permissions, ...
    

    $FileOwner root
    $FileGroup adm
    $ActionFileDefaultTemplate RSYSLOG_FileFormat
    $MarkMessagePeriod      3600
    $RepeatedMsgReduction   on
    $klogConsoleLogLevel    1
    ...
    	

• Include directive(s)

  $IncludeConfig /run/rsyslog/additional-log-sockets.conf
  $IncludeConfig /etc/rsyslog.d/*.conf
  	

• Filters / Selectors and Actions - bulk of the configuration
  Three configuration syntaxes:
  • sysklogd format (original sylog config file format)
  • Legacy rsyslog directives
  • RainerScript
Modules
Check the book, rsyslog.conf, and module html files
Names:
• im - input modules
• om - output modules
• pm - parser modules
• mm - message modifiers/li>
• ...
Legacy directives
Can configure all aspects of rsyslog (global options, modules, filtering, rules).
Mostly used to configure modules and the daemon itself.

	$ModLoad immark.so
	$MarkMessagePeriod      3600
	$ModLoad imuxsock.so
	$RepeatedMsgReduction   on
	$ModLoad imklog.so
	$klogConsoleLogLevel    1


RainerScript
RainerScript syntax : event-stream-processing language with filtering and control-flow capabilites.
Can set global parameters, load modules, .... but main benefits relate to filtering capabilities:
Book example:

	if	($syslogfacility-text == 'auth') then {
		action(type="omfile" file="/var/log/auth.log")
	}

Leap 15.4 / our VM example below ...

More config examples in the textbook
Leap 15.4 Default Example

...
# since rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# provides --MARK-- message capability (every 1 hour)
$ModLoad immark.so
$MarkMessagePeriod      3600
# provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock.so
# reduce dupplicate log messages (last message repeated n times)
$RepeatedMsgReduction   on
# kernel logging (may be also provided by /sbin/klogd)
# see also http://www.rsyslog.com/doc-imklog.html.
$ModLoad imklog.so
# set log level 1 (same as in /etc/sysconfig/syslog).
$klogConsoleLogLevel    1
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup root
$FileCreateMode 0640
$DirCreateMode 0750
$Umask 0022
# Use rsyslog native, rfc5424 conform log format as default
# ($ActionFileDefaultTemplate RSYSLOG_FileFormat).
#
# To change a single file to use obsolete BSD syslog format
# (rfc 3164, no high-precision timestamps), set the variable
# bellow or append ";RSYSLOG_FileFormat" to the filename.
# See
#   http://www.rsyslog.com/doc/rsyslog_conf_templates.html
# for more informations.
#
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Include config generated by /etc/init.d/syslog script
# using the SYSLOGD_ADDITIONAL_SOCKET* variables in the
# /etc/sysconfig/syslog file.
#
$IncludeConfig /run/rsyslog/additional-log-sockets.conf
#
# Include config files, that the admin provided? :
#
$IncludeConfig /etc/rsyslog.d/*.conf
###
# print most important on tty10 and on the xconsole pipe
#
if	( \
	    /* kernel up to warning except of firewall  */ \
	    ($syslogfacility-text == 'kern')      and      \
	    ($syslogseverity <= 4 /* warning */ ) and not  \
	    ($msg contains 'IN=' and $msg contains 'OUT=') \
	) or ( \
	    /* up to errors except of facility authpriv */ \
	    ($syslogseverity <= 3 /* errors  */ ) and not  \
	    ($syslogfacility-text == 'authpriv')           \
	) \
then {
	/dev/tty10
	|/dev/xconsole
}
# Emergency messages to everyone logged on (wall)
*.emerg					 :omusrmsg:*
# enable this, if you want that root is informed
# immediately, e.g. of logins
#*.alert				root
#
# Additional filter rules
#
$IncludeConfig /etc/rsyslog.d/*.frule
#
# email-messages
#
mail.*					-/var/log/mail
mail.info				-/var/log/mail.info
mail.warning				-/var/log/mail.warn
mail.err				 /var/log/mail.err
#
# news-messages
#
#news.crit				-/var/log/news/news.crit
#news.err				-/var/log/news/news.err
#news.notice				-/var/log/news/news.notice
# enable this, if you want to keep all news messages
# in one file
#news.*					-/var/log/news.all
#
# Warnings in one file
#
*.=warning;*.=err			-/var/log/warn
*.crit					 /var/log/warn
#
# the rest in one file
#
*.*;mail.none;news.none			-/var/log/messages
#
# enable this, if you want to keep all messages
# in one file
#*.*					-/var/log/allmessages
#
# Some foreign boot scripts require local7
#
local0.*;local1.*			-/var/log/localmessages
local2.*;local3.*			-/var/log/localmessages
local4.*;local5.*			-/var/log/localmessages
local6.*;local7.*			-/var/log/localmessages

TLS ...

• kernel:
  • need record of boot process while avoiding dependancy on particular filesystem or filesystem organization
  • logging into kernel's internal buffer of limited size + systemd-journald
    Reading from /dev/kmsg by journalctl --dmesg / -k or dmesg with output redirected to
    • /var/log/messages - Suse
    • /var/log/boot.msg and/or /var/log/boot.log - Suse
    • /var/log/dmesg - RHEL, Fedora, Ubuntu, Debian
  • klogd - WAS ongoing logging
• startup scripts ...

implements a variety of log management policies

Textbook : "logrotate is normally run out of cron once a day..."
Man page: "logrotate is run as a daily cron job"
BUT ...

• /etc/logrotate.conf - series of specifications for groups of log files to be managed.
• /etc/logrotate.d
  + more in the book ... and lab assignment ...