Linux · Lesson 14

Logs, journalctl & System Evidence

Use the systemd journal, authentication logs, package history, kernel records, service logs, and application evidence to build defensible timelines before making security decisions.

Logs are not just error messages. They are evidence. Strong analysis preserves the original record, narrows the time window, correlates independent sources, and separates what happened from what you think caused it.

Lesson Snapshot

Core concepts6

Journal filters8

Inspection commands14

Main goalEvidence-based timelines

Learning objectives

Turn raw logs into defensible system evidence

Explain the difference between the systemd journal, traditional log files, application logs, and authentication records.

Use journalctl filters to narrow evidence by service, boot, time, priority, user, and executable.

Preserve timestamps, source context, and file metadata before modifying or rotating logs.

Correlate system logs with services, users, processes, network activity, and configuration changes.

Recognize common signs of tampering, gaps, repeated failures, and suspicious authentication activity without over-interpreting one event.

Build a concise evidence timeline that supports defensive decisions and post-change verification.

Core idea

Logs answer different parts of the same story

Authentication logs may show who connected, package logs may show what changed, and a service journal may show what broke. Correlate them by time and context instead of treating each source alone.

Evidence risk

Cleaning logs during investigation destroys your timeline

Do not vacuum, truncate, delete, or rotate evidence just to make the system look cleaner. Preserve relevant records first and make storage changes only when the investigation is complete.

Logging fundamentals

Six concepts behind Linux evidence

Journal

systemd-journald collects structured event records from the kernel, services, user sessions, and other system components.

Traditional logs

Many Ubuntu-family systems also use files under /var/log such as auth.log, syslog, apt history, and application-specific logs.

Timestamp

Time is essential for correlating events, but system clock changes, time zones, log rotation, and delayed writes can complicate interpretation.

Source

A log record is stronger when you know which service, process, executable, unit, user, or kernel component produced it.

Retention

Logs may be rotated, compressed, stored only in memory, or limited by size. Missing older logs do not automatically prove tampering.

Correlation

One event rarely tells the whole story. Strong defensive analysis connects logs with users, processes, services, files, packages, and network evidence.

Important log locations

Know where system evidence may be stored

/var/log/auth.log

Common Ubuntu-family authentication log containing SSH, sudo, login, and related security events.

Availability depends on logging configuration. Do not assume it exists on every image.

/var/log/syslog

General system log on systems using rsyslog or similar traditional logging.

Some events may instead be visible only through the journal.

/var/log/kern.log

Kernel-related events on systems configured to write them separately.

journalctl -k may provide the same or broader evidence depending on configuration.

/var/log/apt/history.log

APT transaction history showing package install, upgrade, and removal activity.

Use together with dpkg logs and current package state.

/var/log/dpkg.log

Detailed package-management activity recorded by dpkg.

Package events are not the same thing as user intent; correlate with time and administrator activity.

/var/log/ufw.log

Firewall events where UFW logging is configured to write here.

UFW records may instead appear in kernel or journal sources depending on the system.

/var/log/

Main directory for traditional system and application logs.

Application logs can live elsewhere. Inspect service configuration before assuming a log path.

/var/log/journal/

Persistent journal storage when persistent journaling is enabled.

If this directory is absent, the journal may be volatile under /run/log/journal.

/run/log/journal/

Volatile journal storage for the current boot or runtime.

Data here can disappear after reboot.

Inspection first

Narrow the evidence instead of scrolling endlessly

journalctl -b

Show journal records from the current boot.

journalctl -b -1

Show the previous boot when retained.

journalctl --list-boots

List retained boot sessions and their time ranges.

journalctl -u service-name --since today

Show service-specific journal records from today.

journalctl --since '2026-10-10 08:00' --until '2026-10-10 09:00'

Review a bounded time window. Replace the example date and time with the actual investigation window.

journalctl -p warning..alert -b

Show warning through alert priority records from the current boot.

journalctl _PID=1234

Filter by one process ID when that PID is relevant and still meaningful to the retained records.

journalctl _UID=1000

Filter records associated with one user ID.

journalctl _COMM=sshd

Filter by command name stored in journal metadata.

journalctl -k -b

Show current-boot kernel messages.

Review recent authentication-related events when auth.log is present.

sudo zgrep -i 'pattern' /var/log/auth.log*

Search current and compressed rotated authentication logs for a specific pattern.

sudo tail -n 100 /var/log/apt/history.log

Review recent APT package transactions.

sudo tail -n 100 /var/log/dpkg.log

Review recent dpkg package activity.

journalctl filters

Filter by context, not just keywords

-b / -b -1

Select the current or previous boot.

Useful for comparing before-and-after reboot behavior.

-u UNIT

Filter by systemd unit.

Best for service-specific troubleshooting and verification.

--since / --until

Limit output to a time window.

Essential for building a focused timeline.

-p PRIORITY

Filter by severity such as warning, err, crit, alert, or emerg.

Useful for triage, but normal-priority events may still matter.

_PID= / _UID=

Filter by process or user metadata.

Useful when correlating activity with a known process or account.

_COMM= / _EXE=

Filter by command name or executable path where journal metadata contains it.

Helps connect events to the program that produced them.

-k

Show kernel messages.

Useful for drivers, interfaces, storage, firewall, and low-level system events.

-o short-iso

Use an ISO-like output format with precise timestamps.

Web errors, database failures, application authentication, startup issues.

Useful sources

application-specific logs and service journal

Evidence timeline

Build a sequence that separates fact from interpretation

Define the event window

Start with the scenario clue, user report, service failure, or suspicious event and choose the smallest useful time range.

Capture system context

Record boot ID, hostname, current time, uptime, users, services, and relevant process state.

Collect primary logs

Review journal records and the most relevant traditional or application log files.

Correlate sources

Match timestamps across authentication, services, packages, processes, network events, and file changes.

Separate facts from interpretation

Write down what the log actually says before deciding what you think caused it.

Verify after remediation

Compare post-change logs and service behavior to confirm the intended result and detect new failures.

Authentication evidence

Read login records in context

Failed password

Authentication attempt did not succeed.

Next question

Correlate source, username, frequency, and later accepted sessions.

Invalid user

Next question

Review source and whether the account was later created.

Accepted password / Accepted publickey

SSH authentication succeeded.

Next question

Determine whether the account and source were authorized and what happened afterward.

sudo:

A sudo-related event was recorded.

Next question

Review invoking user, target command, target user, and scenario authorization.

session opened / closed

A PAM-backed session changed state.

Next question

Use surrounding events to understand the complete login or privilege sequence.

authentication failure

A PAM or application authentication attempt failed.

Next question

Identify the service, account, and source before deciding significance.

Tamper clues

Missing evidence needs explanation, not immediate accusation

Unexpected gap across several independent log sources

More significant than a gap in one rotated file, especially if the time window overlaps suspicious activity.

Logging service stopped or disabled

Can reduce visibility, but may also result from misconfiguration or service failure.

Abrupt timestamp discontinuity

May indicate clock change, reboot, time synchronization correction, or manipulation.

Rotated logs missing unexpectedly

Could indicate retention policy, cleanup, disk pressure, or deliberate removal.

Application logs stop while service continues

May indicate logging failure, changed configuration, permissions, or tampering.

Journal storage changed from persistent to volatile

Service accounts should never appear in logs.

Better reasoning

Determine whether the account was used interactively, by a service, through sudo, or by an application, then compare that behavior with the account's intended role.

Evidence sources

Preserve the records that support the timeline

System journal

journalctl -b -o short-iso

Structured system events from the current boot with useful metadata.

Authentication log

sudo tail -n 100 /var/log/auth.log

Recent SSH, sudo, PAM, and login events where the file exists.

Package history

sudo tail -n 100 /var/log/apt/history.log

Recent APT package transactions that can explain software changes.

Kernel events

journalctl -k -b

Current-boot kernel events that can explain device, network, storage, and firewall behavior.

Service journal

journalctl -u service-name --since today

Focused evidence for one systemd-managed service.

Boot history

journalctl --list-boots

Shows retained boots and helps identify reboots or missing history.

Controlled evidence handling

Export and inspect without destroying originals

Export a focused journal window

journalctl --since 'START' --until 'END' -o short-iso > investigation-journal.txt

Write exports to an authorized evidence location and preserve original timestamps and context.

Export one service's logs

journalctl -u service-name --since today -o short-iso > service-journal.txt

Use service logs as one evidence source, not the only source.

View rotated compressed logs

zless /var/log/auth.log.2.gz

Read compressed logs without modifying them.

Check journal storage use

journalctl --disk-usage

Do not vacuum or delete journals just to save space during an investigation.

Verify journald configuration

systemctl status systemd-journald

If logging is unhealthy, preserve current evidence before restarting services when possible.

Inspect log permissions

stat /var/log/auth.log

Do not weaken permissions to make logs easier to read. Use appropriate authorized privileges.

High-risk shortcuts

Log handling mistakes that destroy evidence

Clearing logs before understanding the incident

Destroys evidence needed for forensics, troubleshooting, and verification.

Using journalctl --vacuum-* during active investigation

Can remove historical records that may be relevant to the timeline.

Restarting logging services repeatedly

Can complicate the timeline and create new log gaps or noise.

Treating one failed login as proof of compromise

Failed attempts are evidence of activity, not successful access.

Ignoring rotated logs

Older but relevant evidence may be stored in .1, .2.gz, or other rotated files.

Relying on only one log source

A single source can be incomplete, rotated, misconfigured, or misleading without correlation.

Verification

Use post-change logs to prove the system improved

Logging services

Confirm systemd-journald and any required traditional logging services are healthy.

Service behavior

Verify the remediated service produces expected normal logs and no new failure loop.

Authentication

Confirm authorized access succeeds and unauthorized patterns are no longer producing unexpected successful sessions.

Package state

Confirm package changes match the intended administrative action.

Timeline

Record the before-and-after timestamps that prove when the change occurred and what followed.

Evidence preservation

Ensure the original relevant logs or exports remain available for later review.

Defensive lab

Build a fictional one-hour evidence timeline

Read the fictional scenario: Morgan reports that SSH briefly failed around 08:20, a new package was installed near the same time, and a service restarted repeatedly.

Record current time, uptime, boot history, SSH status, affected service status, and package state without changing anything.

Build a focused 08:00-09:00 timeline using journalctl, auth.log, apt history, dpkg.log, and the affected service journal.

Separate facts from interpretation: note failed SSH attempts, any accepted login, the package transaction, service restart timestamps, and configuration errors.

Determine whether the evidence supports a simple misconfiguration, authorized maintenance, or suspicious activity in the fictional lab.

Apply only the justified remediation and preserve the evidence window before making changes.

Verify SSH, the affected service, logging health, and post-change journal records, then summarize the final timeline.

Lesson checklist

Before moving on

I understand the difference between the systemd journal and traditional log files.

I can filter journalctl by boot, service, time, priority, user, and process metadata.

I know rotated and compressed logs may contain relevant older evidence.

I preserve timestamps and source context before changing or deleting anything.

I correlate authentication, service, package, network, and kernel logs instead of relying on one source.

I do not treat failed authentication as proof of successful compromise.

I understand that missing logs can result from rotation, retention, reboot, or misconfiguration as well as tampering.

I avoid clearing or vacuuming logs during active investigation.

I build timelines that separate facts from interpretation.

I verify that logging and required services remain healthy after remediation.

Reflection

Check your reasoning

Question 1

Why is one log source rarely enough to explain a security event?

Question 2

Why can a gap in auth.log have a non-malicious explanation?

Question 3

What is the difference between a failed SSH login and an accepted SSH login?

Question 4

Why is a bounded time window useful when using journalctl?

Question 5

What evidence would you correlate with a recent package installation?

Question 6

What should you preserve before restarting a failing service during an investigation?

Carry this forward

Treat logs as evidence, not decoration. Narrow the time window, correlate independent sources, preserve originals, and write a timeline that clearly separates facts from conclusions.

Back to Linux CyberPatriot Hub

← Previous Lesson Next Lesson →