Practice Lab · 06 · Ubuntu / Mint
Linux Services, Logs & Scheduled Activity
Correlate systemd services, processes, journal evidence, cron, timers, and network listeners before making defensive changes.
This lab focuses on behavior over names. A service, timer, or script becomes meaningful when configuration, execution, logs, schedule, and network state all line up.
Lab Snapshot
Mission briefing
Preserve required services while reducing unnecessary exposure
This fictional Ubuntu/Mint workstation supports a school robotics team and must remain usable for development and normal network access.
Approved users are alex, priya, and root.
SSH is not required unless the mission briefing explicitly says otherwise.
The robotics application relies on a local background service named robot-agent.service.
The workstation may use cron and systemd timers for legitimate maintenance tasks.
Do not stop services, delete logs, remove cron entries, or disable timers until you understand their role.
Preserve process, service, log, schedule, and network evidence before making changes.
After remediation, approved development tools and robot-agent.service must still work.
Service review
Understand what is configured and running
List running services
Identify active services and build a baseline before deciding what is expected or unnecessary.
systemctl --type=service --state=running
systemctl list-units --type=service
Inspect one service
Review service state, unit configuration, executable path, and dependencies.
systemctl status robot-agent.service --no-pager
systemctl cat robot-agent.service
Review enabled services
See which services are configured to start automatically at boot.
systemctl list-unit-files --type=service | grep enabled
Map services to processes
Connect service configuration with the processes actually running.
ps aux
ps -eo pid,ppid,user,comm,args --sort=ppid
Log review
Use journals and auth logs to add time and context
System journal
Review recent warnings and boot activity without assuming every warning indicates compromise.
journalctl -p warning --since today
journalctl -b --no-pager | tail -n 100
Service-specific journal
Correlate service behavior with log messages and timestamps.
journalctl -u robot-agent.service --since today --no-pager
Authentication logs
Review login, sudo, and authentication activity when auth.log is available.
sudo tail -n 100 /var/log/auth.log 2>/dev/null
sudo grep -Ei "sudo|session opened|session closed|failed|accepted" /var/log/auth.log 2>/dev/null | tail -n 80
General log directory
Identify available log sources before assuming a specific file exists.
ls -lah /var/log
Scheduled activity
Review cron and systemd timers
User crontab
Review scheduled commands for the current user and root.
crontab -l
sudo crontab -l
System cron locations
Inspect system-wide scheduled activity and maintenance jobs.
ls -la /etc/cron.d /etc/cron.daily /etc/cron.hourly /etc/cron.weekly 2>/dev/null
sudo cat /etc/crontab
Systemd timers
Identify recurring jobs managed by systemd rather than cron.
systemctl list-timers --all
Inspect one timer
Practice understanding timer triggers and linked service units before making changes.
systemctl status apt-daily.timer --no-pager
systemctl cat apt-daily.timer
Network review
Connect services to exposure
Listening sockets
Identify services listening on TCP or UDP ports and map them to processes.
ss -tulpn
sudo lsof -i -P -n | head -n 80
Active connections
Review active network connections and owning processes.
ss -tunap
IP and route state
Confirm interface and routing state before blaming services for a connectivity issue.
ip addr
ip route
Firewall state
Check host firewall configuration without disabling it to troubleshoot connectivity.
sudo ufw status verbose 2>/dev/null
sudo nft list ruleset 2>/dev/null | head -n 120
Fictional evidence set
Correlate behavior before remediation
robot-agent.service is active and enabled. Its unit file launches /opt/robotics/agent/robot-agent.
Investigation prompt
Which mission requirement makes this service important to preserve?
ssh.service is active and enabled even though SSH is not required by the mission briefing.
Investigation prompt
What listener, package, account, and usage evidence should you review before changing it?
ss -tulpn shows a listener on TCP 22 owned by sshd.
Investigation prompt
How does this strengthen the case that SSH is actually exposed rather than merely installed?
auth.log shows several successful SSH logins for priya from a local network address.
Investigation prompt
Does this automatically mean the service is required, or does the mission briefing still control the decision?
root's crontab runs /usr/local/bin/backup-robotics.sh every night at 23:00.
Investigation prompt
What evidence should you collect before removing or editing the job?
A systemd timer named cleanup-temp.timer runs /usr/local/bin/cleanup-temp.sh every hour.
Investigation prompt
What should you inspect about the linked service and script before deciding whether it is expected?
robot-agent.service restarted twice today after exiting with a non-zero status.
Investigation prompt
Why should you troubleshoot the service rather than simply disable it?
A process named updater.sh is running from /tmp/update-cache/updater.sh under user alex.
Investigation prompt
What path, parent process, command line, hash, and network evidence should you preserve before terminating it?
Correlation flow
Connect service, process, log, schedule, and network evidence
Service
Identify the unit, enabled state, executable path, service account, and dependencies.
Process
Confirm what process is actually running and capture PID, PPID, user, and command line.
Logs
Use journal or file-based logs to understand recent service activity and failures.
Schedule
Check cron and systemd timers for recurring execution related to the service or script.
Network
Determine whether the process listens on a port or creates active connections.
Decision
Compare the full evidence chain with mission requirements before making a change.
Preserve first
Service and schedule changes can destroy useful context
Record service status, enabled state, unit file, and executable path.
Record process PID, PPID, user, command line, and executable path.
Record relevant journal entries and timestamps.
Record cron lines, timer definitions, and linked service units before editing.
Record listening ports, active connections, and owning processes.
Hash suspicious scripts or executables before deleting or replacing them.
Check whether a scheduled task supports required backups or maintenance.
Document the exact remediation and the post-change verification result.
Authorized remediation examples
Use narrow changes only after the evidence supports them
These examples are for an authorized training VM and should follow the mission briefing and evidence.
Disable an unnecessary SSH service
Use only after confirming the mission does not require SSH and approved workflows do not depend on it.
sudo systemctl disable --now ssh
Stop a non-required timer
Use only after reviewing the linked service and script and confirming the schedule is not required.
sudo systemctl disable --now cleanup-temp.timer
Hash a suspicious script
Preserve file identity before terminating the process or removing the script.
sha256sum /tmp/update-cache/updater.sh
Inspect a cron-owned script
Record ownership, permissions, timestamps, and hash before modifying a scheduled job.
sudo stat /usr/local/bin/backup-robotics.sh && sudo sha256sum /usr/local/bin/backup-robotics.sh
Verification
Confirm required services and connectivity still work
Required service
robot-agent.service remains active and enabled after remediation.
systemctl status robot-agent.service --no-pager
systemctl is-enabled robot-agent.service
SSH
SSH state matches the mission and TCP 22 is not exposed if the service is not required.
systemctl status ssh --no-pager 2>/dev/null
ss -tulpn | grep ':22 ' || true
Schedules
Approved maintenance jobs remain and unjustified scheduled activity is addressed.
crontab -l
sudo crontab -l
systemctl list-timers --all
Processes
Unexpected processes are re-checked and approved robotics processes remain available.
ps -eo pid,ppid,user,comm,args --sort=ppid
Network
Required connectivity remains functional and unnecessary listeners are re-checked.
ss -tulpn
ip route
Logs
No new service failures or obvious breakage appeared after remediation.
journalctl -p warning --since today --no-pager | tail -n 80
Reflection
Debrief the behavior analysis
Why is a service being enabled different from it currently running?
Why is a listening socket stronger evidence of exposure than package installation alone?
Why should cron and systemd timers both be checked on modern Linux systems?
Why can a successful SSH login still be inconsistent with the mission briefing?
Why should a required service that crashes be troubleshot instead of disabled?
What evidence should be preserved before killing a suspicious process?
Why is a scheduled backup job not automatically suspicious?
Which evidence source in this lab was most useful for linking configuration to actual execution?
Completion checklist
Finish only after required behavior is verified
I reviewed running and enabled services.
I inspected robot-agent.service and preserved its required state.
I reviewed journal and authentication logs.
I checked user, root, and system-wide cron jobs.
I reviewed systemd timers and their linked services.
I mapped suspicious services and scripts to processes and network activity.
I preserved hashes and relevant log evidence before destructive changes.
I made only mission-supported service or schedule changes.
I verified required robotics functionality after remediation.
I confirmed unnecessary listeners were no longer exposed when appropriate.