Linux · Lesson 07
Services & systemd
Inspect Linux services, systemd units, startup behavior, dependencies, logs, process context, and network exposure before stopping or disabling anything.
Services provide the functions the operating system and applications depend on. Strong defensive administration is not about disabling the longest list of services. It is about understanding each service's role, reducing unnecessary exposure, and preserving required functionality.
Lesson Snapshot
Learning objectives
Understand what a service does before changing its state
Explain the difference between a service, a systemd unit, a process, a package, and a socket.
Inspect running, enabled, failed, and masked services before changing their state.
Use systemctl and journalctl to understand why a service exists, what starts it, and what depends on it.
Distinguish required, unnecessary, misconfigured, suspicious, and unknown services using scenario evidence.
Make controlled service changes without breaking networking, SSH, logging, applications, or other required dependencies.
Verify service health after changes using status, logs, listening ports, and client-facing tests.
Core idea
Service security is role security
A required service should be secured, not automatically disabled. An unnecessary service should be removed from the attack surface only after its dependencies and mission impact are understood.
Availability risk
One systemctl command can break multiple functions
Network managers, DNS resolvers, authentication services, SSH, databases, logging, and application backends can have hidden dependencies. Check the relationship graph before high-impact service changes.
systemd fundamentals
Six concepts behind Linux service management
Service
A service is a long-running or background function such as SSH, web hosting, logging, networking, printing, or application support.
Unit
systemd manages resources through units. Service units are common, but sockets, timers, mounts, targets, paths, and other unit types can also influence behavior.
Process
A process is a running program. A service may create one or more processes, and not every process is managed as a standalone service.
Enablement
Enabled means systemd is configured to start a unit automatically in the appropriate context. Running describes current state. They are not the same thing.
Dependency
Units can require, want, order, or otherwise relate to other units. Disabling one service can break another service that depends on it.
State
A unit can be active, inactive, failed, enabled, disabled, static, masked, or in another state. Interpret the state before acting.
Unit states
Running and enabled are different questions
active (running)The service is currently running.
Running does not prove that the service is required or correctly configured.
inactiveThe service is not currently running.
It may still be enabled for future startup or triggered on demand.
failedThe unit entered a failed state.
Investigate logs and dependencies before repeatedly restarting it.
enabledThe unit is configured to start automatically through installed systemd links or related mechanisms.
Enabled does not mean currently running.
disabledThe unit is not configured for normal automatic startup through enablement links.
It can still be started manually or triggered by another mechanism.
staticThe unit does not have ordinary enablement instructions and is typically started as a dependency or trigger.
Do not assume static means unnecessary.
maskedThe unit is deliberately blocked from being started through normal systemd mechanisms.
Masking is stronger than disabling. Use it only when the effect is understood and justified.
Inspection first
Build the service inventory before hardening it
systemctl --type=service --state=runningList currently running service units.
systemctl list-unit-files --type=serviceShow installed service unit files and their enablement states.
systemctl --failedShow units currently in a failed state.
systemctl status service-nameShow current service state, recent log lines, main process information, and unit metadata.
systemctl cat service-nameDisplay the unit definition and any drop-in configuration that systemd is using.
systemctl show service-nameDisplay detailed systemd properties for deeper inspection.
systemctl list-dependencies service-nameShow related dependency units in the service graph.
systemctl list-dependencies --reverse service-nameShow units that may depend on or reference the selected service.
journalctl -u service-name --since todayReview service-specific journal entries from the current day.
ss -tulpnInspect listening sockets and help correlate services with network exposure.
ps -efReview running processes and compare them with systemd service information.
dpkg -S /path/to/executableIdentify the Debian package that owns a service executable when known to the package database.
Unit locations
Know where systemd configuration can come from
/etc/systemd/system/Local administrator units, overrides, and links often live here.
High-value location for custom services and persistence review. Preserve evidence before removing unfamiliar units.
/usr/lib/systemd/system/Common location for vendor-provided unit files on many modern systems.
Often package-managed. Identify package ownership before editing.
/lib/systemd/system/Common vendor-unit location on Ubuntu-family systems, sometimes related to the merged /usr layout.
Inspect the actual system because path layout can vary by version.
/etc/systemd/system/*.d/Drop-in override directories can modify unit behavior without editing the vendor file.
A service may behave differently from its base unit because of an override.
/run/systemd/system/Runtime units and transient configuration can appear here.
Content may be temporary and disappear after reboot.
Common required-service examples
Service value depends on the scenario
ssh / sshdRemote administration when explicitly required by the scenario.
Dependencies
Accounts, authentication, network interfaces, firewall rules, host keys, and SSH configuration.
NetworkManager or systemd-networkdNetwork interface and connectivity management depending on the image.
Dependencies
Disabling the active network manager can remove connectivity.
systemd-resolvedDNS resolution on systems using it.
Dependencies
Name resolution can fail if DNS handling is changed without understanding /etc/resolv.conf and the active resolver design.
apache2 / nginxRequired web hosting when the scenario includes an internal or public web application.
Dependencies
Ports, firewall, configuration, certificates, files, permissions, and application backends.
cronScheduled job execution on systems using cron.
Dependencies
Legitimate maintenance and application jobs may depend on it even while some cron entries require investigation.
rsyslogTraditional logging on systems where it is installed and active.
Dependencies
Logs may support troubleshooting or forensics. Do not disable logging casually.
Service classification
Required, unnecessary, misconfigured, suspicious, or unknown
Required
Evidence
Named by the scenario, supports a required application, or is a clear dependency of required functionality.
Response
Preserve it, secure its configuration, and verify the client-facing function.
Unnecessary
Evidence
Clearly prohibited, outside the intended role, or confirmed to expose functionality that the system does not need.
Response
Review dependencies, preserve evidence if relevant, then stop/disable or remove only when justified.
Misconfigured
Evidence
Required service with insecure permissions, wrong interface exposure, weak authentication, bad configuration, or repeated failures.
Response
Fix the specific configuration issue while preserving the service role.
Suspicious
Evidence
Unexpected executable path, unusual unit file, unexplained root privilege, odd persistence, unknown package origin, or evidence tying it to a security event.
Response
Preserve unit files, logs, process data, executable metadata, and dependencies before remediation.
Unknown
Evidence
Purpose or dependency relationship is not yet clear.
Response
Investigate package ownership, unit definition, logs, process tree, ports, and reverse dependencies.
Evidence-based service reasoning
Replace service-name guesses with dependency analysis
An unfamiliar service is running as root
Weak response
Stop and disable it immediately.
Better reasoning
Inspect the unit file, executable path, package ownership, logs, ports, scenario role, and reverse dependencies before deciding whether it is unnecessary or suspicious.
SSH is enabled
Weak response
Disable remote access because remote services are risky.
Better reasoning
Check whether SSH is required. If it is, harden authentication and firewall exposure while preserving authorized access.
A service is failed
Weak response
Disable it because it is broken.
Better reasoning
Determine whether the service is required, inspect journal logs, identify the failure cause, and decide whether to repair or remove it.
A service is disabled
Weak response
It cannot be running.
Better reasoning
Check current state and triggers. A disabled unit can still be started manually or by another unit.
A unit file exists in /etc/systemd/system
Weak response
Custom unit means malware.
Better reasoning
Local administrators legitimately create custom units. Investigate who created it, what it runs, what package/application it supports, and whether the scenario requires it.
A web service listens on all interfaces
Weak response
Disable the whole service.
Better reasoning
If the web application is required, review bind addresses, firewall rules, TLS, authentication, and application configuration to reduce exposure while preserving service.
Dependency review
Ask what starts it and what needs it
What starts this service?
systemctl list-dependencies service-nameWhat depends on this service?
systemctl list-dependencies --reverse service-nameWhat unit file and overrides are active?
systemctl cat service-nameWhat executable does it run?
systemctl show service-name -p ExecStartWhat user runs it?
systemctl show service-name -p User -p GroupWhat network ports are associated with it?
ss -tulpnService evidence
Preserve context before cleanup
Unit definition
systemctl cat service-nameShows vendor unit content plus drop-ins that may modify behavior.
Service properties
systemctl show service-nameShows detailed properties such as executable, user, restart behavior, dependencies, and state.
Service logs
journalctl -u service-nameProvides service-specific journal evidence when retained.
Process tree
ps -ef --forestHelps correlate services with child processes and runtime activity.
Network exposure
ss -tulpnShows listening sockets and associated processes where permissions allow.
Package ownership
dpkg -S /path/to/executableIdentifies the package responsible for a service binary when package-managed.
Controlled service actions
Change state only after role and dependency review
Goal
Start a required service
Example
sudo systemctl start service-nameCaution
Use after understanding why the service should be active and what configuration it will load.
Goal
Stop a confirmed unnecessary service
Example
sudo systemctl stop service-nameCaution
Stopping affects the current session only. Verify no required dependency or client function breaks.
Goal
Enable a required service
Example
sudo systemctl enable service-nameCaution
Enablement affects future startup behavior. Confirm automatic startup is actually desired.
Goal
Disable unnecessary automatic startup
Example
sudo systemctl disable service-nameCaution
Disabling does not necessarily stop a currently running service or prevent manual activation.
Goal
Restart after a validated configuration change
Example
sudo systemctl restart service-nameCaution
A restart can interrupt users and may fail if the new configuration has errors. Validate configuration first when the service supports it.
Goal
Reload configuration without full restart
Example
sudo systemctl reload service-nameCaution
Only works when the service supports reload behavior. Check service documentation and status.
High-risk shortcuts
Service changes that deserve extra caution
systemctl disable --now everything unfamiliarCombines current shutdown and future disablement without understanding dependencies or system role.
Masking a service as the first responseMasking strongly blocks startup and can break dependencies or troubleshooting workflows.
Deleting unit files before preserving evidenceCustom unit files can answer forensic questions about persistence, executable paths, users, and command lines.
Restarting repeatedly without reading logsRepeated restarts can hide the real failure pattern, alter evidence, and waste time.
Disabling logging servicesCan reduce visibility and destroy the team's ability to troubleshoot or answer forensic questions.
Assuming package removal is the same as service disablementPackage removal changes installed software and may affect configuration and dependencies beyond one running service.
Verification
Prove service state and real functionality
Service state
Re-run systemctl status and confirm active/inactive state matches the intended result.
Startup state
Check systemctl is-enabled when future startup behavior matters.
Logs
Review journalctl for new errors or warnings after the change.
Network exposure
Re-run ss -tulpn when the service listens on the network.
Dependency health
Confirm related required services remain active and functional.
Client function
Test the actual SSH, web, DNS, application, or other scenario-required function from the appropriate perspective.
Active is not the same as healthy
A service can show active while the application is misconfigured, blocked by the firewall, or failing from the client perspective. Test the actual scenario-required function.
Defensive lab
Audit fictional Linux services without breaking dependencies
Read the fictional scenario: SSH and an internal web application are required; printing is not required; logging must remain available; an unfamiliar root-owned service named sync-helper.service requires investigation.
Inventory running, enabled, failed, and masked services using systemctl without changing anything.
Inspect ssh.service, the web service, the printing service, rsyslog if present, and sync-helper.service using status, cat, show, and journalctl.
Check reverse dependencies before stopping or disabling any service.
Classify the printing service as unnecessary only after confirming no scenario requirement depends on it. Classify sync-helper.service as Investigate until executable, package, logs, and persistence evidence are understood.
Apply only the justified service-state change and preserve the unfamiliar unit's evidence before remediation.
Verify SSH, the internal web application, logging, service states, and listening ports after the change.
Lesson checklist
Before moving on
I understand the difference between a service, unit, process, and package.
I know enabled and running describe different things.
I can identify active, inactive, failed, disabled, static, and masked unit states.
I use systemctl status, cat, show, and dependency commands before changing a service.
I inspect journalctl evidence instead of repeatedly restarting failed services.
I check listening ports when a service exposes network functionality.
I do not disable unfamiliar services without checking package ownership and dependencies.
I preserve custom unit files and logs when a service may matter to forensics.
I make narrow service changes instead of bulk disabling.
I verify both systemd state and the real client-facing function after changes.
Reflection
Check your reasoning
Question 1
Why is a disabled service not necessarily stopped?
Question 2
Why can a static service still be important?
Question 3
What evidence would you collect before disabling an unfamiliar root-owned service?
Question 4
Why should reverse dependencies be checked before changing service state?
Question 5
How can a required network service be hardened without disabling it?
Question 6
Why is systemctl status alone not enough to prove a required service is functioning correctly?
Carry this forward
A service should be judged by its role, dependencies, exposure, configuration, and evidence. Inspect first, change state deliberately, and prove the required function still works.