Cisco & Networking · Lesson 16
Network Troubleshooting Workflow
Troubleshoot Cisco and Packet Tracer networks by following the evidence: requirement, packet path, layer, hypothesis, controlled change, and end-to-end verification.
Strong troubleshooting is not about running the largest number of commands. It is about choosing the next command that can eliminate the most incorrect possibilities.
Lesson Snapshot
Learning objectives
Diagnose first, configure second
Use an evidence-based process to isolate network failures before changing configuration.
Separate physical, Layer 2, Layer 3, policy, service, and application problems.
Choose Cisco IOS inspection commands that match the symptom and expected packet path.
Use ping, traceroute, MAC, ARP, routing, ACL, DHCP, DNS, NAT, and logging evidence together.
Recognize partial reachability, one-way traffic, and other symptoms that can point to specific layers.
Verify the fix end to end, document the exact change, and save only after the result is proven.
Core idea
Troubleshooting is controlled elimination
Every successful test removes possibilities. Use working evidence to shrink the fault domain until one change is justified.
Competition warning
Broad workarounds hide root cause
permit any any, trunk allow all, random static routes, clearing tables, or disabling controls can restore traffic while breaking required security or hiding the real fault.
Troubleshooting principles
Six rules for a controlled investigation
Start with the requirement
Define what should work before deciding what is broken. Troubleshooting without a target encourages random changes.
Follow the packet path
Trace traffic from source to destination and then verify the return path instead of checking devices in a random order.
Use working evidence
What still works is just as important as what fails because each successful test eliminates possible causes.
Inspect before modifying
Prefer read-only show commands, controlled pings, traceroute, and endpoint checks before configuration commands.
Change one thing
Once evidence identifies the cause, modify only the affected interface, route, ACL, VLAN, service, or setting.
Verify the real service
A successful ping does not prove DNS, SSH, HTTPS, DHCP, NAT, or an application. Test the actual requirement.
Symptom definition
Ask these questions before opening configuration mode
Who is affected?
One endpoint, one access port, one VLAN, one subnet, one site, one application, or everyone?
What still works?
Same-subnet traffic, gateway access, raw IP connectivity, DNS, remote routing, or one specific service?
When did it begin?
After a configuration change, after reload, intermittently, or from initial deployment?
What is the expected path?
Source port, VLAN, trunk, gateway, routed hops, ACLs, NAT boundary, destination VLAN, and service.
What changed recently?
Interface state, VLANs, routes, ACLs, port security, DHCP, DNS, NAT, credentials, or topology.
What proves the failure?
Interface state, MAC learning, route table, ACL hits, DHCP lease, DNS answer, logs, or application test.
Layered troubleshooting
Move through the network in dependency order
You do not have to start at Layer 1 every time. Start at the lowest layer the existing evidence has not already proven.
Physical & Interface State
Is the link actually operational?
Check cable or simulated connection, interface status, administrative state, line protocol, speed, duplex, and error counters.
An administratively down interface points to configuration. A down/down interface points toward the physical or peer side.
Increasing CRC or input errors deserve investigation instead of being dismissed as an old historical counter.
show interfaces statusshow interfacesshow ip interface briefshow running-config interface <interface>Layer 2 Switching
Can the frame move through the correct VLAN?
Confirm access VLAN membership, trunk state, allowed VLANs, MAC learning, and spanning-tree forwarding.
Same-subnet traffic normally stays inside the Layer 2 domain and should not require a router.
A working trunk for three VLANs does not prove a fourth VLAN is allowed across it.
show vlan briefshow interfaces trunkshow interfaces <interface> switchportshow mac address-tableshow spanning-treeIPv4 Addressing
Do endpoint and gateway addresses describe the same network design?
Check source IP, subnet mask, gateway, router interface address, and overlapping or incorrect prefixes.
A wrong mask can make a host treat a remote destination as local or a local destination as remote.
ARP evidence can help show whether local IPv4-to-MAC resolution is occurring.
show ip interface briefshow running-config | section interfaceshow arpRouting
Does every Layer 3 hop know where to send the packet and the reply?
Inspect connected, static, and dynamically learned routes and verify the next hop is reachable.
A route to the destination is only half the conversation; the return route must also exist.
Longest prefix match, administrative distance, and routing-protocol state can explain why traffic takes an unexpected path.
show ip routeshow ip protocolsping <next-hop>traceroute <destination>Policy & Security
Is required traffic being intentionally or accidentally filtered?
Review ACL order, interface direction, hit counters, port-security state, and management restrictions.
Do not remove a security control just because traffic fails. Determine whether the scenario expects that traffic to be permitted.
One-way traffic frequently points toward reverse ACL, return-route, NAT, or endpoint-policy problems.
show access-listsshow ip access-listsshow ip interface <interface>show port-security interface <interface>Network Services
Are DHCP, DNS, and NAT doing their own jobs correctly?
A client may have routing but the wrong DNS server, or correct DNS while NAT is missing at the outside edge.
Test each service independently instead of treating them as one large feature.
An empty NAT table is not proof of broken NAT if no matching traffic has crossed the device.
show ip dhcp poolshow ip dhcp bindingshow ip nat translationsshow ip nat statisticsApplication
Does the actual service respond?
ICMP reachability does not prove SSH, HTTPS, DNS, or another application is healthy.
Compare access by IP and hostname to separate DNS from application reachability.
Verify the destination server process, transport port, endpoint firewall, and required service state.
Test the required applicationCompare IP vs hostname accessVerify the service and transport portFirst five minutes
Build context before chasing commands
Read the scenario
Mark required services, prohibited traffic, management paths, and anything that must remain enabled.
Map the topology
Label switches, routers, interfaces, VLANs, subnets, gateways, servers, and expected traffic paths.
Reproduce the symptom
Record exactly what fails and what still succeeds before changing anything.
Collect a baseline
Use the relevant show commands and preserve useful counters, tables, and logs.
Shrink the fault domain
Decide whether the evidence points to one host, one link, one VLAN, one routed hop, one policy, or one service.
Baseline commands
Ten commands that answer different questions
show ip interface briefFast Layer 3 interface addressing and up/down state.
show interfaces statusSwitchport status, VLAN, speed, and duplex where supported.
show running-config | section interfaceInterface addressing, shutdown, VLAN, and related configuration.
show vlan briefVLAN existence and access-port membership.
show interfaces trunkTrunk status and VLAN carriage.
show mac address-tableLayer 2 source-MAC learning.
show arpIPv4-to-MAC mappings on Layer 3 devices.
show ip routeConnected, static, and dynamic Layer 3 paths.
show access-listsPolicy entries and counters where supported.
show loggingRecent operational and security events.
Ping ladder
Test from near to far
A controlled sequence of tests can show exactly where reachability stops.
Same-subnet peer
What success supports
Tests local Layer 2 and ARP behavior without normal router forwarding.
What it does not prove
Remote routing or application availability.
Default gateway
What success supports
Tests endpoint addressing, local VLAN path, ARP, and gateway availability.
What it does not prove
A route beyond the gateway.
Next-hop router
What success supports
Tests the directly connected routed segment from the router.
What it does not prove
The rest of the path.
Remote gateway
What success supports
Shows multiple routed hops may be functioning.
What it does not prove
The final endpoint or application.
Destination IP
What success supports
Shows IP reachability if ICMP is permitted.
What it does not prove
DNS or a TCP/UDP service.
Required service
What success supports
Tests the thing the scenario actually needs.
What it does not prove
Every unrelated service or path.
ICMP is evidence, not absolute proof
Some devices or policies intentionally suppress ICMP. Interpret ping results with routing, ACL, and application evidence.
Traceroute reasoning
Use hop-by-hop clues without overtrusting them
Stops near the first hop
Investigate local gateway, first routed link, nearby policy, or interface state.
Reaches several hops then stops
Focus near the last responding router, downstream routing, policy, or destination network.
Uses an unexpected path
Check longest-prefix match, static routes, dynamic protocol preference, and summaries.
Shows timeouts but application works
Do not declare failure. Devices or policies may suppress traceroute-related responses.
Forward route looks correct but session fails
Check return routing, reverse ACLs, NAT state, or destination policy.
Layer 2 patterns
Use switching evidence to narrow the fault
Access port up but no endpoint MAC
Generate endpoint traffic, confirm correct physical port, VLAN, port security, cable, and NIC.
Endpoint MAC on unexpected port
Check patching, endpoint movement, topology documentation, or possible Layer 2 loop.
One VLAN fails across a working trunk
Inspect VLAN existence, allowed list, native VLAN expectations, and STP state.
Same-subnet peers cannot communicate
Check access VLAN, masks, ARP, MAC learning, port security, and endpoint firewall.
Many MAC addresses on one interface
This may be normal on an uplink or downstream-switch connection.
CRC counters are rising
Inspect media, peer interface, connector quality, and speed/duplex behavior.
Layer 3 patterns
Use routing evidence to explain the path
Host reaches gateway but not remote subnet
Check route table, next hop, ACL, remote route, and destination subnet.
Router cannot ping next hop
Fix the directly connected segment before adding remote routes.
Connected route is missing
Inspect interface address, shutdown state, line protocol, and subnet overlap.
Static route uses unreachable next hop
Repair local reachability to the neighbor first.
Dynamic route disappeared
Check adjacency, protocol participation, passive-interface, area/AS, and filtering.
Default route carries internal traffic
Look for a missing more-specific route or withdrawn dynamic route.
Policy patterns
Correct routing can still be filtered
Ping works but HTTPS fails
Inspect TCP/443 policy, server service, DNS if name-based, and endpoint firewall.
ACL counter rises during the failed test
That ACE is matching the flow; compare its action with the scenario requirement.
Authorized PC fails after port-security change
Check secure MAC, maximum, violation mode, sticky state, and err-disabled status.
SSH works only from Management VLAN
This may be an intentional VTY source restriction.
ACL exists but does not affect traffic
Check whether it is applied, on the correct interface, in the correct direction, and on the actual path.
Traffic works only one direction
Inspect reverse ACL, return route, NAT, and endpoint policy.
Service patterns
DHCP, DNS, and NAT leave different evidence
Client gets 169.254.x.x
Check VLAN, DHCP server, relay, pool, gateway interface, and server reachability.
Client reaches server by IP but not name
Check DNS resolver address, reachability, record correctness, and UDP/TCP 53 policy.
No NAT translations appear
Generate matching traffic, then inspect inside/outside roles, NAT ACL, routing, and outside link.
DHCP works for one VLAN only
Compare per-VLAN helper, scope, gateway, and routing.
DNS resolves but application fails
Check service process, transport port, ACL, and destination firewall.
Translation exists but reply fails
Check upstream route, return traffic, ACL, destination service, and NAT state.
Fault-domain clues
The number of affected users is evidence
One endpoint
Endpoint IP, mask, gateway, NIC, host firewall, access port, port security.
One access port
Physical state, VLAN membership, port security, interface configuration.
One VLAN
SVI/subinterface, trunk carriage, DHCP relay, VLAN ACL, gateway addressing.
One remote subnet
Route, return route, ACL, remote gateway, routing advertisement.
One application
DNS, protocol/port, ACL, service state, endpoint firewall.
Everything
Core link, major trunk, default route, gateway, broad ACL, device availability.
Controlled changes
Turn a hypothesis into one measurable change
Write the hypothesis
Example: VLAN 20 fails because the switch trunk does not permit VLAN 20.
Collect confirming evidence
Use show interfaces trunk and compare the failed VLAN to working VLANs.
Record the original state
Capture the relevant interface, route, ACL, or service configuration before editing.
Make one narrow change
Modify only the setting the evidence supports.
Retest the symptom
Repeat the exact test that originally failed.
Check unaffected services
Confirm management, neighboring VLANs, and required applications still work.
Document the result
Record cause, change, verification, and rollback.
Save deliberately
Persist the configuration only after the final state is proven.
Common mistakes
Actions that make the network harder to understand
Changing before reproducing
Destroys the original evidence and can create a second failure.
Assuming the first warning is the cause
A visible warning can be unrelated to the actual packet path.
Testing only one direction
Misses return-route, reverse-policy, and asymmetric failures.
Treating ping as application proof
ICMP success does not prove DNS, SSH, HTTPS, or another service.
Clearing evidence too early
MAC, ARP, ACL, NAT, and log state can disappear before it is understood.
Using broad workarounds
permit any any, allow all VLANs, or default routes can hide the root cause.
Ignoring working behavior
A successful test can eliminate entire layers from suspicion.
Saving immediately
An unverified workaround becomes persistent and harder to undo.
Command map
Pick commands based on the question you are asking
Link/interface
show interfaces status; show interfaces; show ip interface brief
VLAN membership
show vlan brief; show interfaces <interface> switchport
Trunking
show interfaces trunk; show running-config interface <interface>
MAC learning
show mac address-table; show mac address-table interface <interface>
ARP
show arp
Routing
show ip route; show ip protocols; ping; traceroute
ACL policy
show access-lists; show ip access-lists; show ip interface <interface>
Port security
show port-security interface <interface>; show port-security address
DHCP
show ip dhcp pool; show ip dhcp binding
NAT
show ip nat translations; show ip nat statistics
Management
show users; show ip ssh; show login; show running-config | section line
Events
show logging; show clock
Evidence-based reasoning
Let successful tests remove bad hypotheses
A user cannot reach a server, but the gateway responds
Weak response
Reconfigure the access port immediately.
Better reasoning
The local path is at least partially working. Move outward to routing, policy, destination gateway, and server state.
One VLAN fails across a trunk while three others work
Weak response
Replace the entire trunk configuration.
Better reasoning
Compare the failed VLAN against VLAN existence, allowed list, native VLAN expectations, and STP state.
Server IP ping succeeds but HTTPS fails
Weak response
Add another static route.
Better reasoning
IP reachability works. Check TCP/443 policy, server process, endpoint firewall, and DNS if the test uses a hostname.
No destination route exists and ACL counters stay at zero
Weak response
Edit the ACL.
Better reasoning
Routing fails before the expected policy point. Fix route learning or static routing first.
Port-security counter rises when a second device appears
Weak response
Clear the counter and continue.
Better reasoning
The counter confirms policy is matching. Determine whether the second source MAC is authorized before changing the limit.
Hostname fails but raw IP works
Weak response
Replace router configuration.
Better reasoning
The IP path works. Check resolver configuration, DNS server reachability, record accuracy, and DNS policy.
Competition scenarios
Choose the next evidence, not the next random command
Scenario A - One PC cannot reach its gateway
Focus on that host, its access port, IP settings, MAC learning, cable/NIC state, and port security before touching the gateway.
Scenario B - Entire VLAN reaches gateway but nothing remote
Inspect the VLAN gateway route table, ACLs, routing-protocol participation, and return routes.
Scenario C - Server works by IP but not hostname
Compare DHCP DNS option, resolver address, DNS reachability, DNS record, and port 53 policy.
Scenario D - SSH stops after a management ACL change
Use console access, inspect VTY access-class and source subnet, correct only the management rule, and verify before saving.
Scenario E - NAT works for Staff but not Students
Compare both source subnets to the NAT classification ACL and verify Student routing toward the same outside path.
Scenario F - Remote gateway responds but server does not
Continue to destination VLAN, endpoint addressing, policy, service state, and return path instead of changing upstream routing.
Verification matrix
Prove the complete path, not just one hop
Source endpoint
Correct IP, mask, gateway, DNS, physical link, and access VLAN.
Layer 2
MAC learning, VLAN membership, trunk carriage, and STP forwarding.
Gateway
The local gateway is operational and correctly addressed.
Routing
Every required hop has a valid destination route and reachable next hop.
Policy
ACLs and security controls permit required traffic while preserving required restrictions.
Services
DHCP, DNS, NAT, and management functions behave as required.
Destination
The intended endpoint and application respond correctly.
Return path
Replies return through routing, policy, and translation successfully.
Defensive lab
Troubleshoot one hidden fault in a fictional Packet Tracer network
Build or open a fictional topology with Staff, Students, Servers, Management, two switches, two routers, and a simulated outside network.
Create a baseline of interface states, VLANs, trunks, gateways, routes, ACLs, DHCP, DNS, NAT, and management reachability.
Verify one known-good internal path and one authorized path toward the simulated outside network.
Introduce exactly one fault: wrong access VLAN, missing trunk VLAN, wrong mask, missing route, ACL deny, missing helper, wrong NAT match, or disabled service.
Start from the user-visible symptom and narrow the fault domain with read-only evidence instead of looking directly at the fault.
Write one clear hypothesis before entering configuration mode.
Correct only the confirmed fault and retest the failed requirement plus at least one unrelated known-good service.
Document symptom, evidence, root cause, change, verification, and rollback note before saving.
Troubleshooting log
Record the evidence trail
Symptom
Student PC cannot reach HTTPS server 192.168.30.10.
What still works
Student PC reaches its 192.168.20.1 gateway.
Expected path
PC -> access switch -> trunk -> router -> Server VLAN.
Evidence
Route exists; ACL deny TCP/443 counter increments during test.
Root cause
Incorrect ACE blocks required HTTPS.
Change
Corrected only the affected ACL sequence.
Verification
HTTPS succeeds, prohibited SSH remains blocked, other VLANs unchanged.
Rollback
Restore the previous ACE if unintended behavior appears.
Lesson checklist
Before moving on
I define the expected behavior before troubleshooting.
I record what still works before changing configuration.
I trace the packet from source to destination and back.
I separate physical, Layer 2, Layer 3, policy, service, and application evidence.
I use read-only show commands before configuration commands.
I know same-subnet traffic normally does not require router forwarding.
I check routing before blaming ACLs, DNS, or NAT.
I compare IP-based and name-based tests when DNS may be involved.
I preserve counters, tables, and logs long enough to use them as evidence.
I make the smallest justified change.
I retest both the failed requirement and unaffected services.
I document and save only after the fix is proven.
Reflection
Check your reasoning
Question 1
Why is knowing what still works as important as knowing what fails?
Question 2
How can one successful gateway ping eliminate several possible fault areas?
Question 3
Why must the return path be checked even when the forward route looks correct?
Question 4
Why is permit ip any any a poor troubleshooting shortcut?
Question 5
How can MAC tables, ARP tables, ACL counters, NAT translations, and logs act as evidence?
Question 6
What makes a troubleshooting change safe enough to save?
Carry this forward
Define the requirement, identify what still works, trace the packet path, collect evidence, change one justified item, verify forward and return behavior, and document before saving.