Layer By Layer Troubleshooting Approach

Category: Network Troubleshooting
Type: Network Tools & Commands
Generated on: 2025-07-11 01:35:09
For: Network Engineering, Administration & Technical Interviews

This cheatsheet provides a practical guide to network troubleshooting using a layer-by-layer approach. It covers essential tools and commands for diagnosing network issues, from the physical layer to the application layer.

1. Tool Overview

Tool	Layer(s)	Description	When to Use
`ping`	3 (ICMP)	Verifies basic IP connectivity to a host. Sends ICMP Echo Request packets and listens for replies.	Quickly check if a host is reachable. Simple network availability testing. Troubleshooting routing issues.
`traceroute`/`tracert`	3 (TTL)	Maps the path packets take to reach a destination by incrementing the Time-To-Live (TTL) field.	Identify routing loops, determine network latency between hops, and diagnose network outages.
`ip` (Linux) / `ipconfig` (Windows)	1, 2, 3	Displays and configures network interfaces, IP addresses, routing tables, and more.	View interface status, IP configuration, routing information. Configure network settings (IP address, gateway, DNS). Troubleshoot IP address conflicts.
`netstat`	4	Displays active network connections, listening ports, routing tables, and network statistics.	Identify which processes are listening on specific ports. Check for established connections to a server. Monitor network traffic and identify potential bottlenecks.
`ss` (Linux)	4	A modern replacement for `netstat`. Provides more detailed socket statistics and filtering capabilities.	Similar to `netstat`, but more efficient and scalable. Analyze socket states, connection details, and network performance.
`tcpdump`/`Wireshark`	2, 3, 4, 7	Captures and analyzes network traffic. Provides detailed packet information.	Deeply inspect network traffic, troubleshoot protocol issues, analyze application performance, and identify security threats.
`nslookup`/`dig`	7 (DNS)	Queries DNS servers to resolve domain names to IP addresses and vice versa.	Troubleshoot DNS resolution problems, verify DNS records, and diagnose DNS server issues.
`curl`/`wget`	7 (HTTP)	Transfers data from or to a server using various protocols (HTTP, HTTPS, FTP, etc.).	Test web server connectivity, download files, and interact with web APIs.
`mtr` (Linux)	3 (ICMP/UDP)	Combines the functionality of `ping` and `traceroute` to provide real-time network path analysis.	Identify network bottlenecks and latency issues over time. Pinpoint where packet loss is occurring along the path. Useful for diagnosing intermittent network problems.
`arp`	2	Displays and manages the Address Resolution Protocol (ARP) cache.	Troubleshoot MAC address resolution problems, identify IP address conflicts, and diagnose network spoofing attacks.
`ethtool` (Linux)	1	Displays and configures Ethernet interface parameters.	Troubleshoot physical layer issues, verify link speed and duplex settings, and diagnose hardware problems.
`tshark` (CLI Wireshark)	2, 3, 4, 7	Command-line version of Wireshark for capturing and analyzing network traffic.	Automate packet capture and analysis, perform batch processing, and integrate with scripting environments.

2. Basic Syntax

ping:

ping <destination>
ping <destination> -c <count>  # Limit the number of pings
ping <destination> -i <interval> # Set the ping interval in seconds (caution: use sparingly and responsibly)

traceroute / tracert:

traceroute <destination>  # Linux/macOS
tracert <destination>     # Windows
traceroute -n <destination> # Disable hostname resolution (faster)

ip (Linux):

ip addr show <interface>     # Show interface IP address
ip route show                  # Show routing table
ip link show <interface>     # Show interface link status

ipconfig (Windows):

ipconfig /all               # Show all network configuration
ipconfig /release <interface> # Release IP address
ipconfig /renew <interface>  # Renew IP address

netstat:

netstat -an                 # Show all active connections and listening ports
netstat -tulnp               # Show TCP, UDP, listening ports, and process IDs (Linux)
netstat -b                  # Show executable involved in creating each connection (Windows - requires admin privileges)

ss (Linux):

ss -tulnp                    # Show TCP, UDP, listening ports, and process IDs
ss -s                       # Show socket statistics
ss -o state established '( dport = :ssh )' # Show established SSH connections

tcpdump:

tcpdump -i <interface>      # Capture traffic on a specific interface
tcpdump -i <interface> port <port_number> # Capture traffic on a specific port
tcpdump -i <interface> host <host_ip>   # Capture traffic to/from a specific host
tcpdump -i <interface> -w <filename.pcap> # Save captured traffic to a file

nslookup:

nslookup <domain>          # Resolve domain name to IP address
nslookup <domain> <dns_server> # Use a specific DNS server
nslookup -type=MX <domain>  # Query for MX records

dig:

dig <domain>             # Resolve domain name to IP address
dig <domain> MX          # Query for MX records
dig <domain> @<dns_server> # Use a specific DNS server
dig -x <ip_address>      # Reverse DNS lookup

curl:

curl <URL>                # Fetch the content of a URL
curl -I <URL>             # Get HTTP headers only
curl -v <URL>             # Verbose output (shows request/response details)
curl -X POST -d "data=value" <URL> # Send POST request

wget:

wget <URL>                # Download a file from a URL
wget -O <filename> <URL>   # Download and save to a specific filename

mtr (Linux):

mtr <destination>            # Run MTR to the destination
mtr -n <destination>         # Disable hostname resolution
mtr -r <destination>         # Report mode (just print the final results)

arp:

arp -a                       # Display ARP cache
arp -d <ip_address>        # Delete ARP entry (requires root/admin)
arp -s <ip_address> <mac_address> # Add static ARP entry (requires root/admin)

ethtool (Linux):

ethtool <interface>          # Display interface settings
ethtool -s <interface> speed <speed> duplex <duplex> autoneg off # Configure speed and duplex (requires root)

tshark:

tshark -i <interface> -w <filename.pcap>  # Capture traffic to a pcap file
tshark -r <filename.pcap> -T fields -e ip.src -e ip.dst -e tcp.port # Read a pcap file and extract specific fields
tshark -i <interface> "tcp port 80"  # Capture only HTTP traffic

3. Practical Examples

Example 1: Verify basic connectivity to Google:

ping google.com

Expected Output:

PING google.com (142.250.184.142) 56(84) bytes of data.
64 bytes from fra16s42-in-f14.1e100.net (142.250.184.142): icmp_seq=1 ttl=117 time=7.84 ms
64 bytes from fra16s42-in-f14.1e100.net (142.250.184.142): icmp_seq=2 ttl=117 time=7.79 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 7.790/7.815/7.840/0.025 ms

Example 2: Trace the route to a remote server:

traceroute example.com

(Example Output - will vary based on your network)

traceroute to example.com (93.184.216.34), 30 hops max, 60 byte packets
 1  gateway (192.168.1.1)  1.234 ms  1.345 ms  1.456 ms
 2  10.0.0.1 (10.0.0.1)  5.678 ms  5.789 ms  5.890 ms
 3  router1.isp.net (203.0.113.1)  12.345 ms  12.456 ms  12.567 ms
 4  ...

Example 3: View IP address and interface information (Linux):

ip addr show eth0

(Example Output)

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.100/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::211:22ff:fe33:4455/64 scope link
       valid_lft forever preferred_lft forever

Example 4: Check for listening ports (Linux):

netstat -tulnp | grep 80

(Example Output - If a process is listening on port 80)

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1234/apache2

Example 5: Capture HTTP traffic on port 80 (using tcpdump):
Terminal window
```
sudo tcpdump -i eth0 port 80 -w http.pcap
```
This will capture all HTTP traffic on interface eth0 and save it to a file named http.pcap. You can then analyze this file with Wireshark.
Example 6: Resolve a domain name to an IP address using dig and a specific DNS server:
Terminal window
```
dig example.com @8.8.8.8
```
Example 7: Download a file using wget:
Terminal window
```
wget https://example.com/file.txt
```
Example 8: Use mtr to check network latency and packet loss to a server:
Terminal window
```
mtr example.com
```
This will show a real-time display of the network path to example.com, including latency and packet loss at each hop.

4. Common Options

Tool	Option	Description
`ping`	`-c <count>`	Send only `<count>` number of packets.
	`-i <interval>`	Wait `<interval>` seconds between sending each packet. Use with caution.
	`-s <size>`	Specify packet size.
`traceroute`/`tracert`	`-n`	Do not resolve hostnames.
`ip`	`addr show`	Show IP address information for an interface.
	`route show`	Show the routing table.
`ipconfig`	`/all`	Show all network configuration details.
	`/flushdns`	Purge the DNS Resolver cache.
`netstat`	`-an`	Show all active connections and listening ports.
	`-tulnp` (Linux)	Show TCP, UDP, listening ports, and process IDs.
	`-b` (Windows)	Show the executable involved in creating each connection (requires admin privileges).
`ss`	`-tulnp`	Show TCP, UDP, listening ports, and process IDs.
	`-s`	Show socket statistics.
`tcpdump`	`-i <interface>`	Specify the interface to capture traffic on.
	`port <port>`	Filter traffic by port number.
	`host <host>`	Filter traffic by host IP address or hostname.
	`-w <file>`	Save captured traffic to a file (pcap format).
`nslookup`	`-type=<type>`	Query for a specific DNS record type (e.g., A, MX, CNAME).
`dig`	`-x <ip>`	Perform a reverse DNS lookup.
	`@<dns_server>`	Specify a DNS server to use for the query.
`curl`	`-I`	Get HTTP headers only.
	`-v`	Verbose output (shows request/response details).
	`-X <method>`	Specify the HTTP method (e.g., POST, PUT, DELETE).
	`-d <data>`	Send data in a POST request.
`wget`	`-O <file>`	Save the downloaded file to a specific filename.
`mtr`	`-n`	Disable hostname resolution.
	`-r`	Report mode (just print the final results).
`arp`	`-a`	Display ARP cache.
`ethtool`	`-s`	Set interface parameters (speed, duplex, autoneg). Requires root/admin privileges.
`tshark`	`-i <interface>`	Specify the interface to capture traffic on.
	`-r <file.pcap>`	Read traffic from a pcap file.
	`-T fields`	Output specified fields.
	`-e <field>`	Specify a field to extract (e.g., ip.src, ip.dst, tcp.port).

5. Advanced Usage

tcpdump with BPF filters: Use Berkeley Packet Filter (BPF) syntax for complex filtering.

sudo tcpdump -i eth0 'tcp port 80 and host 192.168.1.100 and tcp[((tcp[12:1] & 0xf0) >> 4)*4:4] = 0x47455420' # Capture GET requests from 192.168.1.100 on port 80

Combining tcpdump and tshark for real-time analysis:
Terminal window
```
sudo tcpdump -i eth0 -w - | tshark -r - -T fields -e ip.src -e ip.dst -e tcp.port
```
This pipes the output of tcpdump directly to tshark for real-time analysis and display of IP addresses and port numbers.

Using ss to filter by socket state:

ss -o state syn-sent '( dport = :80 or dport = :443 )' # Show sockets in SYN-SENT state to ports 80 or 443

Using curl to test API endpoints with authentication:

curl -u username:password -X GET https://api.example.com/resource

Using mtr to diagnose intermittent packet loss: Run mtr for an extended period and analyze the results to identify hops with consistent or intermittent packet loss.

6. Troubleshooting Scenarios

Scenario	Likely Cause(s)	Troubleshooting Steps
Cannot ping a host	Network cable disconnected, incorrect IP address, firewall blocking ICMP, routing issue, host down.	1. Verify physical connection. 2. Check IP address configuration using `ip addr show` (Linux) or `ipconfig` (Windows). 3. Ping the gateway. 4. Use `traceroute`/`tracert` to identify where the path breaks down. 5. Check firewall rules. 6. Verify the target host is powered on and responding.
Slow network performance	Network congestion, high latency, duplex mismatch, faulty network hardware, application issues.	1. Use `ping` to measure latency to various destinations. 2. Use `traceroute`/`tracert` to identify high-latency hops. 3. Check interface statistics for errors using `ip -s link show` (Linux) or performance monitor (Windows). 4. Check CPU and memory usage on the affected hosts. 5. Use `tcpdump`/Wireshark to analyze network traffic and identify bottlenecks. 6. Check duplex settings with `ethtool` (Linux).
DNS resolution failure	Incorrect DNS server configuration, DNS server down, firewall blocking DNS traffic, domain name not registered.	1. Verify DNS server configuration using `ipconfig /all` (Windows) or `/etc/resolv.conf` (Linux). 2. Use `nslookup` or `dig` to query DNS servers. 3. Try a public DNS server like 8.8.8.8. 4. Check firewall rules for DNS traffic (port 53).
Cannot connect to a specific port	Firewall blocking the port, service not running on the target host, incorrect port number, network connectivity issues.	1. Verify service is running on the target host using `netstat` or `ss`. 2. Check firewall rules on both the client and server. 3. Use `tcpdump` to see if traffic is reaching the target host on the specified port. 4. Test connectivity using `telnet <host> <port>` or `nc -vz <host> <port>`.
Website is slow or unavailable	Web server down, network latency, DNS resolution issues, application problems, CDN issues.	1. Ping the web server to check basic connectivity. 2. Use `traceroute`/`tracert` to identify network latency. 3. Check DNS resolution using `nslookup` or `dig`. 4. Use `curl` or `wget` to test HTTP connectivity and measure response times. 5. Check web server logs for errors. 6. Check CDN status if applicable.
Intermittent network connectivity problems	Faulty network hardware, congested network links, wireless interference, routing loops, temporary DNS issues.	1. Use `mtr` to monitor network path and identify intermittent packet loss. 2. Check network hardware (cables, switches, routers) for errors. 3. Analyze wireless signal strength and interference. 4. Review routing tables for loops. 5. Monitor DNS server performance. 6. Check system logs for hardware errors.
IP Address Conflict	Two or more devices configured with the same IP address.	1. Identify the devices with the conflicting IP address. 2. Examine the ARP cache on affected devices to identify the MAC addresses associated with the conflicting IP address. 3. Reconfigure one of the devices with a unique IP address, either manually or through DHCP. 4. Consider implementing DHCP reservations to prevent future conflicts.
Duplex Mismatch	One device configured for full duplex while the other is configured for half duplex.	1. Check the interface configuration on both devices using `ethtool` (Linux) or interface settings in Windows. 2. Ensure both devices are configured for the same duplex setting, preferably auto-negotiation. 3. If auto-negotiation is not possible, manually configure both devices for full duplex.
High CPU Usage on Network Device	Excessive traffic, routing loops, misconfigured firewall rules, malware.	1. Identify the process or processes consuming the most CPU resources. 2. Analyze network traffic using `tcpdump` or Wireshark to identify the source and type of traffic. 3. Review routing tables and firewall rules for misconfigurations. 4. Scan the device for malware. 5. Consider upgrading the device’s hardware if it is consistently overloaded.
High Memory Usage on Network Device	Large routing tables, numerous active connections, memory leaks, buffer overflows.	1. Identify the process or processes consuming the most memory resources. 2. Examine routing tables for excessively large entries. 3. Monitor active connections to identify potential connection leaks. 4. Check system logs for memory-related errors. 5. Consider upgrading the device’s memory if it is consistently overloaded.

7. Output Interpretation

ping: Interpreting RTT (Round Trip Time), packet loss, and reachability.
traceroute/tracert: Identifying hops with high latency or packet loss. Recognizing potential routing loops.
ip/ipconfig: Understanding IP address, subnet mask, gateway, and DNS server settings.
netstat/ss: Interpreting connection states (ESTABLISHED, LISTEN, TIME_WAIT, etc.). Identifying listening ports.
tcpdump/Wireshark: Analyzing packet headers, protocol fields, and application data. Identifying abnormal traffic patterns.
nslookup/dig: Verifying DNS record accuracy and identifying DNS server issues.
curl/wget: Interpreting HTTP status codes (200 OK, 404 Not Found, 500 Internal Server Error, etc.).
mtr: Identifying hops with high latency or packet loss over time. Recognizing network bottlenecks.
arp: Mapping IP addresses to MAC addresses. Identifying potential ARP spoofing attacks.
ethtool: Understanding link speed, duplex settings, and autonegotiation status.

8. Security Considerations

tcpdump: Capturing sensitive data (passwords, etc.). Use filters to minimize captured data and store captured data securely. Disable promiscuous mode when not needed.
ping: Ping floods can be used for denial-of-service attacks. Rate-limit ICMP traffic.
traceroute/tracert: Can reveal network topology. Consider disabling ICMP responses on sensitive network segments.
arp: ARP spoofing can be used for man-in-the-middle attacks. Implement ARP inspection and DHCP snooping.
ethtool: Changing interface settings can disrupt network connectivity. Use with caution and only when necessary. Requires root/admin privileges.
Never store passwords or sensitive information in scripts or command-line history.
Be aware of legal and ethical considerations when capturing network traffic. Obtain proper authorization before capturing traffic on networks you do not own or manage.
Use strong passwords and multi-factor authentication to protect network devices.
Keep network devices and software up to date with the latest security patches.

9. Platform Differences

Tool	Linux	Windows	macOS
`ping`	Standard utility	Standard utility	Standard utility
`traceroute`	`traceroute`	`tracert`	`traceroute`
`ip`	`ip` command (iproute2 package)	`ipconfig` command	`ifconfig` (deprecated, use networksetup)
`netstat`	Standard utility	Standard utility	Standard utility
`ss`	`ss` command (iproute2 package)	Not available natively (install from Cygwin or WSL)	Not available natively (install via brew)
`tcpdump`	`tcpdump` command (libpcap)	`tcpdump` (WinPcap/Npcap)	`tcpdump` command (libpcap)
`nslookup`	Standard utility	Standard utility	Standard utility
`dig`	`dig` command (bind-utils package)	Not available natively (install from Cygwin or WSL)	`dig` command (included)
`curl`	Standard utility or install via package manager	Available natively since Windows 10	Standard utility
`wget`	`wget` command (install via package manager)	Not available natively (install from Cygwin or WSL)	`wget` command (install via brew)
`mtr`	`mtr` command (install via package manager)	Not available natively (install from Cygwin or WSL)	`mtr` command (install via brew)
`arp`	Standard utility	Standard utility	Standard utility
`ethtool`	`ethtool` command (install via package manager)	Not applicable	Not applicable
`tshark`	`tshark` command (Wireshark package)	`tshark` command (Wireshark package)	`tshark` command (Wireshark package)

Note: Package names for installing tools may vary depending on the Linux distribution. On macOS, brew install <tool> is a common way to install missing tools. Cygwin or Windows Subsystem for Linux (WSL) can provide a Linux-like environment on Windows. For installing tcpdump on Windows, Npcap is generally preferred over WinPcap.