Linux Troubleshooting Guide for Common System Issues

Linux troubleshooting is easier when you follow evidence instead of changing several settings at once. This linux troubleshooting guide provides a repeatable process for diagnosing disk, memory, systemd service and network problems on common server distributions.

The examples use standard tools found on Ubuntu, Debian, RHEL, Rocky Linux and similar systems. Some package names and log locations may differ between distributions.

What Is a Reliable Linux Troubleshooting Process?

A reliable process starts by defining the symptom, checking recent changes and collecting evidence before applying a fix. Change one variable at a time, verify the result and record what you changed so that it can be reversed.

Use this sequence:

1. Confirm the exact symptom and when it started.
2. Check the server clock, uptime and recent reboots.
3. Measure CPU, memory, disk and network state.
4. Read logs for the affected time period.
5. Form a testable hypothesis.
6. Make the smallest safe change.
7. Verify the service from the user or application perspective.
8. Document the root cause and preventive action.

A useful diagram-in-words is:

User symptom
    -> system health
        -> resource or service evidence
            -> focused test
                -> minimal fix
                    -> end-to-end verification

Begin with a quick system snapshot:

date
hostnamectl
uptime
who -b
free -h
df -h
systemctl --failed
journalctl -p err -b --no-pager

uptime shows load averages and how long the system has been running. who -b shows the last boot, while journalctl -p err -b displays error-level messages from the current boot.

Do not assume that the first error is the root cause. A web service failure, for example, may be caused by a full filesystem rather than an application defect.

How Do You Troubleshoot a Full Linux Disk?

Start by identifying whether the problem is block space, inode exhaustion or an application writing files rapidly. Then locate the affected filesystem and inspect its largest directories without crossing into unrelated mounted filesystems.

Check filesystem usage

df -hT
df -i
lsblk -f
findmnt

Example:

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/xvda2     xfs    20G   20G  120M 100% /
/dev/xvdb1     xfs    50G   12G   38G  24% /data

Here, the root filesystem is full. Free space under /data does not help files being written to /var, because /var belongs to the root filesystem in this example.

df -i checks inodes. A filesystem can have free capacity but reject new files when all inodes are used, often because an application created millions of small files.

Find where the space is used

Use du on the affected filesystem. The -x option prevents it from entering other mounted filesystems.

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -xhd1 /var 2>/dev/null | sort -h
sudo find /var -xdev -type f -size +500M -printf '%s %p\n' 2>/dev/null | sort -n

Typical sources include oversized logs, package caches, container layers, application uploads and old backups. Before deleting anything, identify the owner and purpose of the file.

sudo ls -lh /var/log
sudo journalctl --disk-usage
sudo lsof +L1

lsof +L1 finds deleted files that are still open. Deleting an active log removes its directory entry, but the disk blocks remain allocated until the process closes the file. Restarting or safely reloading the owning service releases that space.

Apply a safe disk-space fix

For systemd journal retention, use supported journal controls:

sudo journalctl --vacuum-time=7d

For a deleted file held by a service:

sudo lsof +L1
sudo systemctl restart application.service
df -h

Do not run broad commands such as rm -rf /var/log/*. They can remove required directories, audit evidence or files expected by running services. Configure logrotate, application retention and disk monitoring after recovery.

If access errors appear while inspecting application directories, review Linux file permissions and ownership before changing modes or owners.

How Do You Diagnose High Memory Usage?

First determine whether memory is genuinely exhausted or Linux is using otherwise idle RAM for cache. Then identify processes consuming resident memory and check whether the kernel has invoked the out-of-memory killer.

Interpret available memory correctly

free -h
vmstat 1 5
ps aux --sort=-%mem | head

Example:

               total        used        free      shared  buff/cache   available
Mem:            7.6Gi       5.1Gi       310Mi       120Mi       2.2Gi       2.0Gi
Swap:           2.0Gi       420Mi       1.6Gi

Low free memory alone is not a fault. The available value is more useful because it estimates memory that can be allocated without heavy swapping.

In vmstat, sustained non-zero si and so values indicate swap input and output. The first line contains averages since boot, so examine the later samples before drawing a conclusion.

Identify the responsible process

ps -eo pid,ppid,user,%mem,rss,cmd --sort=-rss | head -15
sudo systemd-cgtop
pidstat -r 1 5

RSS is resident physical memory, normally displayed in KiB by ps. systemd-cgtop is useful when services or containers are organised into control groups. pidstat may require the sysstat package.

Check for out-of-memory events:

sudo journalctl -k -g 'Out of memory|Killed process|oom-kill'
sudo dmesg -T | grep -Ei 'out of memory|killed process|oom-kill'

If the kernel killed a process, the logs normally identify its PID and memory use. Restarting it may restore service temporarily, but the root cause could be a memory leak, an undersized host, excessive worker processes or an incorrect container limit.

Recover without causing another outage

Prefer a controlled application restart during an approved window. Do not clear caches or drop page cache as a routine fix; this usually removes useful cached data and does not solve application memory growth.

For a systemd service, inspect its limits before changing them:

systemctl show application.service -p MemoryCurrent -p MemoryMax
systemctl cat application.service

After recovery, monitor memory over time and compare it with traffic or job schedules. Cloud and DevOps engineers can practise this type of operational diagnosis in an AWS DevOps course, where Linux evidence is connected with deployment and monitoring workflows.

How Do You Troubleshoot a Failed systemd Service?

Use systemctl status to identify the immediate failure and journalctl to read the complete event sequence. Validate configuration, dependencies, ports, permissions and environment files before repeatedly restarting the service.

A practical investigation looks like this:

sudo systemctl status nginx --no-pager -l
sudo journalctl -u nginx --since '30 minutes ago' --no-pager
sudo systemctl cat nginx
sudo systemctl show nginx -p User -p Group -p EnvironmentFiles

Important service states include:

Worked example: web service will not start

Assume systemctl status nginx reports an address-in-use error. Confirm the configuration and identify the process already listening:

sudo nginx -t
sudo ss -lntp | grep ':80 '
sudo systemctl list-dependencies nginx

Possible output:

LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:((apache2,pid=1842,fd=4))

Nginx cannot bind to TCP port 80 because Apache already owns it. The correct fix depends on the intended design: stop the unintended service, move one service to another port or configure a deliberate reverse-proxy arrangement.

sudo systemctl disable --now apache2
sudo nginx -t
sudo systemctl restart nginx
curl -I http://127.0.0.1/

On RHEL-based systems, the conflicting unit may be httpd.service rather than apache2.service.

If you edit a unit file or drop-in, reload the systemd manager before restarting:

sudo systemctl daemon-reload
sudo systemctl restart application.service
sudo systemctl is-active application.service
sudo systemctl is-enabled application.service

A running process is not complete proof of recovery. Test the listening port, local application response and remote client path.

How Do You Troubleshoot Linux Network Connectivity?

Separate the problem into interface, address, route, DNS, transport and application checks. Test each stage in order so that a DNS failure is not mistaken for a routing problem or a closed port is not mistaken for packet loss.

The path can be pictured as:

Application name
    -> DNS resolution
        -> destination IP
            -> routing decision
                -> local interface and gateway
                    -> firewall and remote service

Check interface, address and route state

ip -br link
ip -br address
ip route
ip route get 8.8.8.8

An interface marked DOWN cannot transmit normal traffic. A missing address, incorrect prefix or missing default route can prevent connectivity even when the physical or virtual link is up.

Test in increasing scope:

ping -c 3 127.0.0.1
ping -c 3 <local-gateway-ip>
ping -c 3 8.8.8.8
dig example.com
getent hosts example.com

Successful access to an IP address but failed name resolution points towards DNS. getent hosts is useful because it follows the system name service configuration, while dig queries DNS directly.

Check ports and packet flow

ss -lntup
curl -v --connect-timeout 5 http://server-ip:8080/
nc -vz server-ip 8080
sudo tcpdump -ni any host server-ip and port 8080

Interpret the evidence carefully:

Check host firewall configuration with the tool used by the distribution:

sudo nft list ruleset
sudo firewall-cmd --list-all
sudo ufw status verbose

Do not flush firewall rules on a remote server. You may lose administrative access and remove security controls. Network fundamentals from a structured CCNA course can help administrators understand addressing, routing and transport behaviour behind these tests.

How Can You Correlate Multiple Linux Symptoms?

Create a timeline and look for the earliest abnormal event rather than treating every alert independently. Disk, memory, services and networking often affect one another, so the first resource failure may explain several later messages.

Consider this incident:

09:10 /var reaches 100%
09:11 application cannot write session data
09:12 service health check fails
09:13 load balancer removes the server
09:15 users report a network error

The reported network error is only the visible symptom. The root cause is disk exhaustion, and the failed health check is the connection between them.

Build the timeline with:

journalctl --since '2026-09-21 09:00:00' --until '2026-09-21 09:20:00' -o short-iso
last -x | head
sudo grep -Rin 'error\|failed\|no space' /var/log 2>/dev/null | head -50

Verify timestamps and time zones when comparing Linux logs with cloud monitoring, load balancer events or application records. Also preserve relevant logs before rotating or deleting files during recovery.

What Should a Linux Troubleshooting Checklist Include?

A useful checklist should cover evidence collection, resource checks, service validation and end-to-end testing. It should also remind the operator to preserve access, avoid destructive commands and document the final root cause.

Summary

Linux troubleshooting should move from symptoms to measurable evidence and then to the smallest safe fix. Check disk blocks and inodes, interpret available memory correctly, use systemd journals for service failures, and test networking from the local interface through the application endpoint.

For practical training in Linux operations, automation, cloud monitoring and deployment troubleshooting, review the AWS DevOps course and contact Network Rhinos for current batch and enquiry details.

Reviewed by Network Rhinos Linux and DevOps trainers.