Linux Performance Monitoring with top, vmstat and iostat

Linux performance monitoring is the process of observing CPU, memory, processes, disk I/O and system events to find resource constraints. The commands top, vmstat, iostat and journalctl provide different parts of this evidence and are most useful when used together.

This guide explains what each command measures, how to read its output and how to connect metrics with relevant log messages. The examples apply to common Linux servers, cloud virtual machines and DevOps environments.

What should you monitor on a Linux system?

Linux performance monitoring should cover workload, resource demand, waiting time and system errors. A server can feel slow because CPU time is exhausted, memory pressure causes swapping, storage requests are delayed or an application repeatedly fails.

Think of the monitoring path as a diagram in words:

Users or automation → application processes → CPU and memory → filesystem and storage → kernel and service logs

The four tools in this guide map to that path:

No single metric proves the root cause. For example, high load average can indicate CPU demand, but it can also include tasks waiting in uninterruptible sleep, often because of storage or network filesystem delays.

How do you prepare the monitoring tools?

top, vmstat and journalctl are normally available on systemd-based Linux distributions. The iostat command is supplied by the sysstat package and may need to be installed.

On Ubuntu or Debian:

sudo apt update
sudo apt install sysstat

On RHEL, Rocky Linux, AlmaLinux or Fedora:

sudo dnf install sysstat

Confirm that the commands are available:

command -v top vmstat iostat journalctl

Run monitoring commands as a normal user where possible. Some journal entries require membership in groups such as systemd-journal, adm or wheel, depending on the distribution, or temporary access through sudo.

How do you use top to find busy processes?

top gives a live view of system load, CPU states, memory and individual processes. Start with the system summary, then sort the process list to identify which workload is consuming the constrained resource.

Run it interactively:

top

A shortened display may look like this:

top - 14:25:31 up 12 days,  2:18,  2 users,  load average: 3.42, 2.70, 1.91
Tasks: 214 total,   3 running, 210 sleeping,   0 stopped,   1 zombie
%Cpu(s): 71.0 us, 18.0 sy,  0.0 ni,  6.0 id,  4.5 wa,  0.3 hi,  0.2 si,  0.0 st
MiB Mem :  7936.0 total,  410.0 free, 5320.0 used, 2206.0 buff/cache
MiB Swap: 2048.0 total, 1790.0 free,  258.0 used. 2230.0 avail Mem
    PID USER      PR  NI    VIRT    RES  S  %CPU  %MEM COMMAND
  18422 app       20   0 2864520   1.1g R 162.0  14.2 java

Interpret the important fields as follows:

• Load average: runnable tasks and tasks waiting in uninterruptible sleep over 1, 5 and 15 minutes. Compare it with the number of logical CPUs, but do not treat that comparison as a complete diagnosis.
• `us`: CPU time running user-space code.
• `sy`: CPU time running kernel code.
• `id`: idle CPU time.
• `wa`: idle time while the system has outstanding I/O requests. It is a clue, not a direct measurement of disk utilisation.
• `st`: time taken from a virtual machine by the hypervisor. Sustained steal time may indicate host contention.
• `RES`: physical memory currently resident for a process.
• `S`: process state, such as R for running, S for sleeping, D for uninterruptible sleep and Z for zombie.

Useful interactive keys include:

For a non-interactive snapshot suitable for an incident record, use batch mode:

top -b -n 1 | head -n 30

A process showing 162% CPU is not automatically an error. On a multi-core system, that value can mean that its threads together are using approximately 1.62 logical CPUs.

How do you read vmstat output?

vmstat shows whether tasks are runnable, blocked, swapping or waiting while the CPU handles user and kernel work. It is especially useful for distinguishing CPU saturation from memory pressure and I/O-related blocking.

Collect five samples at one-second intervals:

vmstat -w 1 5

Example:

--procs-- -----------------------memory---------------------- ---swap-- -----io---- -system-- ----------cpu----------
   r    b         swpd         free         buff        cache   si   so    bi    bo   in   cs us sy id wa st
   6    2       264192       385120       102400      2124800    0    0   420  3800 5200 9100 72 16  5  7  0
   7    3       264192       380600       102400      2129000    0    0  1200  9600 6800 12500 68 18  3 11  0

The first line of values usually represents averages since boot, so focus on the later interval samples when investigating a current event.

Linux uses otherwise idle memory for filesystem cache, so a low free value is not necessarily a problem. Check available memory, active swap traffic and application behaviour rather than clearing caches as a routine response.

How do you diagnose disk performance with iostat?

iostat reports throughput, request rates, queueing and latency for block devices. Extended device statistics help determine whether a slow application is waiting on storage rather than consuming CPU.

Run extended output with readable units and repeated samples:

iostat -xz 1 5

A simplified device section might show:

Device            r/s    rkB/s   w/s    wkB/s  aqu-sz  await  %util
nvme0n1          42.0   1860.0  95.0   7420.0    3.20  18.40  92.10
sdb               0.2      8.0   1.0     24.0    0.01   1.30   0.40

Important columns include:

• `r/s` and `w/s`: completed read and write requests per second.
• `rkB/s` and `wkB/s`: read and write throughput.
• `aqu-sz`: average number of requests waiting or being serviced.
• `await`: average request completion time in milliseconds, including queue time and service time.
• `%util`: percentage of elapsed time during which the device had I/O in progress.

In this example, nvme0n1 has a queue, elevated latency and high utilisation during the sample. You would next identify the processes generating I/O, check kernel messages and compare the values with a known healthy period.

Do not apply one universal await threshold to every device. Acceptable latency depends on whether the storage is local HDD, SSD, NVMe, virtualised block storage, RAID or network-backed storage. Also, %util near 100% does not always mean a modern parallel device has reached its maximum throughput.

Map devices and mount points before drawing conclusions:

lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
findmnt

How do you investigate errors with journalctl?

journalctl searches the systemd journal for service, kernel and boot events. Use it to explain why a metric changed, such as a service restart during a CPU spike or an I/O error during high disk latency.

Show warnings and more severe messages from the last 15 minutes:

sudo journalctl -p warning --since "15 minutes ago"

Inspect one service:

sudo journalctl -u nginx.service --since "2026-11-05 14:00:00"

Follow new service messages in real time:

sudo journalctl -u nginx.service -f

Check kernel messages for the current boot:

sudo journalctl -k -b

Show logs from the previous boot:

sudo journalctl -b -1

Limit output and include precise timestamps:

sudo journalctl -u docker.service -n 100 -o short-iso-precise --no-pager

Useful journal fields include the timestamp, hostname, process or unit name, PID and message. Search terms such as timeout, oom, I/O error, reset, failed and killed process can help, but always read the surrounding events:

sudo journalctl -k --since "1 hour ago" --grep='oom|I/O error|reset'

Journal retention depends on the system configuration. If persistent storage is not enabled, older logs may disappear after a reboot. Check current usage with journalctl --disk-usage and review /etc/systemd/journald.conf before changing retention settings.

What is a practical Linux performance monitoring workflow?

Begin with the reported time and symptom, capture broad system evidence, and then narrow the investigation. Correlating multiple samples is more reliable than reacting to a single high value.

Use this sequence:

1. Record the exact time, affected service and user-visible symptom.
2. Run top to inspect load, CPU states, memory and busy processes.
3. Run vmstat -w 1 10 to separate CPU demand, blocked tasks and swapping.
4. If wa, b, bi or bo is elevated, run iostat -xz 1 10.
5. Search journalctl over the same time window.
6. Compare results with a normal period or another healthy server.
7. Make one controlled change, then measure again.

For example, suppose an API becomes slow. top shows moderate CPU use but several tasks in D state. vmstat reports b between 8 and 12 with increasing I/O wait, while iostat shows rising await for the application volume. journalctl -k then reports repeated block-device timeouts.

The evidence points toward the storage path rather than insufficient CPU. The next action is to inspect the cloud volume, virtual storage layer, device health or filesystem—not to add application worker processes.

This evidence-first method complements a broader Linux troubleshooting guide while keeping the focus here on performance measurements.

How can you save evidence for later analysis?

Command output can be timestamped and written to an incident directory for comparison. Keep the collection interval reasonable and avoid scripts that grow without retention limits.

A simple manual capture is:

mkdir -p "$HOME/perf-capture"
date --iso-8601=seconds | tee "$HOME/perf-capture/time.txt"
vmstat -w 1 10 | tee "$HOME/perf-capture/vmstat.txt"
iostat -xz 1 10 | tee "$HOME/perf-capture/iostat.txt"
top -b -n 1 | head -n 50 > "$HOME/perf-capture/top.txt"
sudo journalctl --since "10 minutes ago" --no-pager > "$HOME/perf-capture/journal.txt"

For repeated collection, add timestamps, error handling, access controls and log rotation. The Linux shell scripting basics for automation article explains how to turn command sequences into maintainable scripts.

Common monitoring mistakes

Performance diagnosis fails when metrics are treated as isolated pass-or-fail tests. Use trends, workload context and logs before changing production settings.

Avoid these mistakes:

• Treating load average as CPU percentage.
• Assuming low free memory means Linux has run out of RAM.
• Treating allocated swap as proof of current swapping.
• Killing the highest-CPU process before identifying its function.
• Assuming high %util has the same meaning for HDD, NVMe and virtual storage.
• Reading the first vmstat or iostat sample as the current interval.
• Changing several parameters at once and losing the ability to verify the cause.
• Collecting metrics without noting the incident time and affected service.

Summary

Linux performance monitoring works best as a correlation exercise. Use top for the live process view, vmstat for scheduling and memory pressure, iostat for storage behaviour, and journalctl for events that explain the measurements.

Practise these commands on a test server under normal and controlled load. If you want hands-on Linux, automation, cloud monitoring and deployment practice, review the AWS DevOps course and enquire about current batch details.

Reviewed by Network Rhinos Linux and DevOps trainers.