Security tools, servers, applications and cloud platforms generate thousands of events during normal operation. A Security Information and Event Management platform brings these events together so that security teams can detect suspicious patterns, investigate incidents and retain evidence.
This guide explains SIEM architecture and detection logic rather than focusing on one product. Learners who want structured practice with log analysis, alert triage and incident investigation can explore the Cybersecurity & SOC course.
What is SIEM?
SIEM stands for Security Information and Event Management. It is a platform that collects logs from multiple systems, converts them into searchable events, correlates related activity and alerts security teams when defined threat conditions are met.
A SIEM does more than store logs. Its main value comes from connecting events that appear harmless individually but become suspicious when viewed together.
For example, one failed login is common. Fifty failed logins against several accounts, followed by a successful login from the same source address, may indicate password spraying or account compromise.
A SIEM normally provides these functions:
- Centralised log collection
- Event parsing and field extraction
- Data normalisation and enrichment
- Search and investigation
- Correlation and detection rules
- Dashboards, alerts and case integration
- Log retention for operational or compliance needs
Why do security teams need logs?
Logs provide records of what happened, when it happened, where it originated and which identity or system was involved. Security teams use this evidence to identify attacks, confirm their scope and reconstruct activity after an incident.
No single log source provides the complete story. A firewall may record a network connection, while an identity service records the user login and an endpoint tool records the process that ran afterwards.
Consider these three events:
09:14:02 Firewall: 203.0.113.25 connected to VPN gateway
09:14:08 Identity: user arun authenticated successfully
09:16:41 Endpoint: powershell.exe started an encoded commandThe firewall event alone could be legitimate. The successful login is also not automatically malicious. When the endpoint launches an unusual encoded command shortly afterwards, the combined sequence deserves investigation.
Useful security log sources include:
| Source | Example security information |
|---|---|
| Firewalls | Allowed and denied connections, NAT details and policy identifiers |
| Windows systems | Logons, account changes, service creation and process execution |
| Linux systems | SSH authentication, sudo use, services and kernel messages |
| DNS servers | Domain queries, responses and requesting clients |
| VPN gateways | Remote user sessions, source addresses and authentication results |
| Web proxies | Requested URLs, response codes and transferred bytes |
| Endpoint tools | Processes, file changes, malware findings and device isolation |
| AWS | CloudTrail API activity, VPC Flow Logs and service-specific logs |
| Microsoft Azure | Azure Activity Log, sign-in logs and resource diagnostic logs |
| Applications | User sessions, errors, administrative actions and API requests |
Firewall logging quality directly affects SIEM visibility. The practical guide to FortiGate policies, NAT and logging explains how firewall decisions become useful log records.
How does SIEM architecture work?
A SIEM uses a pipeline: sources generate events, collectors receive them, parsers identify fields, the platform enriches and stores the data, and detection rules evaluate it. Matching activity becomes an alert for investigation.
A diagram in words looks like this:
[Endpoints] [Firewalls] [Cloud services] [Applications]
\ | | /
\--------------+----- Log collection ---------------/
|
Parsing and normalisation
|
Enrichment and time alignment
|
Searchable event storage
|
Correlation and detection rules
|
Alerts and cases
|
Analyst investigationCollection
Agents installed on endpoints can read local files, Windows Event Logs or operating-system journals. Agentless methods include Syslog, APIs, cloud connectors and log files placed in object storage.
Collection must be reliable. Production designs may use local buffering so that events are not immediately lost when the SIEM destination is unavailable.
Parsing and normalisation
A parser converts unstructured text into fields. The following raw SSH message contains a timestamp, account, source IP address and source port:
Sep 18 10:22:31 web01 sshd[4312]: Failed password for admin from 198.51.100.44 port 51872 ssh2After parsing, the event could resemble:
{
"host": "web01",
"event_category": "authentication",
"outcome": "failure",
"user": "admin",
"source_ip": "198.51.100.44",
"source_port": 51872
}Normalisation maps different vendor names into consistent fields. One product may use src_ip, while another uses clientAddress. A common field such as source_ip allows one rule to evaluate both products.
Enrichment
Enrichment adds context not present in the original event. A SIEM may add asset ownership, system criticality, identity department, approved administrator lists or threat intelligence matches.
Context changes priority. A suspicious login to a disposable test server may be less urgent than the same activity involving a privileged account on a production identity server.
Detection and alerting
Detection rules evaluate events using thresholds, sequences, statistical baselines or known indicators. An alert should contain enough context for an analyst to decide whether the activity is expected, suspicious or confirmed malicious.
How does a raw log become a threat alert?
A raw log becomes an alert when the SIEM parses its fields and a detection rule matches a suspicious condition. Strong rules combine event evidence, time limits and relevant context instead of alerting on every unusual message.
Imagine the following authentication activity:
10:00:04 failure user=admin source_ip=198.51.100.44
10:00:11 failure user=backup source_ip=198.51.100.44
10:00:19 failure user=helpdesk source_ip=198.51.100.44
10:00:27 failure user=admin source_ip=198.51.100.44
10:01:10 success user=admin source_ip=198.51.100.44A useful detection could look for repeated failures from one source followed by a success. The logic can be expressed in vendor-neutral pseudocode:
GROUP authentication events BY source_ip
WHERE failures >= 4 WITHIN 5 minutes
AND a successful login occurs from the same source_ip
WITHIN the next 2 minutesThe threshold is only an example for a lab. Production thresholds must reflect normal behaviour, internet exposure, remote-access design and the authentication controls in use.
The SIEM could enrich the resulting alert with:
- Whether the address is internal or external
- The affected user's privilege level
- Previous alerts involving the address
- Device ownership and business importance
- VPN, endpoint and DNS activity around the same time
How can you send Linux logs to a SIEM?
Linux systems commonly forward logs through rsyslog, Syslog-ng or a dedicated SIEM agent. A basic rsyslog lab can send events over TCP, but production environments should add TLS, authentication, buffering and access controls.
Create /etc/rsyslog.d/60-siem.conf on a Linux source:
*.* action(
type="omfwd"
target="10.20.30.40"
port="514"
protocol="tcp"
action.resumeRetryCount="-1"
queue.type="linkedList"
queue.filename="siem_fwd"
queue.saveOnShutdown="on"
)Validate the configuration and restart rsyslog:
sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo systemctl status rsyslog --no-pagerGenerate a controlled test message:
logger -p authpriv.warning -t siem-lab "SIEM_TEST failed_login user=labuser source_ip=198.51.100.44"Check whether the local service processed it:
sudo journalctl -t siem-lab --since "5 minutes ago"This example uses unencrypted TCP for an isolated lab. Syslog over TLS commonly uses TCP port 6514, but the sender, receiver, certificates and peer verification must all be configured consistently.
How can Cisco IOS logs be forwarded?
Cisco IOS and IOS XE devices can send Syslog messages to a remote collector. Configure accurate timestamps, a useful severity level and the collector destination, then verify the active logging state.
A typical lab configuration is:
configure terminal
service timestamps log datetime msec localtime show-timezone
logging facility local6
logging trap informational
logging host 10.20.30.40 transport tcp port 514
end
show logginglogging trap informational sends severity level 6 and more severe messages. This can create substantial volume on busy devices, so production teams should choose the level according to their monitoring requirements and test device capacity.
Time synchronisation is essential. Configure approved NTP sources so that firewall, server, identity and network events can be placed on the same timeline.
How should an analyst investigate a SIEM alert?
An analyst should validate the alert, identify the affected entities, build a timeline and determine whether the activity has a legitimate explanation. Containment should follow the organisation's incident process rather than being based on one isolated event.
For the failed-login-then-success example, an investigation can follow this sequence:
- Confirm that the source events exist and were parsed correctly.
- Check the source address, destination system, username and timestamps.
- Determine whether the source belongs to a VPN, proxy, scanner or approved administrator.
- Search for other accounts targeted by the same source.
- Review endpoint process activity after the successful login.
- Check DNS, proxy and firewall records for subsequent connections.
- Ask the account owner or system owner to validate the activity when appropriate.
- Record the evidence, decision and containment actions in the case.
Windows Event ID 4625 represents a failed logon, while 4624 represents a successful logon. Event ID 4688 can show process creation when the required audit policy is enabled. Event IDs still need fields such as logon type, account, host and source address to be interpreted correctly.
For a broader view of triage, escalation and documentation, read what a SOC analyst does during a working day.
What is the difference between SIEM, SOAR, EDR and XDR?
SIEM centralises and analyses events, while SOAR automates workflows and EDR monitors endpoint activity. XDR correlates telemetry and response across multiple security layers, often within an integrated vendor platform.
| Technology | Primary role | Typical example |
|---|---|---|
| SIEM | Central log analysis and correlation | Detect related identity, firewall and server events |
| SOAR | Workflow automation and orchestration | Enrich an alert and open a case automatically |
| EDR | Endpoint detection and response | Detect a malicious process and isolate a laptop |
| XDR | Cross-domain detection and response | Correlate endpoint, email, identity and network activity |
These tools can work together. For example, a SIEM alert may start a SOAR playbook that checks reputation data, asks an EDR platform for process details and creates an incident ticket.
How do teams reduce SIEM false positives?
Teams reduce false positives by improving log quality, adding asset and identity context, tuning thresholds and excluding verified administrative activity carefully. Tuning should make a detection more precise without creating blind spots.
Useful tuning methods include:
- Separate internet-facing systems from internal systems.
- Apply different thresholds to users, service accounts and administrators.
- Exclude a vulnerability scanner only after its addresses and schedule are controlled.
- Alert on a sequence of events rather than a single event.
- Suppress duplicate alerts while retaining the underlying events.
- Review rules after network, identity or application changes.
- Track why every exception exists and when it should be reviewed.
A dashboard is not a detection strategy. Teams need documented use cases stating the threat, required data, rule logic, expected false positives, investigation process and owner. Product-specific searching and visualisation can then be practised through the Splunk beginner SIEM dashboard lab.
How do you troubleshoot missing SIEM logs and alerts?
Troubleshoot the pipeline one stage at a time: source generation, local processing, network delivery, collector receipt, parsing, indexing and rule evaluation. This prevents teams from changing detection rules when the real problem is blocked transport or an incorrect timestamp.
1. Confirm that the source creates the event
sudo journalctl --since "10 minutes ago"
sudo tail -n 50 /var/log/auth.logLog locations differ by distribution and service. Some systems use the systemd journal without writing the expected traditional file.
2. Test network delivery
For a TCP listener:
nc -vz 10.20.30.40 514
sudo tcpdump -nn -i any host 10.20.30.40 and port 514A successful TCP connection does not prove that parsing or indexing works. It only confirms that the destination accepted the connection.
3. Verify the collector
On a Linux collector, check listening sockets and service messages:
sudo ss -lntup | grep -E ':514|:6514'
sudo journalctl -u rsyslog --since "15 minutes ago"If TLS is enabled, check certificate trust, certificate names, validity dates and whether both sides expect TLS on the same port.
4. Inspect parsing and timestamps
Search for the raw test marker before searching extracted fields. If the raw event exists but source_ip is empty, the parser may not match the current message format.
Also compare these times:
- Time generated on the source
- Time received by the collector
- Time indexed by the SIEM
- Time range selected in the search
Wrong time zones or unsynchronised clocks can make current events appear outside the investigation window.
5. Validate detection logic
Confirm field names, event categories, grouping keys, threshold values and sequence order. Check whether maintenance windows, allowlists or suppression rules prevented the alert.
Use known test data instead of performing uncontrolled attacks. A repeatable test message makes it easier to verify collection and parsing after every configuration change.
What should a practical SIEM learning lab include?
A useful lab should include several log sources, a documented data pipeline, testable detection rules and an investigation worksheet. The goal is to understand why an alert fired and how to validate it, not only how to create a dashboard.
A small lab can contain:
- One Linux server sending authentication logs
- One Windows system providing security events
- A router or firewall sending Syslog
- A SIEM or log analytics platform
- Accurate time synchronisation
- Test cases for failed logins, privilege use and configuration changes
- A case template covering evidence, decision and next action
Start with one source and prove every pipeline stage before adding more devices. Record the expected event format, parser fields, detection rule, test procedure and troubleshooting commands.
Summary
SIEM platforms collect security events, extract useful fields, enrich them with context and correlate activity across systems. Effective detection depends on reliable logs, accurate timestamps, clear use cases and rules that analysts can investigate.
The most important practical skill is following an event through the complete pipeline: generation, transport, parsing, storage, detection and response. When a result is wrong, test each stage separately rather than treating the SIEM as a single black box.
To practise log collection, SIEM searches, alert triage and incident investigation in guided labs, enquire about upcoming batch details for the Cybersecurity & SOC course.
Reviewed by Network Rhinos cybersecurity trainers.