Practice log analysis and evidence collection before moving to reading resources.

Mental Model: The Black Box Recorder

In aviation, a black box does not prevent crashes. It makes crashes explainable. After something goes wrong, investigators can reconstruct what happened, understand why, and prevent future incidents.

Logs function similarly in computing:

• They don't guarantee safety: logging doesn't stop attacks
• They enable reconstruction: what happened, in what order
• They allow accountability: who did what, and when
• They turn mystery into investigation: evidence replaces speculation

But a black box is only useful if it records the right signals, remains intact, and can be trusted. A corrupted or incomplete black box is worse than useless — it provides false confidence.

Observability is broader than logging: it's your ability to infer system behavior from all available signals — logs, metrics, traces, alerts, state changes. Security depends on answering: What happened? When? Who triggered it? What changed? How confident are we in these answers?

Mental model: logs are the system's black box. They don't prevent incidents, but without them, you can't investigate, learn, or prove anything.

1) What Logs Are (and What They Are Not)

Logs are records of events emitted by software. They are not reality — they are claims about reality. This distinction is crucial for security reasoning.

A log entry claims:

• Something occurred: according to some software component
• At some time: according to the system clock at that moment
• In some context: a process, user, session, or request identifier
• With some outcome: success, failure, or specific result

The critical security question becomes: Who controls the claim?

If the software making the claim is compromised, the log may be a lie. If the clock is manipulated, the timestamp may be wrong. If the context is spoofed, the attribution may be false. Logs are evidence, but evidence can be fabricated, altered, or destroyed.

Key insight: logs are claims about events, not the events themselves. Trust in logs depends on trust in the systems that generate them.

2) Where Logs Come From: Layers of Evidence

Modern systems generate logs at multiple layers, each with different trust properties and coverage:

• Application logs: "My service received request X from user Y" — most detailed but controlled by the application
• System/service logs: "Daemon started/stopped/crashed" — OS-level visibility into service lifecycle
• Authentication logs: "Login succeeded/failed for user Z" — critical for tracking access and intrusion attempts
• Network logs: "Connection from IP A to port B" — captures traffic patterns regardless of application behavior
• Audit/security logs: "Permission changed on file X" — tracks sensitive operations for compliance and forensics
• Kernel/hardware logs: "Memory error detected" — lowest level, hardest to tamper with

Each layer has different trust properties. Application logs are detailed but easily manipulated by a compromised application. Kernel logs are harder to fake but less detailed about application behavior.

Attackers will target the layer they can control or silence. A sophisticated attacker who compromises an application may alter its logs while kernel-level evidence remains intact.

Key insight: defense in depth applies to logging too. Multiple layers of logs make it harder for attackers to erase all evidence.

3) Why Attackers Target Logging

In many real compromises, attackers do not simply try to "avoid leaving evidence." They actively work to control it:

• Disable: turn off logging services or reduce verbosity settings — future actions won't be recorded
• Blind: overload the system with noise so real events are lost in the flood of meaningless entries
• Erase: delete local log files — if logs aren't backed up remotely, they're gone forever
• Distort: modify timestamps or rewrite entries — create false timelines or alibis
• Impersonate: make malicious activity look like routine automation — hide in plain sight among legitimate scheduled tasks

Weeks 6–7 should make this clear: background services and scheduled tasks are ideal "cover stories." If attackers can make their activity look like normal cron jobs or service operations, defenders may never notice.

The first thing sophisticated attackers do after gaining access is assess what logging exists and how to neutralize it. Logging is a threat to them — so they treat it as a target.

Key insight: attackers don't just avoid detection — they actively work to undermine the evidence systems that could expose them.

4) Time, Ordering, and Confidence

Logs are strongly dependent on time. But time in computing is fragile:

• Clocks drift: without synchronization, systems diverge by seconds or minutes
• Systems disagree: different machines may have different clock settings
• Attackers manipulate: timestamps can be changed before or after logging
• Timezones confuse: UTC vs local time creates reconstruction errors

Therefore investigations rely not only on timestamps, but also on:

• Ordering: what clearly happened before/after, even if exact times are uncertain
• Correlation: matching signals across multiple sources — network logs, application logs, authentication logs
• Consistency: whether multiple independent logs agree on what happened
• Sequence numbers: log entries numbered in order, independent of timestamps

Evidence becomes stronger when it is duplicated across independent systems. If your application logs, network firewall logs, and authentication server logs all agree that user X connected at time T, you have corroboration. If only one source says it, you have a claim.

Key insight: timestamps matter, but corroboration across independent sources matters more. Attackers can forge one log; forging many synchronized logs is harder.

5) Centralisation: The "Outside the Host" Principle

A log stored only on the compromised machine is fragile evidence. If attackers control the host, they can often control the record. Local logs are the first thing sophisticated attackers delete.

A powerful defensive principle is:

Store evidence outside the thing you are investigating.

This is why organisations centralise logs:

• Forward logs in real-time: to a SIEM or log aggregator the attacker can't reach
• Use append-only storage: write-once media or immutable cloud storage
• Separate access controls: system admins shouldn't be able to delete security logs
• Monitor the logging pipeline: alert if log forwarding stops unexpectedly

Centralisation is not about convenience — it is about integrity. Logs that exist only on the compromised system are logs the attacker controls.

Modern observability extends beyond logs to include metrics (CPU, memory, network), traces (request paths through services), state changes (new users, new tasks), and alerts (thresholds and anomaly detection). Attackers can hide from one signal, but hiding from multiple independent signals is exponentially harder.

Key insight: the value of a log is inversely proportional to the attacker's ability to modify or delete it. Remote, immutable storage is essential.

Real-World Context: Logging in Security Incidents

The importance of logging becomes clear in real-world incidents:

SolarWinds Attack (2020): The attackers specifically targeted logging and monitoring systems. They disabled security tools, avoided noisy operations, and timed their activity to blend with normal traffic. Organisations with centralised, immutable logging were better able to reconstruct the attack timeline. Those relying on local logs often found them deleted or corrupted.

Log4j Vulnerability (2021): CVE-2021-44228 turned logging itself into an attack vector. The Log4j library — used to create log entries — contained a vulnerability that allowed remote code execution. The irony: the very system meant to record events became the entry point for compromise. This demonstrated that logging infrastructure is part of the attack surface.

Ransomware Operations: Modern ransomware routinely targets backup systems and log servers before encrypting primary data. Attackers understand that defenders need logs for incident response — so they destroy them first. Organisations with off-site, immutable log storage recover faster.

A defender who cannot reconstruct events is not defending — they are guessing. Logs are claims about events, not events themselves, but they are often the only evidence available after a compromise.

Common thread: in each case, logging was either a critical defensive asset or a specific target. Attackers understand the value of evidence — do you?

Guided Lab: Exploring System Evidence

This lab focuses on discovery and analysis. You will examine how your system records events and evaluate the reliability of that evidence.