From Detections to Proof: Validating Flow Logs with Tests and Metrics

Unit Tests for Detections

Each detection rule is paired with a test:

• Input: sample flow log events (JSON/CSV).

• Expected Output: detection triggers with correct context.

• Automation: CI pipelines (pytest, GitHub Actions) check rules whenever environments change.

This prevents “silent failures” where detections stop working without anyone noticing.

In Part 1, we explored how VPC Flow Logs can be mapped to ATT&CK and built into a detection pack. But there’s a catch: a detection that’s never been tested is a detection you can’t trust.

This post covers how to prove detections work—using synthetic attacks, unit tests, and operational metrics like MTTD and MTTR.

Testing with Synthetic Traffic and PCAPs

To validate, we need to simulate attacks safely:

• Sample PCAPs – replay traffic from scanning or exfiltration in a controlled lab.

• Synthetic Generators – use tools (nmap, hping3, Python scripts) to generate predictable attack traffic.

Result: logs that mimic attacker behavior flow into the SIEM, and rules fire as expected.

Measuring Security Effectiveness: MTTD and MTTR

Validated detections must connect to business outcomes. Two key metrics:

• MTTD (Mean Time to Detect):

  • Measure from attack simulation → SIEM alert.

  • Lower is better; shows pipeline health.

• MTTR (Mean Time to Respond):

  • Measure from alert → analyst action.

  • Lower means faster incident handling.

Instrumenting these metrics transforms detections from static rules into performance indicators leadership cares about.

Why This Approach Stands Out

Most blogs about flow logs end with “you can detect X.” This approach is different:

• Proven → every detection tested with synthetic traffic.

• Measured → every detection tied to MTTD/MTTR.

• Mapped → every detection linked to ATT&CK.

This makes flow logs not just data, but defensible proof of security capability.

📌 Key Takeaway (Part 2)

It’s not enough to collect or even detect—you must prove your detections work. Unit tests, synthetic attacks, and performance metrics make detections reliable, measurable, and leadership-ready.

Call to Action

If your team already collects VPC Flow Logs, don’t stop at storage.

• Map them to ATT&CK techniques.

• Validate detections with synthetic traffic.

• Measure success with MTTD/MTTR dashboards.

Because in modern cloud security, it’s not about what logs you have—it’s about the detections you can prove.