Skip to content

Track 09 — Python for Security

You already write Python; the copilot writes the boilerplate. This track is about the skill that's left: engineering security tooling that survives adversarial input, runs concurrently at scale, is typed, observable, and tested — and directing and catching the AI where it reliably fails. You build one tool, sift, and grow it across nine modules into a portfolio centerpiece.

This is an intermediate-plus track. It assumes you're comfortable with Python (functions, classes, the stdlib) and work with an AI copilot. If you don't yet write Python, start at Foundations · Module 10 — Scripting & Automation and come back.

What you'll be able to do

  • Drive the copilot with spec-driven development — write the spec, let AI implement, review the implementation against the spec — instead of pasting unread output.
  • Validate untrusted input at the boundary with pydantic — parse, don't trust — and carry that same discipline to the AI edge (instructor) and the measurement layer (pydantic-evals).
  • Build async, concurrent enrichment that handles rate limits and backoff without introducing races.
  • Drive external tools safely (no shell=True), and serve one core as a CLI, an HTTP API, and an MCP server.
  • Red-team the MCP server you built, then prove the fix holds with an eval regression gate.
  • Review and own AI-generated code — catch the shell=True, the subtly-wrong type, the resource leak.

The spine — one evolving tool

The whole track builds sift, an alert enrichment-and-triage tool: ingest a real Suricata EVE JSON feed (eve.json) → validate → enrich (threat-intel APIs) → score/triage → serve (CLI · API · MCP). Every module adds a real capability and targets a bug-class the copilot reliably ships. The data is not invented: sift parses genuine Suricata output — real alert events with real Emerging Threats signature IDs, generated by running Suricata over a real infection PCAP — and the optional dissector thread grows sift from an alert-only parser into a multi-event_type EVE dissector (dns, http, tls, flow, fileinfo).

The shape of the track

At a glance — three phases grow the one sift tool: first it becomes a real, typed, linted project that validates untrusted input; then it enriches concurrently and serves two surfaces; then it goes AI-native, gets attacked, and is finally measured. The "parse, don't trust" through-line runs the length of it — untrusted input (M2), untrusted LLM output (M7), and measurement (M9).

flowchart LR
    P1["Phase 1 · Foundation &amp; correctness<br/>01–03 · typed, linted, validated, at scale"]
    P2["Phase 2 · Concurrency, integration, scale<br/>04–06 · async · safe tools · CLI + API"]
    P3["Phase 3 · Trust, AI-native, measure<br/>07–09 · MCP · red-team · eval-gated"]
    P1 --> P2 --> P3
    P1 -.parse untrusted INPUT (M2).-> P3
    P2 -.the sift core.-> P3

Modules

# Module Type What you add to sift Modern stack
01 Modern Toolchain & Spec-Driven Skeleton Migration + ADR Migrate a legacy script into a uv project with a CI gate; adopt a spec-driven workflow; write the toolchain ADR uv, ruff, pyright, openspec
02 Parse, Don't Validate Tool-Build A typed EVE-JSON boundary that rejects malformed/adversarial Suricata events (+ a dns dissector stretch) pydantic v2, pydantic-settings
03 Data at Scale & Structured Logs Tool-Build Stream a real eve.json + columnar triage queries + JSON logs (+ an http dissector stretch) polars/duckdb, structlog
04 Async & Structured Concurrency Build-&-Operate An async enricher with bounded concurrency, backoff, rate-limit handling — plus a durable huey task-queue beat httpx async, asyncio, huey
05 Driving Tools Safely Tool-Build + Review Safe subprocess wrappers (no shell=True) + robust output parsers subprocess, shlex
06 Two Surfaces, One Core Build-&-Operate A typer CLI and a FastAPI service sharing the same models typer, FastAPI
07 LLM-Native Python & MCP Tool-Build An MCP server exposing sift; typed LLM output validated like an API response MCP, instructor
08 Red-Team Your Own MCP Server Red-team-the-AI A working prompt-injection exploit against your enrich tool + the eval that catches it promptfoo/garak
09 Eval Harness, Property Tests & Supply Chain Eval Harness + Review A held-out corpus + scorecard + CI regression gate; property tests; a supply-chain gate pydantic-evals, hypothesis, pip-audit

Phases & projects

The nine modules run in three phases; each phase advances sift into a genuinely more capable tool.

  • Phase 1 · Foundation & correctness (01–03) — the tool becomes a real, typed, linted project that validates untrusted input and scales past toy data.
  • Phase 2 · Concurrency, integration, scale (04–06) — it enriches concurrently and safely, drives external tools without injection, and serves the same core as a CLI and an HTTP API.
  • Phase 3 · Trust, AI-native, measure (07–09) — it becomes callable by an LLM (MCP), gets attacked and hardened, and is finally measured: eval-gated, property-tested, supply-chain-audited.

The through-line — parse, don't trust

One discipline, three edges: pydantic validates untrusted input — real Suricata EVE JSON — at the boundary (M2), instructor validates untrusted LLM output (M7), pydantic-evals measures the whole system (M9). No other security curriculum teaches this Pydantic-native spine end to end — it's the track's identity. (Teach the pattern as provider-agnostic; the Pydantic ecosystem is the concrete OSS instance, not the only way.) A parallel dissector thread runs through the labs' Stretch tasks: the core parses alert events, and each module optionally adds a typed dissector for one more EVE event_type, quarantining unknown types rather than crashing on them.

Prerequisites

Foundations — Module 10 (Scripting & Automation) is the floor. This track starts above it.

Capstone

The evolved sift: typed, validated, async, served (CLI + API + MCP), property-tested, eval-gated, supply-chain-audited, and red-teamed against itself. The kind of repo that ends an interview, not one that starts a tutorial. Deliverable: the tool, its tests and eval harness, and a write-up of what AI wrote vs. what you changed and why. (Honor system: the committed tool is the proof.)

The starter scaffold and acceptance checks live in plaintext-labs/python-for-security/capstone/.

Capstone rubric

Proficient is the bar to ship. It must be a genuinely useful tool you own — typed, tested, reviewed line by line, and fed real data — the track's own real Suricata eve.json (generated by running Suricata over a real infection PCAP), optionally augmented with a live feed (abuse.ch URLhaus/Feodo) or real CVE records. Note the corpus provenance (source PCAP + hash, Suricata version, ruleset).

Dimension Developing Proficient Exemplary
Usefulness A toy that re-implements a one-liner Solves a real triage/enrichment task you'd reach for Fills a real gap; handles a workflow end to end
Typed boundaries Untyped; trusts input pydantic models validate input and LLM output; pyright clean Exhaustive typed boundaries; invalid states unrepresentable
Concurrency Sync loop, or a race Async enrichment with bounded concurrency + backoff Handles rate limits, retries, and partial failure gracefully
Tests & eval None, or happy-path only pytest + a pydantic-evals scorecard with a CI regression gate Property tests (hypothesis) fuzz the validator; gate blocks regressions
Safety shell=True; secrets hardcoded No shell injection; secrets via pydantic-settings/env; supply chain pip-audited Red-teamed against itself; residual-risk note
Ownership of AI code Pasted AI output unread Write-up names what AI generated vs. what you changed and why Demonstrates a caught bug/risk in generated code you fixed and explained

AI & automation

This is the track where "AI authors → you review → you own it" becomes a daily habit — and gets sharp teeth. The workflow is spec-driven development: you write the spec for each sift increment, the copilot implements it, and you review the implementation against the spec. Each module targets a copilot failure-class: unvalidated input, concurrency races, subtly-wrong types, shell=True injection, resource leaks, dependency risk, prompt injection. The competency isn't typing the code the copilot already writes; it's specifying it precisely, directing it, and catching it where it reliably fails.

Spec-driven development is taught as a provider-agnostic pattern (spec → implement → verify against the spec → eval). The concrete instance is openspec; GitHub's spec-kit is the mainstream Python-native alternative — use either, the discipline is the point.

Standards & further reading

Comments

Sign in with GitHub to comment. Choose the type: Feedback (errors or suggestions on this page) · Hints (help for fellow learners — no spoilers) · General (anything else).