Chatbot latency profile

Where the guardrail pipeline spends its time

One chat turn, two architectures, measured against the real claude CLI with the project's own flags, models and prompts. Pattern A spawns three separate one-shot processes (input guardrail → main agent → output guardrail). Pattern B runs all three as sequential messages inside one persistent process. Each bar segment is a mean over 5 iterations.

Pattern A · turn

–s

3 processes, sequential

Pattern B · turn

–s

1 persistent process

B saves

–%

–

…of which cold-start

–%

the rest is warm-context speedup

Pattern A — three separate processes

guardrails: haiku · main: sonnet · cold start ×3

Each stage is its own cold claude -p. Cold start and a long time-to-first-token are paid three times.

Pattern B — one persistent process

sonnet (warm) · cold start ×1

Cold start is paid once, up front. The guardrail stages become cheap warm follow-up messages. Same time axis as above.

The main problem is not cold start — it is the guardrail's time-to-first-token

Consolidating three processes into one warm process cuts the turn roughly in half (– → –). But only about – of that saving comes from paying the ~2 s Node cold start once instead of three times.

The bigger loss in Pattern A is that each guardrail runs as a full cold inference: the haiku input guardrail spends – just reaching its first token (thinking + a fresh, un-cached system prompt on every spawn) for what is a one-word verdict. In Pattern B the same check, sent into an already-warm context, reaches first token in – and generates in well under a second.

The main agent's generation (–) is the irreducible core in both — it is real answer tokens. Attack the guardrail overhead, not the answer.

Raw data · per-iteration turn totals (ms)

Iteration	A · input grd	A · main	A · output grd	A · turn	B · cold	B · turn

Iteration 0 is a warm-up (disk / CLI cache cold) — shown muted, included in means.

Method & honest caveats

Harness: standalone Python, spawns the real claude CLI v2.1.212. Cold start = wall time from Process.Start to the CLI's system/init event; time-to-first-token and generation come from the CLI's own result event (ttft_ms, duration_ms). The app's own code was not modified.
Faithful to the app in flags, models and prompt shape. The prompts published in this repository are stand-ins of the same size and structure, not the measured application's own text.
Not apples-to-apples on the guardrail model. Pattern A runs the guardrails on haiku with --json-schema (as shipped). Pattern B cannot switch model or schema per message inside one process, so its guardrail stages run on sonnet with the instruction inline — yet they are still far faster, because the context is warm. This strengthens, not weakens, the conclusion.
Main agent is tool-less (no MCP server was stood up). Real turns that call tools add API round-trips on top of both patterns equally.
N = 5 (credits were limited). Time-to-first-token is high-variance; treat the shape, not the third digit, as the finding.

—