How fast can semantic policy be? Latency budgets for governing agent actions

A semantic policy layer has to decide in single-digit milliseconds on the path where an agent calls a tool, or it becomes the thing the team turns off. That budget is reachable only by keeping the hot path deterministic and local, and moving model-based judgment to where waiting is already normal: monitor mode, the push, the pull request, the deploy.

This post covers Claude Code's hooks as documented in November 2026, OPA 1.x with Rego v1 syntax, the MCP stdio transport, and small local models run through llama.cpp.

The budget is per tool call, not per session

An agent's loop is mostly serial, so latency stacks. Forty tool calls at one second per check adds forty seconds to a job a developer is watching, where two milliseconds per call adds eighty milliseconds. A model call takes one to five seconds, so two milliseconds is noise, while two seconds adds forty percent to a five second call and triples a one second one.

Four tiers, four budgets

Tier	Decision source	Runs in	Budget per call	Needs network
T0 Deterministic	Policy as code (Rego) and pattern checks on the call	Agent hook path, in-process or a warm local daemon	0.2 to 2 ms	No
T1 Sequence	Run-state automaton over the session's history	Same process, one state per session	5 to 50 microseconds added	No
T2 Semantic	Small local model judging the call and its context	Same host, off the critical path or at a boundary	0.3 to 5 s on CPU, tens to hundreds of ms accelerated	No
T3 Boundary	Diff review, test evidence, session-wide verdicts	Push hook, pull request, CI, deploy gate	2 to 30 s	Yes

T0 and T1 run synchronously, T2 runs in monitor mode and at T3, and T3 is where enforcement lands: the model gets a vote without a veto on every call, and a veto where seconds are already being spent. These are budgets to hold, not measurements, so replace them with your p99.

Where the milliseconds go

Claude Code fires PreToolUse before a tool call executes and accepts a decision that can deny it (hooks reference). Adapters map that payload onto the policy input, so tool_name and tool_input.file_path arrive as input.tool and input.params.path. Parsing the payload, a round trip to a warm local daemon, evaluating a linear policy and queuing the decision to an audit spool should come to roughly 0.2 to 2 milliseconds.

Do not spawn a process per call: budget a few milliseconds of fork and exec for a shell plus a binary before any policy runs, where a resident daemon on a local socket pays a fraction of that. And do not put the cloud in the decision path: queue the decision to a local spool, since a synchronous audit sink puts the round trip back in.

Hook defaults hint at expected cost: 600 seconds for a command hook, 30 for a prompt hook that sends a single-turn prompt to a model. Those are ceilings, not targets, but a model in the hook path is budgeted in seconds.

Why the deterministic tier stays fast

OPA's performance guidance puts a microservice authorization budget "in the order of 1 millisecond" and describes a linear fragment engineered to evaluate in near constant time, with rule indexing narrowing which rules are evaluated, so cost tracks the indexed lookup rather than every rule that could have matched (policy performance).

The rule's shape matters more than the engine. This policy denies one class of call, matching paths by component so /repo/.env is denied as well as ./.env:

package coding.tools

# input is adapted from the hook payload: tool, params.path.
default allow := false

allow if {
  input.tool == "Read"
  not secret_path(input.params.path)
}

secret_path(p) if {
  some part in split(p, "/")
  part == ".env"
}

secret_path(p) if {
  some part in split(p, "/")
  part == ".ssh"
}

Classify paths with a small component set rather than a regex sweep per rule, and pass metadata instead of file contents: the engine needs a classification and a path, not the bytes of a private key. Evaluation stays flat as repositories grow, and secrets stay out of the policy process. OPA's agent tool calling page describes the same split.

Why a cheap check can be smarter than it looks

The rules that catch real incidents are about order: "after reading a secret, do not allow network egress", or "after a web search, require a passing review before a push". A per-call allowlist cannot express either, because each call is legal on its own.

Sequence state does, without a model. Keep a small, bounded record per session, update it on every call, and compare positions rather than booleans wherever order decides the answer:

state    : secret_read = false, search_seq = 0, review = none, review_seq = -1
events   : read(path)     -> secret_read = true if path in secret set
           search(query)  -> search_seq = search_seq + 1
           review(result) -> review = result, review_seq = search_seq
decisions: push, and secret_read, and target outside this repo           -> deny
           push, and search_seq > 0,
             and (review != passed or review_seq < search_seq)           -> deny
           everything else                                                -> T0 rules

Counting searches is what makes the rule correct: a review finished before the latest search has review_seq < search_seq, so it no longer satisfies the push, where booleans alone would still read as reviewed. That is one map lookup and two integer comparisons, exact and explainable without a transcript.

A model reaching the same verdict has to read the session history, infer that a file with credentials was read, and infer that the push would leave the host: a second or three on that budget. State is cheaper than inference, and more accurate: the rule says what happened, the model can only guess. Write that state into the audit record, since a deny without it is unarguable a week later.

The model tier: slow, and worth it where waiting is free

Deterministic rules cover the calls you thought about, not the open set: whether a diff hardcodes a credential, or whether a command is assembled into something else. That is what a small local model is for; there is a worked policy on the semantic policy examples page.

On CPU it does not fit a synchronous hook: prompt processing dominates, so a thousand tokens of context is seconds before the first token. With a one to three billion parameter model at four bit quantization through llama.cpp, budget roughly 0.3 to 5 seconds per judgment, a spread that model size, quantization and prompt length move, so measure on the hardware you plan to use.

In monitor mode it judges every call asynchronously, recording verdicts without blocking: detection coverage and false-positive data at no latency cost. At the boundary, a two to four second check is invisible next to a CI run.

Acceleration can move the budget by close to an order of magnitude: plan on tens to low hundreds of milliseconds for a short judgment on a GPU or Apple silicon, close enough to inline for more classes of call. It also has the least predictable tail, so the deterministic tier has to be complete on its own. Verdicts drift across versions, so pin the model and record its version with the decision (exporting and verifying the chain).

What a gateway or cloud LLM judge adds

A gateway or hosted judge puts the decision on the far side of a network: a TLS handshake, a round trip, provider queueing, inference and retries, plus the provider's p99. Budget single digit milliseconds for a same-region round trip and tens cross-region, tens to a couple of hundred milliseconds for a classifier, and a few hundred milliseconds to a couple of seconds for a generative judge. Fail open, and policy stops when the network does; fail closed, and the agent stops when the provider does.

Gateways remain the right tool for much of this: one chokepoint for remote MCP traffic and model calls, and policy in one place with nothing to deploy on a laptop. What they miss, unless traffic is routed through them, is a stdio MCP server, which the MCP transport spec defines as a subprocess talking over stdin and stdout on the same machine, nor local file reads or shell commands.

How to measure your own numbers

opa eval --data coding.rego --input samples/policy-input.json \
  --profile --count 100 'data.coding.tools.allow'

samples/policy-input.json is the adapted input, not the raw payload. Report p99, not the mean, measure with the audit sink enabled on the oldest laptop, and include cold start.

How Alyria splits the tiers

Alyria Beacon's local policy service evaluates Lyra policy on the hook path itself, deterministically, with no cloud round trip and no dependency on being online (Lyra policy as code, Alyria Beacon): T0 and T1, policy as code for the call in front of it, and run state for the order of calls in a session.

Model-based judgment runs off the critical path, in monitor mode while a team builds a baseline and at the boundaries where a check can wait. Decisions are signed and streamed to the audit chain after the fact, which decouples the cost of evidence from the cost of the decision.

Every latency claim, ours included, comes down to p99 on your fleet, with your policy, your audit sink and your slowest machine.