Rollouts Are Decision Problems
A rollout reviewer should not merely detect anomalies. It should decide what the available evidence justifies doing next.
Most rollout systems are organized around signals: error rate, latency, saturation, logs, alerts, and perhaps a statistical detector.
That is necessary. It is not the decision.
A rollout review must answer a more consequential question: given the evidence, risk, uncertainty, and available actions, what should happen next?
An anomaly is not a verdict#
A metric can move for many reasons. The candidate may be harmful. The control may have changed. Traffic composition may have shifted. Telemetry may be delayed. A downstream dependency may be failing equally for both populations. The movement may be real but operationally irrelevant.
Conversely, a dangerous rollout may show no statistically dramatic anomaly because exposure is small, the affected path is rare, or the most relevant signal is not instrumented.
So the mapping is not:
anomaly → unhealthy
It is closer to:
observations + context + counterfactual + risk + evidence quality → action
The action might be continue, hold, roll back, gather more evidence, or escalate. A useful reviewer chooses among them rather than decorating a dashboard with adjectives.
Evidence has a shape#
A metric value alone is not evidence enough. A reviewer needs to reason about:
- the candidate and comparison populations,
- exposure and sample size,
- time windows and seasonality,
- telemetry freshness and coverage,
- changes in dependencies,
- known rollout events,
- and the cost of waiting versus acting.
This is why a report can be numerically correct and still misleading. If traffic only reached two percent, a green SLO does not prove the remaining population is safe. If telemetry disappeared, “no regression detected” may mean “we stopped seeing.”
A trustworthy reviewer separates three statements:
- What the evidence supports.
- What the evidence cannot resolve.
- What action the risk policy permits anyway.
The reviewer needs an authority model#
Not every conclusion should produce the same power.
An agent may have authority to pause a rollout when a hard invariant is breached, but only authority to recommend a rollback when evidence is ambiguous. It may freely run read-only queries but require approval to change traffic. It may automatically continue a low-risk ramp while escalating an equivalent signal on a critical service.
This is not a limitation of autonomy. It is the architecture that makes autonomy deployable.
Review is a long-running process#
A rollout is not one task. It is a sequence of changing states.
At the start, the reviewer establishes the candidate, policy, topology, expected duration, and baseline. At each checkpoint, it updates evidence and tests whether the next action is justified. When new information arrives, it revises the same durable review rather than generating an unrelated report.
The system therefore needs ownership of time and state: scheduled wake-ups, idempotent checkpoints, versioned artifacts, and a clear terminal condition. “Remember to check again” is not a timer. A prompt is not durable orchestration.
Intelligence begins after the verdict#
The most valuable data arrives later.
Did the rollout succeed? Was it rolled back for a reason the reviewer missed? Did an alert prove noisy? Did the team override the recommendation, and were they right? Which evidence changed the outcome?
By joining review-time reasoning with eventual production outcomes, the system can measure more than anomaly-detector accuracy. It can learn when evidence was insufficient, when policy was too strict, and which precedents matter for a new service.
That is rollout intelligence: not a growing pile of reports, but an improving decision system.
The standard to aim for#
A good rollout reviewer does not promise omniscience. It earns the right to make a recommendation.
It gathers the relevant evidence, preserves provenance, distinguishes absence of harm from absence of visibility, represents material uncertainty, respects its authority, and returns to the decision when the outcome becomes known.
The verdict is the smallest part of that system.
What matters is whether the system deserves to say it.