Drift Detection

Detect gradual performance changes by comparing metric trends, production examples, and simulation baselines through a deliberate monitoring workflow.

Drift is a sustained change in the inputs an agent receives, the behavior it produces, or the requirements used to judge it. The platform supplies metric results, analytics, run history, and simulation evidence that teams can use to identify these changes.

Drift response is currently a monitoring and release-management practice. The platform does not automatically turn a trend threshold into a promotion block or production rollback.

What Causes Drift

Input Drift

The mix of conversations changes. A scheduling agent tested mostly on routine requests may begin receiving multi-provider coordination calls, a new language mix, or questions caused by a policy change.

Performance Drift

Measured behavior changes over time. Task completion may decline, latency may rise, or a safety evaluation may worsen after a configuration, dependency, or model change.

Requirement Drift

The agent may behave consistently while the acceptance criteria change. Updated clinical guidance, organizational policy, or regulatory requirements can make an earlier test suite incomplete.

Available Evidence

Production Metric Trends

Configured evaluations produce per-interaction results that can be aggregated over time. Use the analytics views or APIs to compare equivalent windows before and after a release. Keep the metric definition and population stable so a scoring change is not mistaken for an agent regression.

Where filters are available, break the result down by service, channel, outcome, or another relevant dimension. A workspace-wide average can hide a regression concentrated in one workflow.

Conversation and Run Review

Open representative interactions behind a changed metric. Transcripts, executed actions, tool results, completion reasons, and best-effort trajectory evidence help distinguish an agent problem from input changes or an external dependency failure.

Simulation Baselines

Saved cases and suites provide repeatable checks for known behaviors. Compare current results with the acceptance criteria for the same cases, then add production examples that reveal a missing scenario.

Simulation and production populations are not automatically compared as a statistical blueprint. Teams decide which production segments correspond to which test cases and interpret the gap.

A Practical Response Process

1. Confirm the signal. Compare enough interactions and an appropriate time window to separate a sustained change from normal variance.
2. Locate the affected population. Narrow the change by service, channel, release, failure type, or another available dimension.
3. Inspect examples. Review successful and failed interactions, including tool and dependency outcomes.
4. Update the test set. Add cases for new inputs or failure modes and update criteria when requirements changed.
5. Validate a focused change. Run the affected suite and important safety guardrails before release.
6. Make an explicit release decision. An authorized user promotes, pauses, or restores a configuration based on the evidence.
7. Measure the same population again. Confirm that the target recovered without creating a new regression.

Alerts and Rollback

The documentation's earlier alert, promotion-gate, and automatic-rollback workflow is not a current platform capability. If a team needs automated notification, it can query the analytics APIs from its monitoring system and route a threshold breach through its normal incident process.

For a severe regression, operators should follow the organization's release procedure: stop or limit the affected workflow when appropriate, select a known-good version deliberately, and verify the result. Keep a human decision in this path, especially when a metric is noisy or a rollback could reintroduce an older safety problem.

Monitoring Checklist

• Choose a small set of outcome and safety metrics with clear owners.
• Record the baseline window, metric definition, filters, and released version.
• Use review windows that fit traffic volume; low-volume services need longer windows.
• Define investigation thresholds in the monitoring or operating process.
• Preserve representative interactions and convert new failure modes into simulation cases.
• Recheck the same measures after every material release.

Drift is not always a defect. Input drift can reveal a new use case, while requirement drift can reveal that the test suite needs to change. Investigate the cause before restoring an older configuration.