AWS released the Agent Evaluation Metric (AEM) to address the shortcomings of single-turn evaluation in multi-turn conversations. The new metric measures correctness at the turn level, enabling precise identification of errors that propagate across interactions. This marks AWS’s first major update to its evaluation framework for conversational agents since its initial release.
AEM evaluates correctness through two sub-metrics: truthfulness and completeness. It uses semantic similarity scoring to determine if predicted values align with expected outcomes, rather than relying on exact string matching. This approach accounts for variations in phrasing, such as 'New York City' and 'NYC', which are semantically equivalent.
The metric is designed to work with both response and action turns, providing a structured way to assess agent behavior across the full conversation trajectory. It decomposes quality into named sub-metrics that can be independently measured and recombined into a single score. This enables more granular analysis of performance and facilitates the addition of new evaluation dimensions in the future.
"One early error cascades through later turns, and turn-level evaluation isolates the root cause from its downstream effects," said the post. The framework allows for a more actionable score by identifying which sub-metric, truthfulness or completeness, drives a decline in performance.
The announcement follows AWS’s ongoing efforts to improve the reliability and transparency of conversational agents. The company emphasized that AEM provides a foundation for extending the evaluation framework to new dimensions, such as safety and instruction retention, without re-architecting the system.
AWS did not specify the exact threshold for semantic similarity scoring, noting that the value depends on the domain’s tolerance for false positives versus false negatives. The company also highlighted the need for further refinement and testing before broader implementation.
Source: awsml
