What changed

A new report from the AI evaluation nonprofit METR updates its time horizon plot, which tracks AI progress by task difficulty. The report adds more tasks, including those that require judgment, which were previously excluded due to their difficulty in scoring. METR evaluates models by comparing their performance on tasks to human completion times, with the original plot showing AI systems improving rapidly on verifiable tasks.

What this means in practice

The updated report includes tasks that are more open-ended or require human judgment, such as designing AI evaluations or scoring philosophical arguments. METR researchers note that these tasks are harder to evaluate and score, and the new report includes fewer of them due to the challenges in designing and quantifying them. The most advanced models, such as Opus 5.5, show some improvement on these tasks, achieving 'borderline above the bar' performance, but they are not yet usable for METR's research. The report also highlights that progress on verifiable tasks continues, but it remains unclear how quickly models might improve on more complex, judgment-laden tasks.

Limitations and availability

The new report faces challenges in accurately measuring performance on open-ended and judgment-heavy tasks due to the difficulty in designing and scoring them. Completion times for these tasks are mostly estimates, as paying people to complete them is prohibitively expensive. Additionally, some tasks, like scoring philosophical arguments, rely on limited expert input, which may affect the reliability of scores. METR acknowledges these limitations and notes that predicting progress on such tasks remains difficult. The report does not specify when these tasks will be fully integrated into the evaluation framework or how they will be scored in the future.

Source facts for METR Updates AI Task Evaluation to Include Judgment-Based Challenges: METR evaluated models by having them attempt a large suite of tasks, each of which had also been completed by humans. For the same reason, it’s tough to evaluate AI performance on these tasks, so METR’s original time horizon plot didn’t include any of them. But it’s precisely this type of task that will prove essential if AI systems are to automate research and development.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. mittr: The most misunderstood graph in AI just got a major update