A new evaluation from METR, an AI safety and measurement nonprofit, is drawing attention for what it reveals about where today's most advanced models excel—and where they still fall short. The organization's latest update to its widely discussed time horizon plot confirms that frontier AI systems are getting better at an exponential rate on tasks with clear, automatic success criteria. At the same time, the report highlights a persistent weakness: models struggle with open-ended work that depends on judgment, taste, and hard-to-measure outcomes.

What Happened

METR refreshed its time horizon plot, a benchmark that rates AI systems by the length and difficulty of tasks they can complete. In the original version, researchers measured difficulty by timing how long human testers needed to finish each task, then tracked how long a task an AI could handle. The earlier plot suggested capability was doubling roughly every four months.

The updated report expands the task set to include longer, more ambiguous challenges. Some new assignments ask models to design AI evaluations that METR staff could actually use, while others require scoring philosophical arguments. On these judgment-heavy tasks, even top models did not meet the bar. None of the AI-generated evaluations were deemed useful by METR researchers, and models underperformed humans on philosophical reasoning exercises.

The report arrives amid broader debate about AI-driven research acceleration. A separate, heavily redacted METR document from earlier this month suggested AI might be doubling the pace of progress at Anthropic, raising questions about recursive self-improvement. The new evaluation does not settle that debate, but it does show that automating research and development remains constrained by the difficulty of measuring and training on tasks without clear right-or-wrong answers.

Why This Matters

Much of recent AI progress has been driven by reinforcement learning on verifiable domains such as coding and mathematics, where success can be checked automatically. That approach works less well when success is subjective or takes a long time to confirm—exactly the kind of work involved in choosing research directions or designing new experiments.

If AI is to automate meaningful portions of AI research and development, it must get better at these fuzzy, judgment-laden tasks. METR's inclusion of them in its benchmark signals that the field is beginning to grapple with evaluation challenges that go beyond leaderboard scores. Experts interviewed for the story welcomed the shift, though they cautioned that drawing firm trend lines from sparse, hard-to-score data remains difficult.

Key Takeaways

  • METR's updated time horizon plot shows rapid, exponential improvement on verifiable tasks, with capable models handling longer assignments over time.
  • Frontier models still lag humans on open-ended, judgment-heavy work such as designing AI evaluations and assessing philosophical arguments.
  • No AI-designed evaluation in the new suite met METR's internal standard for research use.
  • Progress on easy-to-verify tasks like coding does not automatically translate to success on ambiguous, real-world research tasks.
  • Because hard-to-verify tasks are expensive and time-consuming to score, METR includes relatively few of them, making long-term projections uncertain.
  • Outside researchers say the move toward broader evaluation is promising, but emphasize that much more work is needed before conclusions can be drawn about automated AI research and development.

Conclusion

METR's latest report paints a nuanced picture of AI's trajectory. Models are undeniably getting stronger at tasks with clear metrics, yet they remain uneven when the path to success is unclear. For anyone tracking the prospect of automated research and development, the takeaway is straightforward: raw capability on coding benchmarks is not the same as research judgment, and closing that gap may prove harder than previous progress curves suggest.