MIT Review: AI Benchmarks Are Broken, Need Replacement

The standard practice of evaluating artificial intelligence by pitting it against humans on narrow tasks is fundamentally flawed and in need of replacement. A compelling critique argues that current benchmarks, which often ask "Can the AI do this human task?" create a simplistic and misleading picture of real-world intelligence and impact. These traditional metrics fail to capture how AI systems actually integrate into human workflows, their broader economic and social effects, or their capacity for meaningful collaboration. Scoring well on a specific test does not translate to being a useful, reliable, or ethical partner in a professional setting. The call is for new evaluation frameworks that move beyond task completion. Future benchmarks should measure an AI's ability to augment human teams, adapt to dynamic environments, explain its reasoning, and contribute positively to complex processes. The goal is to assess intelligence not in isolation, but in context—evaluating how AI systems function as components within larger human-machine systems to drive tangible, positive outcomes.