What ARC-AGI Was Testing
François Chollet introduced ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) in 2019 as a benchmark specifically designed to measure a kind of intelligence that large language models were believed to lack: the ability to recognize novel abstract patterns and apply them to new instances with very few examples. The tasks look simple — small grids of colored cells, input-output pairs showing a transformation rule, and a question asking you to apply that rule to a new grid. They're hard for AI because they require identifying the right abstraction from limited examples rather than applying memorized patterns.
The benchmark was designed as a measure of "core knowledge" reasoning — the kind of flexible, contextual pattern recognition that humans perform automatically but that LLMs trained on text had consistently failed at. Early GPT-4 scored around 33% on ARC-AGI. Claude and Gemini in their early versions fared similarly. The benchmark became a convenient symbol for "the thing AI can't do" in popular science writing about AI limitations.
o3's 87.5% score under the competition's standard conditions (and 91.5% under high-compute conditions with more inference time) changed the narrative substantially. It did not, as some headlines suggested, "solve AGI." It demonstrated that a specific approach — extended chain-of-thought reasoning with significant test-time compute — can handle the kind of abstract reasoning ARC-AGI tests much better than previous approaches.
How o3 Works Differently
The o-series models from OpenAI are trained and operate differently from standard instruction-following models. Rather than producing output tokens directly from the input, they generate an extended internal reasoning trace — a chain of thought that works through the problem step by step before producing the final answer. This reasoning trace is not shown to the user by default, but it is where the model's actual "thinking" happens.
The key innovation is that this reasoning trace is itself trained through reinforcement learning rather than purely through supervised learning on human-written chains of thought. The model learns what reasoning strategies lead to correct answers across a large set of problems, and develops its own internal heuristics that don't necessarily look like human problem-solving when read literally but produce correct answers more reliably.
For ARC-AGI specifically, o3 appears to be using its reasoning trace to enumerate possible transformation rules, test them against the provided examples, discard rules that don't fit, and converge on the rule that explains all the examples before applying it to the test case. This is a form of program synthesis through search — which is computationally intensive, explaining why the high-compute version scores higher than the standard version. More compute allows more hypotheses to be explored.
The Cost of Reasoning
The ARC-AGI performance came at a price. OpenAI's reported cost for o3 to score 87.5% on ARC-AGI was approximately $3,500 for the full evaluation — roughly $17 per task. At high-compute settings, the cost for the 91.5% result was higher still. For a benchmark of 400 tasks, this represents a per-task compute cost that is orders of magnitude higher than standard LLM inference.
This cost reality shapes where o3 is and isn't appropriate for production use. For tasks where correctness is critical and cost is secondary — safety analysis, mathematical proof verification, complex code debugging — the reasoning compute overhead is justifiable. For high-volume, latency-sensitive applications, o3's inference cost and latency make it the wrong tool regardless of its capabilities.
OpenAI released o3-mini alongside o3 as a significantly cheaper and faster variant with a much more favorable cost-quality tradeoff for most practical applications. The mini variant scored significantly lower on ARC-AGI (around 73%) but performed competitively on coding and mathematical benchmarks at a fraction of the compute cost. For most production applications that benefit from reasoning models, o3-mini is the more appropriate choice.
What This Means for Application Development
The practical implication of o3-class reasoning models for application developers is the same as it was for o1 — with the volume turned up. Tasks that previously required carefully engineered prompt chains, external verification steps, or human review because standard models were unreliable can now be handled end-to-end by reasoning models with higher reliability. Code generation, mathematical computation, data analysis, and multi-step logical deduction all benefit from the reasoning trace approach.
The cost and latency implications mean that reasoning models should be used selectively rather than as universal replacements for standard models. The emerging pattern in 2025 is a tiered model architecture: fast, cheap models for simple classification and generation tasks; medium-tier models for standard question-answering and document processing; and reasoning models reserved for high-stakes, complex tasks where their error rate reduction justifies the cost premium.
The ARC-AGI result is significant primarily as a signal about the trajectory of reasoning capabilities rather than as a direct product feature. The research path it validates — scaling test-time compute through extended chain-of-thought reasoning — is now confirmed to work substantially better than anyone expected two years ago. The capabilities that this path will enable in the next two years are genuinely uncertain and worth tracking closely.