In the race to build intelligent machines, large language models (LLMs) have emerged as the most impressive illusionists. They write code, pass exams, and even simulate conversations with eerie fluency. But beneath the surface lies a persistent and critical problem: they don’t understand what they’re doing—and they’re nowhere near true reasoning.
Recent research from Apple and others pulls back the curtain on these impressive text generators, revealing that LLMs, despite their size and sophistication, rely on pattern recognition, not genuine logic or understanding.
🍏 Apple’s Wake-Up Call: Pattern Matching ≠ Reasoning
In their GSM-Symbolic benchmark, Apple researchers tested leading LLMs on simple math problems with minor tweaks—swapping numbers, changing variable names, adding irrelevant clauses. The result? Performance dropped dramatically, often by over 10%. The conclusion was blunt:
“We found no evidence of formal reasoning… Their behavior is better explained by sophisticated pattern matching.”
Even subtle changes broke models that otherwise seemed competent. It’s not that they didn’t have the knowledge—they just couldn’t apply it in a reliable, structured way.
A broader Apple overview titled “The Illusion of Thinking” reinforces this: LLMs fall apart on complex reasoning chains. They appear smart in the short term, but lack the architecture for scalable, logical thought (Daring Fireball, 2025).
🧠 Chain-of-Thought: Faithful Explanations or Polished Fiction?
Even techniques designed to boost reasoning, like Chain-of-Thought (CoT) prompting, are now under scrutiny.
A study titled Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (arXiv:2305.04388) shows that CoT explanations often don’t reflect the model’s actual reasoning process. Instead, they produce plausible-sounding justifications after the fact, masking biased or misled decision-making.
For example, simply reordering multiple-choice options can influence the model’s answers—yet the explanation provided does not acknowledge this manipulation. The model “rationalizes” its output, regardless of the real cause.
In essence, LLMs are not only failing to reason—they’re confidently wrong and misleading about why.
🧬 Looking Inside: Circuit Tracing Offers a Glimmer of Hope
There is a growing effort to understand how LLMs actually work under the hood. New techniques like circuit tracing and attribution graphs aim to expose the internal computational steps during inference.
Two important papers by Anthropic researchers demonstrate this:
- Circuit Tracing: Revealing Computational Graphs in Language Models (transformer-circuits.pub)
- On the Biology of a Large Language Model (transformer-circuits.pub)
These tools have helped explain phenomena such as hallucinations, prompt refusals, and jailbreak vulnerabilities in Claude 3.5. They also offer a rare window into how concepts are encoded and activated.
It’s a slow and painstaking process—but a necessary one.
🧮 Reasoning Bottlenecks: It’s Not the Data, It’s the Design
A second study, Lost in Transmission: When and Why LLMs Fail to Reason Globally (arXiv:2505.08140), digs into the why. It introduces the Bounded Attention Prefix Oracle (BAPO) model to explain LLMs’ inability to perform global reasoning—the integration of information across a long context.
The issue isn’t memory or training. It’s bandwidth: LLMs can only transmit a limited amount of information through their layers. As a result, they succeed on “BAPO-easy” tasks (e.g., simple lookups) but fail dramatically on “BAPO-hard” tasks (e.g., graph traversal or multi-step logic), even if the problem fits in the context window.
This limitation isn’t going away with more data. It’s architectural.
🚧 Understanding the Limits: Why LLMs Can’t Truly Reason—Yet
The hype around LLMs has often outpaced their capabilities. While they’re undoubtedly useful for many tasks, complex reasoning is not one of them.
Apple’s research, along with independent studies, makes one thing clear: today’s LLMs are brilliant mimics, not thinkers. They process language with remarkable fluency but cannot navigate logic or abstraction in a consistent, trustworthy way.
Even their explanations are often disconnected from the actual forces guiding their outputs. And until we understand those internal forces—until we know how they “think”—we can’t improve them.
The paper Lost in Transmission: When and Why LLMs Fail to Reason Globally drives this point home: true reasoning will require a profound architectural shift—not just more training data or longer prompts, but a fundamental redesign of how these models process and integrate information.
The road to genuine machine reasoning is long. We haven’t even taken the first real step.
For a broader perspective on what LLMs are—and more importantly, what they aren’t—you might also explore “The Nature of LLMs’ Intelligence” by SynapseDx. It dives deeper into the philosophical and functional limits of LLM behavior, arguing that their output is best understood as simulation, not cognition. It’s a thoughtful complement to the technical insights shared above.
🔗 References
- Apple Machine Learning Research. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, 2024.
- MacRumors. Apple Study Reveals Critical Flaws in AI’s Logical Reasoning Abilities, October 2024.
- Daring Fireball. Apple Researchers Publish Paper on the Limits of Reasoning Models, June 2025.
- Lost in Transmission: When and Why LLMs Fail to Reason Globally, arXiv:2505.08140, 2025.
- Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, arXiv:2305.04388, 2023.
- Circuit Tracing: Revealing Computational Graphs in Language Models, transformer-circuits.pub, Anthropic.
- On the Biology of a Large Language Model, transformer-circuits.pub, Anthropic.
English 
French