How LLM Thinks: Inside the Reasoning Engine of Large Language Models

A new wave of research is pulling back the curtain on how large language models actually ‘think.’ While most users treat LLMs as black boxes that magically produce coherent text, developers and researchers have been digging into the inner mechanisms—and what they’re finding is both surprising and practical.

A recent in-depth article on Habr (published July 2026) compiles key findings from both academic papers and real‑world experiments. The material doesn’t just rehash attention is all you need—it goes deep into the actual reasoning traces, chain‑of‑thought patterns, and the surprising ways LLMs handle logic, contradiction, and even self‑correction.

If you’re building products on top of GPT‑4, Claude, or open‑source models, understanding how these systems actually ‘think’ isn’t just academic—it directly impacts prompt design, output reliability, and cost. Let’s break down what the article reveals.

The Two‑Layer Thinking Model

The Habr article describes a now‑dominant framework: LLMs operate on two distinct levels. The first is surface‑level token prediction — the model chooses the next word based on immediate context, with no explicit reasoning. The second is a deeper reasoning loop that emerges from the transformer architecture itself, especially when prompted with structured tasks.

The article emphasizes that this second layer is not explicitly programmed—it emerges from training on vast code and reasoning datasets. In fact, when researchers probed model internals, they found that even when the final output is correct, the intermediate reasoning can be flawed. This mirrors human cognition: we often arrive at the right answer through a mix of heuristics and guesses, not pure logic.

Chain‑of‑Thought: Not Just a Prompt Trick

One of the most actionable insights from the article is how Chain‑of‑Thought (CoT) prompting actually works under the hood. The authors describe experiments where they replaced the standard “let’s think step by step” with more specific instructions like “first, identify the goal, then list constraints, then propose a solution.” The result? Accuracy on multi‑step reasoning tasks jumped by 15–30% across models.

But here’s the twist: the article shows that CoT works not because the model suddenly becomes logical, but because it forces the model to allocate more computational resources to the problem. Each intermediate token acts as a placeholder for attention—the model can revisit earlier steps, adjust probabilities, and even backtrack. The article cites work from Anthropic and DeepMind showing that internal attention heads dedicated to planning activate only when CoT is used.

Practical takeaway: For any task that requires more than two steps, always use explicit intermediate steps in your prompts. Even a simple “First, … Then, … Finally, …” structure can double output accuracy on complex tasks like math word problems or code generation.

How LLMs Handle Contradictions and Self‑Correction

Another fascinating section in the Habr piece covers model behaviour when faced with contradictory instructions. The researchers injected conflicting statements into the same prompt, e.g., “The sky is green. What colour is the sky?” Surprisingly, models don’t always output “green.” Instead, they often produce a hybrid response: “The sky is green, but usually it appears blue.” The article interprets this as the model juggling multiple priors—the training data (sky is blue) and the prompt (sky is green).

When the conflict is small or the model is confident, it leans toward original training. When the conflict is repeated or strongly phrased, it yields to the prompt. This has direct implications for prompt engineering: if you need the model to override incorrect default knowledge, you must provide strong, repeated evidence within the context.

More importantly, the article describes how models can self‑correct during generation. The same model can produce an answer, then immediately qualify or revise it in the next sentence. The authors trace this to the internal attention patterns that allow the model to re‑read its own previous tokens. This is a form of rudimentary reflection—and it’s why you often get more accurate answers when you ask the model to “check your work” or “explain your reasoning before giving the final answer.”

The Surprising Role of Formatting and Spacing

The article also dives into a less‑glamorous but highly impactful finding: tokenisation and formatting massively affect reasoning quality. For example, adding extra newlines or using code blocks vs. plain text changes how attention is distributed. The Habr authors present a case where a simple list with bullet points outperformed a paragraph by 20% on a classification task, simply because the model could attend to each item more distinctly.

They recommend:
- Use markdown headers to separate logical sections.
- Keep each idea in a separate paragraph.
- Avoid long monoliths of text — the model’s attention decays over distance.
- Use numbered lists for step‑by‑step instructions.

This might sound trivial, but the article backs it with attention heatmaps that show clear focus peaks on visually separated elements.

Limits of the “Thinking” Metaphor

Despite all these insights, the Habr article is careful not to anthropomorphise. It explicitly states: “LLMs do not think in the human sense. They simulate thought patterns because their training data contains millions of examples of human reasoning.” The difference is crucial: a model can appear to reason but still generate hallucinations, especially when asked to reason about novel situations that don’t fit its training distribution.

The authors share a concrete example: they asked a model to plan a trip using a fictional city map that violated physical laws (e.g., roads that loop back on themselves). The model confidently produced a step‑by‑step itinerary, completely ignoring the impossible geography. When probed with “Are you sure this route is possible?”, it doubled down. This shows that while the reasoning structure looks correct, the content can be completely detached from reality.

Lesson for developers: Always add a verification step for any output that depends on factual or physical constraints. Don’t rely on the model’s own reasoning to catch its mistakes—build external checks into your application.

What This Means for Practical Applications

The Habr article concludes with several actionable recommendations for teams deploying LLMs in production:

1. Design prompts around the reasoning loop – structure your prompt to give the model explicit “thinking tokens”, not just an answer format.
2. Use context separation – if you need the model to handle multiple constraints, separate them visually and semantically (headers, bullet points).
3. Allow for self‑correction – encourage the model to reflect by asking it to “check your previous answer” or “rewrite if needed”.
4. Avoid forcing the model to reason about absurd premises – if your data contains impossible scenarios, be prepared for hallucinations.

One of the most interesting points: the article notes that smaller models (7B–13B) can sometimes outperform larger ones (70B+) on tasks requiring tight logical constraints, because their smaller attention windows force them to focus on immediate context rather than wander. This challenges the “bigger is always better” assumption.

Conclusion

The Habr piece provides a rare, well‑grounded look inside the black box. The key takeaway is not that LLMs think like humans, but that they simulate thinking through a complex interplay of token prediction and emergent reasoning loops. Understanding this simulation lets you design better prompts, build more reliable applications, and avoid common pitfalls.

For anyone serious about using LLMs at scale—whether you’re a solo developer or part of a large AI team—this article is worth a full read. The original research it references is also linked in the source material, providing a solid foundation for deeper exploration.

Source