LLMs and Reasoning

Large language models (LLMs) like GPT-4 are not great with reasoning. In fact they often get very simple puzzles wrong.

Let’s dive into why this happens and what it means.

Pattern Recognition vs. Logical Deduction

LLMs excel at recognizing and generating patterns based on vast amounts of text data. However, reasoning tasks require a completely different skill set: step-by-step deductive logic.

Think of it this way: recognizing faces in a crowd is vastly different from solving a Sudoku puzzle.

LLMs are fantastic at the former but struggle with the latter.

Training Data Limitations

The training data for LLMs includes a broad range of text from the internet, books, and other sources. While this data set contains examples of logical reasoning, it is not specifically optimized for solving formal logic problems.

It’s like learning to play chess by watching games without ever understanding the rules. LLMs mimic logical patterns they’ve seen but aren’t built to solve logic problems from first principles.

Context and Interpretation Challenges

Reasoning tasks often hinge on precise understanding and interpretation of the problem’s context. LLMs, despite their sophistication, can misinterpret the problem or miss subtle nuances, leading to incorrect conclusions. It’s like having all the pieces of a jigsaw puzzle but not knowing how they fit together because you’re missing the picture on the box.

Implicit Computation Limits

When LLMs process information, they perform computations implicitly through their network architecture rather than explicitly following logical rules. Think of it like solving a complex problem based on your intuition and past experience rather than following a detailed, step-by-step plan.

These models use patterns learned from extensive data to make educated guesses, which is known as “implicit computation.”

This approach allows them to handle a wide variety of tasks quickly and efficiently.

However, because they’re not following a strict set of logical rules, they can sometimes make errors, especially when precise and exact answers are needed. It’s similar to when you might rely on your gut feeling to make a decision – it’s often effective, but there are times when a more detailed analysis might provide a better outcome.

The implicit nature of their computations can lead to errors when precise logical steps are required.

Overfitting to Language Patterns

LLMs can overfit to common language patterns. You can see a great example of this below, where every model fails a simple puzzle.

LLMs are very good at generating text that sounds coherent and plausible within the patterns they’ve learned.

This is why they can be dangerous in the hands of someone who isn’t a subject matter expert, or someone doesn’t check the output.

Aside: there’s a famous case from 2023 of a lawyer not checking LLM output: Steven A. Schwartz, a lawyer from the firm Levidow, Levidow & Oberman, used ChatGPT to conduct legal research for a personal injury lawsuit against Avianca Airlines in the Southern District of New York. Schwartz cited six cases in a legal brief that were later found to be completely fabricated by ChatGPT. That’s the problem with plausible sounding output.

Reasoning tasks often deviate from standard language patterns, requiring structured, sometimes unconventional logic.

This can trip up LLMs, causing them to produce text that seems logical on the surface but fails under scrutiny.

Understand the limitations

While LLMs like GPT-4 have shown impressive capabilities in many areas, they are not infallible, especially when it comes to strict logical reasoning. For tasks that require rigorous logical deduction, specialized tools or algorithms designed explicitly for logic and reasoning are often more reliable.

LLMs have revolutionized many aspects of how we interact with technology, but they have their limitations. Understanding these limitations allows us to use these tools more effectively, knowing when to rely on their strengths and when to turn to other solutions for more specialized tasks.

Real-World Example

To illustrate these points, here’s an example showing every major frontier model (i.e. the current state-of-the-art LLMs from OpenAI, Google, Anthropic and Meta) failing to reason through a simple problem. These examples highlight the limitations discussed and show why LLMs struggle with tasks that require simple logical deduction.

Anthropic’s newly released Claude 3.5 performed the best, realising its mistake when prompted:

Claude 3.5 Sonnet

ChatGPT 4o

Meta.ai

Gemini

Code-Llama-34b

I tested Code-Llama-34b, a coding specific model, to see if it would do any better:

You can see from this last one how the LLM thinks it recognizes the pattern of a classic puzzle here, which leads it astray.

My Conclusion

While LLMs like Claude and GPT-4o demonstrate impressive capabilities, they are not without their flaws. For tasks that require reasoning and precise logical deduction, specialised tools or algorithms are often more reliable.

LLMs have transformed many aspects of how we use technology. Their ability to process and generate human-like text has changed fields from customer service to creative writing.

But we must know and understand their current  limitations. Understanding helps us use LLMs more effectively, knowing when to check their output with a fine-tooth comb, and when we might need to choose other tools for the task at hand.

This balanced approach allows us to fully benefit from what LLMs offer while ensuring we use the best tools for each task.

It’s also crucial to realise that these current weaknesses are being addressed and improved every day.

Just because they struggle with reasoning today doesn’t mean they will tomorrow!