Improving AI Reasoning with Simple Instructions: A Comparative Study
This is a simple demo to show the reasoning capabilities – or lack thereof – of current language models. You can see how the “frontier” models like ChatGPT-4o and Claude-3.5-Sonnet outperform their smaller counterparts, but also how giving a simple instruction can vastly improve reasoning.
Table of Contents
Frontier models vs small models
Frontier models are advanced AI systems designed to push the boundaries of what’s possible with artificial intelligence. They typically have more parameters, larger training datasets, and enhanced algorithms that enable them to understand and process information more accurately and effectively than smaller models. These enhancements allow them to excel in complex reasoning tasks, making them more reliable for sophisticated applications.
Why are small models useful? They are typically, cheaper, and often faster than larger models. Their context window – i.e. working memory – is usually smaller to optimise for speed and cost.
However, you can also see how using the “take a deep breath” improves the reasoning capability of one of the smaller models.
The prompt we’re going to use is a simple puzzle:
“Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?”
The answer is that if Sally’s brothers all have 2 sisters, and Sally is one of them, then there’s one other sister, so Sally has one sister.
How do AI language models do with this puzzle? Let’s see:
GPT-4o-Mini: Sally has 2 sisters
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Sally has 2 sisters. Each of her 3 brothers shares the same sisters, which includes Sally herself and her 2 sisters.
Model: ChatGPT-4o-Mini
Claude-3-Haiku: Sally has 6 sisters
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Okay, let’s break this down step-by-step:
Sally has 3 brothers
Each of Sally’s 3 brothers has 2 sisters
To find the total number of sisters Sally has, we need to add up the sisters of each of her 3 brothers
So, let’s calculate:
Sally’s 1st brother has 2 sisters
Sally’s 2nd brother has 2 sisters
Sally’s 3rd brother has 2 sisters
Adding these up:
2 + 2 + 2 = 6
Therefore, Sally has 6 sisters.
Model: Claude-3-Haiku
Llama-3: Sally has no sisters
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
A classic lateral thinking puzzle!
Let’s break it down:
Sally has 3 brothers.
Each brother has 2 sisters.
Now, here’s the key insight: Sally is one of those sisters!
So, each brother has Sally as one of their sisters, and another sister (who is also Sally’s sister).
That means Sally has only 1 sister, which is herself!
In other words, Sally doesn’t have any sisters other than herself, because each brother’s “2 sisters” includes Sally herself.
So, the answer is: Sally has no sisters.
Model: Llama-3-70b-Groq
Mixtral gets it right!
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Sally has 1 sister.
Here’s the logic behind this:
We know that Sally has 3 brothers.
Then we are told that each of her brothers has 2 sisters.
Since all of her brothers share the same sisters, there is only one additional sister besides Sally.
Therefore, Sally has 1 sister.
It’s important to note that the number of sisters each brother has is shared among them, so it doesn’t mean that each brother has 2 different sisters.
Model: Mixtral-8x7B-Chat
GPT-4o gets it right
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Let’s start by analyzing the family structure given in the problem.
Sally has 3 brothers.
Each brother has 2 sisters.
From the second point, since each of Sally’s 3 brothers has 2 sisters, it implies there are only 2 sisters in total.
To clarify:
The 2 sisters referred to for each brother include Sally and her one other sister.
Thus, Sally has:
1 sister (excluding herself)
Therefore, Sally has 1 sister.
Model: ChatGPT-4o
Claude-3.5-Sonnet gets it right
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Let’s approach this problem step by step:
We know Sally is a girl and has 3 brothers.
The tricky part is that each brother has 2 sisters.
We know one of those sisters is Sally herself.
So, if each brother has 2 sisters, and one is Sally, there must be one other sister.
This other sister would be the same for all three brothers (just like Sally is a sister to all three).
Therefore, Sally has 1 sister.
To summarize: Sally (a girl) + 1 sister + 3 brothers = a family with 5 children (2 girls and 3 boys).
Would you like me to explain any part of this reasoning in more detail?
Model: Claude-3.5-Sonnet
Now, let’s add the take a deep breath instruction and see what happens:
Claude-3-Haiku: Sally still has 6 sisters
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Take a deep breath and work on this problem step-by-step.
Okay, let’s take this step-by-step:
Sally has 3 brothers
Each of Sally’s 3 brothers has 2 sisters
To find the total number of sisters Sally has, we need to add up the number of sisters each of her 3 brothers has
First, let’s find the number of sisters each brother has:
Each brother has 2 sisters
Now, let’s find the total number of sisters Sally has:
Sally has 3 brothers
Each brother has 2 sisters
3 brothers x 2 sisters each = 6 sisters
Therefore, Sally has 6 sisters.
Model: Claude-3-Haiku
No improvement from Claude-3-Haiku, but the deep breath instruction works wonders with ChatGPT-4o-Mini:
ChatGPT-4o-Mini: Sally has 2 sisters
Sally – a girl – has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Take a deep breath and work on this problem step-by-step.
To solve the problem, let’s break it down step-by-step:
Identify Sally’s siblings: Sally is a girl and has 3 brothers.
Determine the number of sisters each brother has: Each brother has 2 sisters. Since Sally is one of the sisters, we can deduce the following:
If each brother has 2 sisters and one of them is Sally, that means there is 1 other sister.
Conclusion: Therefore, Sally has 1 sister.
So, the answer is that Sally has 1 sister.
Model: ChatGPT-4o-Mini
Why are small models useful? They are typically, cheaper, and often faster than larger models. Their context window – i.e. working memory – is usually smaller to optimise for speed and cost.
