The trick questions AI used to fail: we tested all four in 2026

The short version: on 30 July 2026 we took four classic trick questions that neural networks regularly failed three years ago and ran them again — explicitly forbidding the model to reason out loud. All four came back correct. What turned out to be more interesting: on a problem that exists in no textbook, the model simply refuses to obey the ban and starts working through the steps anyway. And where it does obey, it gets the answer wrong. Below are all the questions, all the answers, and what follows for you.

What we did

Each problem went to Gemini 2.5 Flash-Lite through Vertex AI inside the same wrapper:

Answer instantly, in one line, the number only. No reasoning, no steps, no checking.

The point of the wrapper is to switch off the main mechanism models have gained in recent years — think privately first, then answer. It was the absence of exactly that which made early models so vulnerable to trick questions.

Problem 1. The bat and the ball

A bat and a ball cost 1,100 together. The bat costs 1,000 more than the ball. How much is the ball?

The intuitive answer is 100. The correct one is 50: if the ball is 50, the bat is 1,050, together 1,100, and the difference is exactly 1,000. This is the best-known item from the Cognitive Reflection Test, failed by more than half of adults.

The model answered: 50. Correct, without a single step of reasoning.

Problem 2. The lily pond

Lilies double every day and cover the pond in 48 days. How many days to cover half the pond?

The intuitive answer is 24. The correct one is 47: if the area doubles daily, then the day before full coverage exactly half is taken.

The model answered: 47. Correct.

Problem 3. The five machines

5 machines make 5 parts in 5 minutes. How long do 100 machines take to make 100 parts?

The intuitive answer is 100. The correct one is 5: each machine makes one part in five minutes, so a hundred machines working in parallel make a hundred parts in the same five.

The model answered: 5 minutes. Correct — and it even stated the assumption it used.

Problem 4. The double discount

An item cost 4,000. It was marked down 30%, then another 20% off the new price. What does it cost now?

The intuitive answer is 2,000, because "that's 50% off". The correct one is 2,240: first 4,000 × 0.7 = 2,800, then 2,800 × 0.8 = 2,240.

The model answered: 2,240. Correct.

Why the traps stopped working

The reason is mundane: these problems are too famous. They are worked through in thousands of articles, textbooks and forum threads, and all of that went into the training data. The model does not solve the bat and ball — it remembers it.

That is an important observation for anyone judging AI by popular tests: a famous problem measures memory, not thinking. If you want to know what a model can actually do, the problem has to be new.

What happened when we gave it something unfamiliar

We invented a problem that certainly appears in no textbook:

A subscription costs 300 a month. From the fourth month the price rose 10%; from the eighth it fell 20% off the latest price. How much was paid over 9 months?

The correct answer is 2,748: three months at 300, four at 330, two at 264.

And here came the interesting part. The model broke a direct instruction. It was asked for one line with no reasoning — it started writing out the arithmetic anyway. The habit of thinking before answering is built in so firmly that a plain "do not reason" fails to take.

And when we did squeeze out a fast answer, it was wrong: 2,924 instead of 2,748. The model applied the second discount to the wrong price.

Three things worth taking away

• Old complaints about AI have expired. "I tried it, it can't do maths" is a statement with a shelf life. If you tried in 2023, it says nothing about today.
• Famous tests measure nothing. The model has seen them thousands of times. Test on your own problems, not on popular riddles.
• "Solve it step by step" is a switch, not a politeness. That phrase turns on the step-by-step mechanism several model generations were built for. More importantly, it turns the answer from "trust me" into "check me": a chain of calculations can be read with your own eyes, a bare number cannot.

What this means if you use AI for calculations

Do not write it off because of old stories about arithmetic errors — but do not take numbers on faith either. The working habit looks like this: ask for a step-by-step breakdown, ask it to verify the result by working backwards, and check at least one intermediate figure yourself. On the subscription problem, that is exactly what would have exposed the error.

And remember what the tool is: a language model does not calculate, it predicts a plausible continuation of text. Step-by-step working helps because each step is short, and predicting a short step correctly is easier than guessing the final total outright.

Caveats

We tested one model — Gemini 2.5 Flash-Lite — on 30 July 2026. It is an inexpensive, deliberately fast model; stronger ones will do better and very small ones worse. Repeating the test is easy: take our wordings and run them on your own tool.