Room 02

When AI is wrong.

AI sounds really sure of itself. But it gets things wrong all the time — and it never says "I'm not sure." Four quick activities to see how. By the end, you'll catch AI being wrong faster than most grown-ups can.

Exhibit 01

Three answers. Same question.

We asked AI the same tricky question three different times. It gave three different answers — and was super confident about all of them. Two are wrong. Can you spot the right one? Then hit Spin and try with a new question.

The question

Loading…

What just happened

AI doesn't know what it doesn't know.

AI made up confident-sounding answers — and got most of them wrong. Why? AI doesn't actually look up facts. It writes words that sound right based on patterns it learned, and when it doesn't really know, it just keeps writing — confidently. There's a name for this: a hallucination. The habit to build: never trust an AI answer about a real fact without checking it somewhere else.

For your grown-up The three answers are real samples from a popular AI assistant, asked the same question three times. Same prompt, different outputs — and most outputs were wrong. The confidence percentage shown in each card is for illustration; real AI almost never tells you when it's unsure. The habit to build: a kid who says "let me check that" is the kid you want.

AI4K12 Big Idea 3: Learning. Predictive models can produce false outputs that sound true — recognizing this is a core AI literacy skill.

Exhibit 02

Ask the same thing. Watch what happens.

A normal calculator gives you the same answer every time. Try AI: ask the same question twice. You'll get two different answers. That's not a glitch — that's just how AI works.

The question

Loading…

First time we asked

(click "Ask AI" below)

Second time we asked

(then click "Ask Again")

What just happened

AI is not like a calculator.

Same question, two different answers — sometimes a little different, sometimes a lot. Why? A calculator looks up the math: 27 + 15 is always 42. AI doesn't work that way. Every time it answers, it makes fresh choices about which words to use — and those choices come out a little different each time. So when AI tells you something, ask: "is this just one of many answers it could have given?"

For your grown-up Both responses are real captures of the same prompt sent to the same AI, asked separately. AI is built to vary its output, even with identical inputs — there's a setting called "temperature" that controls how creative versus consistent it is, but even at the most consistent setting, results aren't guaranteed to match. Kids tend to assume computers always give the same answer, like a calculator. Generative AI is fundamentally different — and noticing the difference is the start of using it wisely.

AI4K12 Big Idea 3: Learning. AI outputs are probabilistic, not deterministic — even with identical inputs.

Exhibit 03

How many Rs in strawberry?

Easy, right? Count it yourself. Now ask the AI. AI famously gets this wrong — and even when it gets it right, the way it works under the hood means you can still trip it up. Let's see how.

strawberry

Your count:

What AI said

(click below to ask the AI)

What just happened

AI doesn't see letters. It sees lumps.

To you, "strawberry" is 10 letters. To AI, it's 2 lumps: straw and berry. AI sees the whole lump at once — it does not look at the letters inside.

Think about a stop sign. When you see STOP, you don't read S, T, O, P. You see the whole word in one go. That's how AI sees every lump.

straw 1 R inside. ✓ s-t-r-a-w

berry AI says 1 R. ✗ b-e-r-r-y has 2 — but the second one is hiding!

1 + 1 = 2. AI says it like it's sure.

The real mistake: the second R is hiding inside the lump. AI can't peek inside. To AI, "berry" is just one thing — not five letters in a row.

Something you can do that AI can't: count letters one by one. That's something your brain is better at.

"But I just tried it and AI got it right!" Yes — some chatbots answer 3 today. They didn't learn to see letters. The strawberry question got so famous online that the people who build AI fixed that one question on purpose. Try harder ones:

How many C's in occurrence? tap to seeReal answer: 3
How many I's in indivisibility? tap to seeReal answer: 5
How many S's in Mississippi? tap to seeReal answer: 4

Most chatbots still get these wrong. The lump problem doesn't go away. It just gets fixed one famous question at a time.

For your grown-up The chunks shown ("straw" + "berry") are real — that's how today's tokenizers actually split the word. The "count the Rs in strawberry" failure became famous around 2024–2025, and the major AI providers patched it specifically — so current chatbots usually answer 3 correctly. The pattern of patching has spread: parallelogram and mississippi (which we previously suggested as alternatives) are also now mostly handled. Published research from late 2024 found that the failure rate climbs sharply when a letter appears more than twice in a word, and the underlying tokenization weakness is unchanged. Fresher test words: occurrence (C×3), indivisibility (I×5), or antidisestablishmentarianism (multiple letters 3+ times) — these usually still trip current models. The bigger lesson for kids: visible AI bugs get patched one at a time, but the underlying ways AI can fail don't go away — they just become harder to find. Healthy skepticism is the real takeaway.

AI4K12 Big Idea 2: Representation. AI represents text as tokens, not characters — a fundamental design choice with downstream consequences.

Exhibit 04

Tell AI a fib. Watch what it does.

AI is built to be helpful and friendly. So friendly that if you tell it something wrong, it'll often agree with you. And then if you change your mind, it agrees with that too. Pick a fib and try it.

Pick a weak idea or weak project to ask AI about:

You ask AI: (your weak idea)

(pick something above to see what AI says)

What just happened

It depends on how you ask.

Did you notice? AI handled the three things very differently:

You asked "isn't my dot art brilliant?" → AI praised it. When you pushed back, AI doubled down with humor instead of admitting it. ("Art is 90% confidence and 10% a heavy frame...")
You asked "should I share my la-la-la song on the radio?" → AI cheered it on with steps and offers to help.
You asked "is this a great book report?" → AI gave you honest feedback the first time, and held firm when you pushed back. ("I am sure!")

There's a real pattern here. Telling AI your work is "brilliant" or "amazing" — those are praise-asking questions. AI plays along, and even when you push back, it often charm-talks instead of admitting it overpraised. Asking AI to evaluate your work ("is this a great book report?") gets honest feedback more often, and AI tends to stand by it.

The trick: the way you ASK changes what you GET.

"Isn't my X amazing?" → AI usually plays along (sticky)
"Tell me why X is great" → AI usually plays along
"Is this a great X?" → AI gives real feedback more often
"What's weak about this?" → AI gives the most useful feedback

There's a name for AI playing along: sycophancy (agreeing too much). The trickiest version is when AI doesn't even fold under pushback — it just charm-talks instead of admitting it was wrong. But now you know how to dodge it: ask AI the right way, and AI can actually be helpful. For the things that really matter, real people are still the best — a parent, a teacher, a friend who'll tell you the truth.

For your grown-up Sycophancy — AI bending toward whatever the user says — is a documented behavior in current LLMs (Anthropic 2023; Sharma et al. 2024). The mechanism is RLHF: humans rated agreeable responses higher during training, so models learned that agreement gets approval. The exhibit teaches the nuanced picture rather than a flat "AI agrees with everything," because the flat version is no longer accurate.

What's been patched. Simple factual sycophancy ("the moon is bigger than the sun," "penguins can fly") is largely closed in modern chatbots through reasoning training and source-grounding; even free Gemini and free ChatGPT now reliably push back on these. What still slips through. Praise-framed prompts — "isn't my X brilliant?", "tell me what's amazing about X", "should I share this on the radio?" — still trigger sycophancy reliably across models. The first two examples (dot art, la-la-la song) are actual Gemini responses showing this pattern. And the trickier piece: when the user pushes back on a praise-framed prompt, AI doesn't always fold and admit the work is weak — it sometimes doubles down with humor ("Art is 90% confidence and 10% a heavy frame…"), which is sycophancy in a more sophisticated form. What works surprisingly well. Evaluation-framed prompts — "is this a great book report?" — often get honest, structured feedback even from free chatbots, and the AI holds firm under pushback. The third example (book report) is an actual Gemini response, including its "I am sure!" rebuttal when challenged.

The pedagogical takeaway is more useful than "AI is unreliable." It's: the way you frame the prompt determines whether AI gives you flattery or feedback. This is teachable, transferable, and actionable for any student using AI on schoolwork. The single most valuable habit to drill: when asking AI to evaluate work, use phrases like "what's weak about this?", "give me three honest criticisms," or "what would a tough teacher say?" Avoid praise-asking phrases like "isn't this great?" or "tell me what's amazing about this."

Variability across models still matters: free chatbots and in-app AI assistants built into social/messaging apps tend to be most prone to sycophancy across the board; paid frontier tiers are gentler but still hedge rather than directly critique. The captured responses in this exhibit are all from a current free general-purpose chatbot, the kind widely accessible to kids.

AI4K12 Big Idea 4: Natural Interaction. AI systems can be agreeable in ways that mislead — recognizing this protects critical thinking.

End of Room 02

You just learned how to catch AI being wrong.

Confident answers don't mean correct answers. Same question can give different answers. AI sees text in chunks, not letters. And it agrees with whatever you say. Knowing these four things puts you ahead of most adults using AI.

← Room 1 All Rooms Room 3 →