In the summer of 2024, if you asked an AI "How many r's are in strawberry?", it often answered "two." The answer is three. An AI that writes code and passes exams could not count three letters.

The explanation you heard most often was tokens: AI does not read text letter by letter; it reads the numbers of the chunks the text was cut into. In this post we cut those chunks ourselves and then asked today's models directly.

This post covers:

1. Did older AI really fail to count the r's?
2. What tokens and tokenizers are
3. Cutting up "strawberry," Korean "딸기" (strawberry), and numbers
4. Tokens are also the unit of pricing
5. 322 questions to five current models
6. Three reasons today's models get it right

How to read the numbers. [Official docs] means material published by Anthropic or OpenAI, [Paper] means a public paper, [Our measurement] means numbers we ran and measured ourselves, and [Community report] means cases from user posts or news articles.

1. Did older AI really fail?

Yes, there are records.

• In June 2024, a post on the OpenAI community forum said ChatGPT counted 2 r's in "strawberry," and gave the same answer when asked again. Replies said both GPT-4 and GPT-4o got it wrong. [Community report] (OpenAI Community)
• In August 2024, a developer reproduced it with GPT-4o: asked directly, 2; asked to count with Python, 3. [Community report] (legoktm's blog)
• On August 27, 2024, TechCrunch reported that both GPT-4o and Claude answered "2." The expert explanation in the article: models see words as chunks and do not know the letters. [Community report] (TechCrunch)

2. Tokens and tokenizers

Before a model reads text, the text is cut into small chunks. The program that cuts it is the tokenizer, and each chunk is a token. A token can be a whole word or part of a word.

We fed "Hello world, I love you" into the tokenizer GPT-4o uses (o200k_base in OpenAI's open-source tiktoken). [Our measurement]

• Common words are one chunk each: Hello, world, love.
• The dot before "world" is a space. The space is attached to the next word.
• Each chunk has a number under it. Those numbers are what the model actually receives.

Less common words are cut into several chunks. "unbelievable" with no leading space becomes three chunks: un, bel, ievable. With a leading space, as inside a sentence, it is one chunk. The same word is cut differently depending on where it sits. [Our measurement]

This tokenizer knows about 200,000 chunks [Official docs] (tiktoken). From the model's point of view, that list of chunks is its whole alphabet. The model takes a list of numbers and answers one number at a time; we see the result after the numbers are turned back into text.

3. Cutting up strawberry, 딸기 and numbers

strawberry

• With a leading space, " strawberry" is one chunk, one number (101830).
• Without the space, it is three chunks: st, raw, berry.
• Capitalized, it is cut differently again: Str, aw, berry.

The letters are the same, but the numbers the model receives are completely different. So "how many r's?" is like showing someone these chunks and asking them to count r's. The letters inside a chunk are not in the input. The model has to recall the spelling from text it saw in training, and if that memory is wrong, you get "two."

That said, researchers disagree on the cause.

• One study treats tokenization as the core bottleneck. It tested 19 character-level tasks and proposed a separate character-aware module. [Paper] (The Strawberry Problem, arXiv 2505.14172)
• Another found only a weak link between how a word is split and whether the answer is right; word length and the actual letter count mattered more. Average accuracy across models was 43.6%. [Paper] (CharBench, arXiv 2508.02591)
• A third found that feeding words letter by letter did not help much, and that making the model reason explicitly before answering worked best. [Paper] (The Genius Paradox, arXiv 2410.14166)

In short: "because of tokens, the letters are not directly visible in the input" is a fact you can check with a tokenizer. "That is why models get it wrong" is still debated.

Korean "딸기" (strawberry)

We fed in "딸기," the Korean word for strawberry. The second syllable "기" is one chunk, but the first syllable "딸" is split into two byte chunks. [Our measurement] A byte is the smallest unit a computer uses to store text. One Korean syllable is stored as three bytes, and here it was cut in the middle into two chunks. So "how many ㄹ (the Korean letter for l/r) are in 딸기?" is an even harder question than strawberry.

Numbers

Numbers get cut too. 1234567 is three chunks: 123, 456, 7. With commas, it is five: 1, comma, 234, comma, 567. [Our measurement] When chunks have different sizes, lining up digits to add is hard. This is often named as one reason older models got long addition wrong. We also asked for ten-digit addition this time, and today's models got all of it right. [Our measurement]

4. Tokens are the unit of pricing

When developers use a model from a program (the API), every price is "per million tokens." Claude Opus 5.5 costs $4 per million input tokens and $20 per million output tokens. [Official docs] (Anthropic pricing)

Coding agents use a lot of tokens here, because every turn resends the whole conversation so far: your request, files read, notes, tool results, all counted as tokens again. Prompt caching gives a discount, but the amount still piles up.

Token counts differ by language

"Good morning, how are you?" is 7 tokens in English; the same greeting in Korean is 12. Same meaning, about 1.7 times the tokens. For common sentences the gap is smaller: another example was 6 tokens in English and 7 in Korean. [Our measurement] Chunk sizes depend on what kind of text the tokenizer was built from.

Anthropic's pricing page also says that how many tokens the same text becomes depends on the tokenizer, the language and the content, and that Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text. [Official docs] (Anthropic pricing)

5. 322 questions to today's models

Method

[Our measurement] (October 31, 2026)

• Five models: Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 (run in Claude Code with tools off), GPT-6.1 Terra and GPT-6.1 Sol (run in Codex CLI with command execution forbidden; we confirmed zero actual tool calls). Sol ran at its lowest reasoning effort (low).
• 17 questions: letter counts in strawberry, raspberry, cranberry and Mississippi; altered words ("strawberrrry" has 5 r's, misspelled "strawbery" has 2); random strings (10 and 40 characters); counting e's in a sentence; letter counts of long words; counting Korean letters (the ㄹ in 딸기, the ㅇ in 안녕하세요, and so on); ten-digit addition.
• Each question ended with "answer with a number only."
• How we told whether a model thought first: GPT reports reasoning tokens separately. Claude kept thinking even when we set thinking off, so we used output token counts: 3 tokens means it answered right away; tens to hundreds means it thought first.

Results

The original question, the r's in "strawberry," was right for all five models on every run (19 runs). Of 322 runs in total, 12 were wrong.

Zero thinking tokens vs. 22

While recording the video, we asked GPT-6.1 Terra "how many ㄹ are in 딸기?" three times at once. The answer is 1; it said 2 all three times, with 0 thinking tokens each time. Thinking tokens are the tokens a model writes internally before answering. Zero means it answered without thinking.

We asked GPT-6.1 Sol the same question. It said 1 both times, which is correct, using 22 thinking tokens each time. [Our measurement]

Counting runs before and after recording, all 14 times Terra answered with 0 thinking tokens it said 2, and both times it used 21 thinking tokens it said 1 (correct). [Our measurement]

The trap of a memorized answer

Terra also answers "strawberry" correctly with 0 thinking tokens. That looks like a memorized answer. Our guess is that the question became so famous in 2024 that the right answer ended up all over the training data.

A memorized answer can also be a trap. When we asked Claude Haiku 5.5 about the misspelled "strawbery," it said three r's even though there are two, four times out of eight. The two wrong runs with recorded token counts were both 3 output tokens, meaning an instant answer; the run that used 56 tokens to think got 2, the right answer. It was pulled toward the famous answer. [Our measurement]

6. Three reasons today's models get it right

1. They spell it out internally before answering. Sol reasoned even at its lowest setting, and Claude used hundreds of tokens internally even when told "number only." Almost every wrong answer came with 0 thinking tokens or a 3-token instant reply. The Genius Paradox paper above also names reasoning before answering as the most reliable fix.
2. They memorized famous questions. Like strawberry. But that memory hurts when the spelling is slightly changed.
3. The tokenizer has not changed. "strawberry" still comes in as chunks, and "딸" is still split into two byte chunks. So an instant answer without thinking can still be wrong. The weakness was not fixed; reasoning works around it.

In one line: models do not read letters. They read the numbers of the chunks the text was cut into. Once you know that, odd AI mistakes and AI pricing both make much more sense.

Limits of this test

• The sample is small: 2 to 6 runs per condition. That is why we report "x out of y" rather than percentages.
• Model companies do not publish their internals. "Thinking first was right, instant answers were wrong" is what we observed in these runs, and "the answer got into the training data" is a guess.
• The tokenizer screenshots use the GPT-4o tokenizer (o200k_base). Claude and newer GPT tokenizers may cut text differently.
• All runs used subscription apps (Claude Code, Codex CLI); results under API settings may differ.

Watch the video

We covered the same material, with the real runs on screen, in a Korean-language video.

• AI는 왜 strawberry의 r을 못 셌을까

Sources

• OpenAI, tiktoken — o200k_base (GPT-4o tokenizer)
• Anthropic, Pricing (checked November 2026)
• TechCrunch, Why AI can't spell 'strawberry' (2024-08-27)
• OpenAI Developer Community, Incorrect count of 'r' characters in the word "strawberry" (2024-06)
• legoktm, How many Rs are in the word strawberry? (2024-08-10)
• Cosma et al., The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models (2025)
• Uzan & Pinter, CharBench: Evaluating the Role of Tokenization in Character-Level Tasks (AAAI-26)
• Xu & Ma, The Genius Paradox (NAACL 2025)