The output is on the screen and it is good. The knowledge is not in the head. And the person usually does not know it. Why AI makes us feel we understand, and four ways to check
Beginner ~10 min read
A manager asks an AI assistant for a strategy memo, reads it twice, finds nothing wrong, and signs it. In the meeting, the first question is "why this option and not the other?" There is a pause that lasts a little too long.
Across town, a student submits a Deep Learning project. The model trains, the accuracy is excellent, the code is clean. Then the examiner asks why that learning rate, and what would happen if the BatchNorm layers were removed. The project works. The answer does not come.
Same phenomenon, different desks. In both cases the output is present and the knowledge is absent, and in both cases the person did not know it until someone asked. That last part is the definition of an illusion: you cannot see it from the inside.
The greatest danger is not that people will not know. It is that they will not know while feeling that they do.
An old Arabic verse puts the two cases side by side: "If you knew, that is a calamity; and if you did not know, the calamity is greater."
This article makes one argument in three steps. The illusion is old and well documented. AI did not invent it; it multiplied it, in three specific ways. And it can be caught, by anyone, with a few cheap tests.
In 2002, Leonid Rozenblit and Frank Keil at Yale asked people how well they understood everyday objects: a zipper, a flush toilet, a bicycle, a sewing machine. People rated themselves generously. Then they were asked to write out, step by step, how the thing actually works. The ratings collapsed. Rozenblit and Keil named the gap the illusion of explanatory depth: we feel we understand complex things with far more precision and depth than we really do, and the feeling is strongest for exactly the kind of knowledge, mechanisms and causes, that matters most.[1]
The mechanism behind it is simple. We confuse familiarity with understanding. I have seen a zipper ten thousand times, so my brain files it as known. Familiarity is a feeling. Understanding is the ability to reconstruct, predict and repair. They are not the same thing, and the feeling arrives first.
Nine years later Betsy Sparrow, Jenny Liu and Daniel Wegner showed what happens when information is always one search away. People who expected to be able to look something up later remembered the information less, and remembered where to find it more. They called it the Google effect: the search engine becomes part of our memory, and we quietly reassign what we keep in our heads.[2]
Worse, the search does not just move knowledge out of the head; it makes us believe more of it is in there. Across nine experiments, Matthew Fisher, Mariel Goddu and Frank Keil found that searching the internet for explanations raised people's estimates of their own internal knowledge, including on unrelated questions they had never searched.[3] Adrian Ward later showed the same inflation with a neat twist: people who could google a quiz rated their own memory and intelligence higher, and predicted they would do better on a future quiz without the internet. The process is so fast and seamless that we forget we used it.[4]
A search engine gives you a page and leaves you to read it. A language model gives you an answer, written in confident prose, in your own words if you ask, in seconds. That difference feeds three mechanisms at once.
Psychologists call it processing fluency: the easier something is to read, hear or picture, the more true, likeable and familiar it feels. The effect has been found with clearer fonts, rhyming slogans, repeated statements and simpler words.[5] Daniel Oppenheimer showed that writers who used needlessly long words were judged less intelligent, because their text was harder to process.[6] Now reverse it. A model produces text that is smooth, well structured and free of hesitation. Our brain uses ease of processing as a signal of understanding, so a fluent answer feels both correct and understood. The polish is on the surface; the signal we read is the surface.
When the model writes a paragraph or a function under my name and I read it and nod, I feel I produced it. But recognising something is not generating it. In 1978 Norman Slamecka and Peter Graf showed that words people generate themselves are remembered far better than the same words simply read; the generation effect is one of the most replicated findings in memory research.[7] Reading an answer the model generated gives me the memory of a reader, not of an author, while leaving me with the author's confidence.
Understanding is built from productive struggle: the attempt, the error, the correction. Robert and Elizabeth Bjork call the conditions that make learning feel harder but stick better desirable difficulties: spacing, testing yourself, generating answers before seeing them.[8] Students who explain worked examples to themselves, step by step, learn to solve new problems; students who just read them do not.[9] AI removes the friction. It also removes the learning that the friction was doing. And because the people with the least skill are the least able to see what they are missing, the gap stays invisible to the one person who most needs to see it.[10]
These are not only laboratory worries. Four recent studies, from four different fields, point the same way.
Lisa Messeri and Molly Crockett put the risk for science in one sentence: AI tools promise to let us produce more while understanding less, by exploiting exactly the cognitive shortcuts above.[15] A 2025 survey of 666 people across ages found the same pattern in everyday life: heavier use of AI tools went with more "cognitive offloading" and lower critical-thinking scores, most strongly in the young.[16]
The illusion does not belong to students or programmers. It belongs to anyone who accepts output in place of understanding. Eight desks, one tell: the output is present, the knowledge is absent, and the person does not know it.
Notice that none of these people is lazy or careless. Each did what used to be enough: read the output, found it plausible, moved on. What changed is that plausible output is now free, and understanding is still expensive.
Two pictures separate the healthy use from the harmful one.
A crutch carries the load instead of you. It is wonderful for a broken leg. Lean on it when the leg is healthy and the muscle wastes, quietly, until one day the crutch is missing and you discover you cannot stand.
A scaffold supports you temporarily while you build something you could not yet build alone, and then it is removed. The word comes from education: David Wood, Jerome Bruner and Gail Ross used it in 1976 for the way a tutor holds the parts of a task the learner cannot yet manage, and hands them back one by one.[18] The whole point of a scaffold is that it is designed to disappear.
Gavriel Salomon and colleagues drew the line thirty-five years ago, before any of today's tools existed: there are effects with a technology, what you can do while it is in your hands, and effects of it, what you can do afterwards, without it. Both depend on whether the person stays mindfully engaged, and the second is the one that counts as learning.[19]
The illusion cannot be seen from the inside, so the tests have to come from outside the feeling. Rozenblit and Keil's own method is the first one: the moment people had to write the explanation out, the illusion broke.[1] Four tests, each a few minutes, each usable by anyone at any desk:
| Test | How | What it catches |
|---|---|---|
| Explain it without the tool | Close the laptop. Explain the answer to a colleague at a whiteboard, or write it in your own words from memory. | Familiarity posing as understanding. The explanation stops exactly where the knowledge stops. |
| Predict before you ask | Before sending the prompt, write what you expect the answer, or the error, to be. Then compare. | False ownership. A prediction is something you generated; a match or a miss is something you learn from. Reading the answer first gives you neither. |
| Change one condition | Alter one assumption in the problem: a different data size, a different client, a different deadline. Say what changes in the answer before you ask. | Output without a model behind it. If you cannot say what moves, you did not understand what held it in place. |
| Teach it | Explain the idea to someone who will ask "why?" twice. | The second "why" is where the fog lives. |
For teachers, examiners and managers, the same four tests become a way of assessing people rather than outputs: a short oral follow-up on any submitted work; marks for the process (the predictions, the attempts, the corrections) and not only the product; "what if" questions that move one condition; and asking for the prediction before the tool is opened. None of these bans the tool. All of them make it a scaffold.
The illusion of explanatory depth is twenty years old; the Google effect is fifteen. AI did not create the illusion of knowing. It made fluent, confident, personalised output free, and left understanding at its old price. Three mechanisms do the damage: fluency that reads as truth, recognition that feels like authorship, and the removal of the friction that learning was made of. The 2023 to 2025 studies, in consulting, classrooms, offices and laboratories, all show the same shape: better output, worse understanding, and no alarm.
The danger is not ignorance. Conscious ignorance is the engine of learning; it asks questions. The danger is ignorance dressed in fluency, because it stops asking. Keep the tool. Change the posture: predict before you ask, explain with the laptop closed, change one condition, and let the scaffold come down.
Enjoyed this article?Follow the page for new AI articles and daily space news.
Follow on FacebookAll links accessed 19 October 2026. Illustrations were generated with Google Gemini for this article.
