Last Tuesday afternoon was the culmination of around 15 hours of work on a new kind of Theory of Knowledge assessment. I wanted students to form an opinion on a real-world topic in an unusual corner of the world, then grapple with an apparent complication. I wanted individual work and group work, and room for students to use AI in a way that would not rob them of their learning.
About 70 students, eight lessons into the course, each got one reading about what counts as disadvantage. For the two reading tasks, about forty minutes in all, the teachers told them: “You may use AI or the internet, or not — your choice.”
The readings were printed on purpose. A PDF invites being pasted into a chatbot, and a printed copy adds just enough friction to stop most people bothering, in the same way that a short pause before opening an app cut how often people opened it by 57% over six weeks.
Each student had twenty minutes with their reading and three questions: What is it claiming? What example does it use to show that? Where does it run into trouble? Then they explained it to their group with the page face-down. This is my attempt to apply David Perkins’ idea of understanding as flexible performance: if you understand something, you can perform it flexibly, and reading your notes aloud isn’t much of a performance.
The booklet repeated the permission in a banner: “AI or the internet are fine here.” In my head they already knew how to use it as a sparring partner, something that pushes back on your ideas, brings in a perspective you hadn’t considered and asks you questions, and the only open question was what they would do with it.
What most of them did with it was read. They read the texts, annotated, answered the questions and talked, and the group discussions were equitable. The teachers walking through the rooms all noticed the same thing: few students bothered reaching for a device. The laptops mostly stayed on the floor beside them, a short reach from a booklet that told them more than once that AI was fine.
The booklets tell the same story. The last question asked “Did you use AI or the internet while you read?” Of the 62 booklets that came back, ten circled yes or a bit. One student who circled no wrote underneath: “The readings were informative. I think AI was not needed.”
The ten who did use it wrote down what they asked for. Most wanted a reading summarised or a word explained: “I asked it to summarize ideas in the readings,” “I used it to explain me words I didn’t know,” “just helped me define disadvantage.” One looked up “who were some of the people mentioned.” One used it “to help understand texts and bounce ideas off,” which is notably the only answer that sounds like a conversation.
Nearly all of that is what you’d do with Google. We look up a word. We look up who someone is, and generally it’s one question with no follow-up. Almost none of my students opened with “You are a world-class learner. Think step by step. Make no mistakes,” which, if you believe the internets, is what using AI well looks like.
I got the idea for why from Jeremy Utley’s podcast, Beyond the Prompt. It’s called the Einstellung effect: when you meet a problem that ostensibly looks like other problems you’ve already solved, the method for those other problems gets in the way of seeing a better approach.
I’ve used it since early 2024 to describe learning to use AI. AI looks like a search bar, so you treat it like a search bar. The routine everyone has practised thousands of times is search, and the new tool gets the old routine. So the thing on the floor beside them, which would have argued with them about what disadvantage means for as long as they liked, got asked to define it.
After a while, though, that 2024 idea floated into the background. My own use of AI was improving fast, and I was in a bubble of people making the same leaps. We kept sharing forehead-slapping moments of “I can’t believe I never did that before” and holy-shit moments of “wow, AI can do that.” I came to believe that most people were improving in the same ways. They weren’t.
My own use changed when I did two things at about the same time. I started using voice mode, and I started asking the AI to interview me about things, one question at a time.
Voice is much faster, and you don’t need to organise your thinking before you start. When you type, you compress what you’re thinking and, to an extent, self-censor. If you don’t yet know what matters most, voice lets you get more of it out first and decide afterwards.
Being interviewed works because there are always parts of your situation that you don’t realise are relevant, and AI has no context. When it asks you questions, you end up covering gaps you didn’t know were there.
It’s also a dialogue, which thinking alone isn’t. In your head you have to role-play the other side yourself, and you’re limited to the perspectives you already have. Another person, or an AI, expands them. The Russian thinker Mikhail Bakhtin, whose ideas much of the work on dialogic learning builds on, put it like this in Problems of Dostoevsky’s Poetics: “Truth is not born nor is it to be found inside the head of an individual person, it is born between people collectively searching for truth, in the process of their dialogic interaction.”
The group half of the afternoon was built on that idea, explaining your reading face-down to people who hadn’t read it. What the students didn’t have was any sense that the laptop on the floor could be one of those people.
One student did write the incantation, or something close to it. On the back page they explained their method. They had found a way to feed an AI all the texts and then instructed it to take the role of the most objective TOK student. By the internets’ standards that’s close to textbook: give it all the context, and tell it who to be.
The same student wrote later, in the writing section with devices closed, that though they doubted their answer, they had fed all the information to a large language model and it gave a similar response to their own. AI, they remarked, is currently the most objective information-processing technology, so they would keep their answer.
It reads as though they worked out their own answer, went to the AI afterwards and compared the two, and the fact that they matched settled it. Language models are known to lean towards whatever answer you bring them, and people come away from that agreement more certain they were right. This is a student who knows how to use AI in some effective ways, yet who still doesn’t know how to use it as a thought partner.
When I first shared the idea for this task, one of the worries raised was AI short-circuiting students’ thinking. In March I answered that if the work students did before the final writing wasn’t their own, they would have hollow and brittle ideas to build on, and instead of us having to play AI detective, life would get mad at them for it. On Tuesday, most of them hardly reached for AI, and the ones who did didn’t think with it.
I think part of the reason almost nobody teaches students to use AI for learning is that cheating captures far more of the attention and the worry. When US teachers were asked what AI training they wanted, two of their top three topics were how to catch it and what to do once they had, and only 16% of students had been told how to use it well. My own design leaned the same way. I started from the question of how to prevent widespread cognitive surrender, people letting AI do the work for them. It was also the first time I’ve had a structured chance to let students use AI in an assessment. I don’t usually get that chance, which is also why I rarely have time to teach them to use it better.
“I think AI was not needed,” one student wrote. For the uses they knew, they were right. But the uses they knew were all ways of using it like Google. The task asked them to test their own reading of a text, and there’s nothing in that for a search engine to do. So it looked as though there was nothing in it for AI either.
If I’d had twenty minutes to teach them one thing beforehand, it would have been the reverse interview, where the AI asks the questions, one at a time, and you answer them. For these students, it would keep asking until they could explain their reading in their own words, which is exactly what they then had to do at their table. It works because we think we understand things far better than we do, right up until someone asks us to explain them. Psychologists call this the illusion of explanatory depth.
It also has to be short, since an interview can’t eat all twenty minutes. And because the reading was on paper, the AI can’t tell them whether they’ve got it right. What it can do is test whether what they’re saying is consistent, and push back on anything questionable or circular. No persona, no “make no mistakes.” Something like this would do:
I’m a Theory of Knowledge student. I’ve just read a short text about what counts as disadvantage, and in a few minutes I have to explain it to my group without looking at it. Interview me about it, one question at a time, and wait for my answer before you ask the next. Find out what I think the text is claiming, what example it uses to show that, and where the claim runs into trouble. When I’m vague, ask me for something specific from the text. When I’m circular, or I contradict something I said earlier, point it out and make me fix it. After about ten questions, tell me which part of my explanation is still the weakest.
Plenty of adults brainstorm with AI, but far fewer ask it to argue back. In one large study of Claude conversations, users questioned the AI’s reasoning in fewer than one in six. The people in that study skewed towards early adopters, so I suspect the average user is lower still. Ethan Mollick calls this the capability overhang: “the gap between what these models can do and what almost anyone is doing with them.”
“You may use AI or the internet, or not — your choice.” Most of them chose not, and for the AI they knew, that was the sensible choice. But permission cannot create an option people cannot perceive. With five or ten minutes, it might not be worth reaching for AI at all. With forty minutes across two tasks, if you are confident AI adds significant value to your work, you make sure some of that time goes to exploring the work with it.
What I’d ask other teachers to do differently is to practise it themselves. Take a unit you’re planning this week, ask the AI to interview you about it and let it push back. We have to be good at it before we can show students it’s there.