AI Learns From the Products of Thinking, Not Thought Itself
Our new article in The Conversation argues that AI is trained on the finished traces of human thinking rather than the process that produced them. That gap limits what machines can do, and it changes what happens to us when we hand them the thinking.
A new article by Rachel Kallen, Ayeh Alhasan, Cassandra Crone, David Kaplan, and me has just been published in The Conversation, titled AI learns from the products of thinking, not thought itself. That’s a problem for machines – and for us.
You have probably seen the one-line version of this argument in the AI disclosure at the bottom of every post on this site (though you might not have realised it). This article is the longer version.
Traces, not process
A viewer in a gallery sees the finished painting. They do not see the years of practice, the abandoned canvases, and the accidents that directed the artist toward the painting on the wall. Similarly, with a recording, a novel or a scientific paper, we only get the finished product. We do not see the bad takes, the drafts or the dead ends. We only see the trace of the process that produced the work.
Large language models are trained on these traces and learn to produce more of them. They can generate output that looks like the product of thinking, without any thinking having occurred. In the article we argue that this matters for what AI systems can do, as well as for the people who use them.
Reasoning models might seem to solve this by printing their reasoning, but what they print is just another kind of trace, and research has found that it often does not reflect the computation that led to an answer. Agentic systems, which run a model in a loop and feed the results of its actions back into it, are closer to genuine trial and error. The gap is widest in creative work. Painters and musicians often describe how the friction of the physical world, and the accidents along the way, shaped the finished work. For a generative AI model, there are no accidents. No paint ever runs down the canvas.
Use it or lose it
The human side of the problem is much older than generative AI. In 1997, an American Airlines captain warned that pilots were becoming “children of the magenta line”, following the flight computer’s magenta line so closely that they were losing the manual flying skills needed to fly the aircraft. Since then, aviation has been dealing with automation dependency.
A similar trend is emerging with AI technologies. After clinics started using AI-assisted colonoscopy, clinicians began detecting fewer polyps when they worked without the software than they had before the AI was available. High school mathematics students who had unrestricted access to generative AI tools performed better than their peers while they had the tools, but worse once the tools were removed. In another study, about ten minutes of AI assistance was enough for participants to perform worse without the AI and to give up on problems sooner.
The tool itself is not the problem. When a generative tool takes over the work for us, it erodes our capacity to do that work ourselves.
More process, not less
The remedy we propose is more process, on both sides.
For AI, this means developing systems that work more like humans do. For example, an AI assistant should revise its answer when a user challenges the way it framed a problem, rather than simply agreeing with the user, as most current systems do. An AI agent should also learn from its failures and let feedback from the environment change the way it approaches a task.
For people, it means designing AI tools that keep the cognitive work with the person. In a Harvard trial, students who learned physics with a well-designed AI tutor learned more, and more quickly, than students in an active-learning classroom, because the AI tutor kept the students engaged in generative work. This result informs our advice for AI teaching assistants: more often than not, they should offer the next question rather than the answer, and sometimes they should raise an ambiguity instead of resolving it.
Why it matters for this site
Most of what I write here is about how to use AI well. The course Practical AI for Behavioural Science that Rachel, Ayeh and I developed requires students to use AI and grades not only the product they hand in, but the process behind it, including their chat histories. Marginalia was born from the same concern as it applies to writing. It compares a draft written with AI against your own published work, to show where your voice has become less distinct from the model’s.
The Conversation article sets out the theory behind both. Using AI well is about deciding which parts of the process you keep.
The paper behind it
The article draws on a longer paper with our colleagues Manuel Varlet, Paula Diaz Monfort, Patrick Nalepka and Mark Dras, Process-Constituted Intelligence: A Shared Criterion for Humans and Machines. We start from the cognitive science distinction between weak and strong equivalence and ask what is required for the process behind a machine’s performance, and not just its output, to be equivalent to the process behind a person’s. We then delineate seven features of human and machine processes that can be assessed and measured. Finally, we propose process audits to make the comparison between human and machine processes testable.
The article closes on the point that matters most:
It is currently easy for machines to produce more traces of thought, but process is much harder to build. If we get it right, our machines can become more intelligent without making us less so.
Read the full article in The Conversation
Michael Richardson Professor, School of Psychological Sciences Faculty of Medicine, Health and Human Sciences Macquarie University
AI Disclosure: I produced this post with AI models and tools from Anthropic (Claude), OpenAI (ChatGPT) and Google (Gemini). I edited and verified all of it, and I take responsibility for what it says. The ideas and views are my own. The post is a trace of a much longer process, built up over years of research and teaching. My colleagues and I have argued that intelligence lives in that process rather than in the outputs it leaves behind, and that handing the generative work to AI can leave a person’s own capacities unbuilt. Here the AI helps produce the trace. The thinking behind it is mine.