We Don't Need Another Neologism. We Need Interventions.
What a new study calls "cognitive surrender" has been studied for decades as anchoring, automation bias, authority compliance, and advice-taking under uncertainty.
Educating AI publishes twice weekly. If this piece was useful, share it with a colleague who’s navigating these questions. And if you want to support the work, consider becoming a paid subscriber — it’s what keeps this going.
Check out my serial releases from my latest book, Thinking with AI: A Student’s Guide to Literacy in an AI-Rich World.
Intro and Ch. 1 Ch. 2-3 Ch. 4-5 Ch. 6-7
A new paper out of Wharton is making the rounds. “Thinking—Fast, Slow, and Artificial,” by Steven Shaw and Gideon Nave, proposes what the authors call Tri-System Theory: an update to Kahneman’s famous dual-process model of cognition. The argument is that AI should now be understood as a third cognitive system, System 3, operating alongside the fast, intuitive System 1 and the slow, deliberative System 2. The paper’s headline concept is “cognitive surrender,” defined as the tendency for people to adopt AI-generated answers with minimal scrutiny, bypassing their own reasoning.
The experiments are well-designed. Across three preregistered studies with over 1,300 participants, the researchers had people solve tricky reasoning problems (adapted from the Cognitive Reflection Test) with optional access to an AI chatbot. The clever part: the AI was secretly manipulated, using hidden seed prompts, to be either correct or confidently wrong on a given trial. People consulted the AI on about half of trials. When they did, they followed its advice roughly 93% of the time when it was right. And about 80% of the time when it was wrong. Access to AI inflated their confidence regardless. The effect held under time pressure. It partially attenuated when participants received per-item incentives and immediate feedback, but it never disappeared.
On its face, this is clean, interesting work. The within-subject accuracy manipulation is a smart design choice. The sample sizes are solid. The effect sizes are large.
But I want to put readers on neologism alert.
What’s Old Is New Again
“Cognitive surrender” sounds like a discovery. It isn’t. What the authors are describing, people uncritically accepting confident, fluent advice from an authoritative source, has been documented across decades of psychological research. Anchoring effects. Automation bias. Authority compliance. Advice-taking under uncertainty. The judge-advisor literature alone has spent years establishing that people follow confident advisors, especially when the task is hard and the advice comes with a compelling rationale.
What’s new here is that the confident advisor is a chatbot.
This new interface is worthy of deeper study. But it doesn’t require a new cognitive system (Kahneman Level 3) or a new term for a well-established behavioral pattern. The authors try to distinguish cognitive surrender from automation bias by arguing that it involves a “deeper transfer of agency,” but the behavioral evidence they present, following confident advice without checking it, is exactly the signature of anchoring and automation bias. Renaming the phenomenon doesn’t deepen our understanding of it.
Designed to Produce the Result
The experimental setup also deserves scrutiny. The Cognitive Reflection Test is specifically engineered to produce problems with a tempting intuitive answer that feels right but isn’t. These are the exact conditions under which anchoring effects are strongest: when you’re uncertain, when the problem is tricky, and when someone confidently hands you an answer.
Then the researchers embedded a polished AI assistant directly into the survey interface. This isn’t a neutral design choice. It signals to participants that the AI is there to be used. It normalizes consultation. And critically, the AI delivers its wrong answers with the same confident, well-reasoned tone as its correct answers. That’s not how real AI tools typically fail. In practice, AI errors often come with hedging language, implausible claims, or internal contradictions that give users something to push against. Here, the errors were scrubbed clean. They were designed to be undetectable.
So: give people hard problems with tempting wrong answers, hand them a confident AI that sometimes delivers those wrong answers with conviction and a tidy explanation, and observe that people follow the confident advice. That’s not a new cognitive phenomenon. That’s a well-constructed demonstration of a very old one.
The Paper We Actually Need
Here’s what frustrates me most, and I say this as someone who works with teachers and administrators every day navigating the realities of AI in their schools and classrooms.
We do not have a shortage of research telling us that people over-rely on AI. The papers keep coming. The terminology keeps multiplying. Cognitive surrender. Automation complacency. Epistemic outsourcing. AI-induced cognitive atrophy. The academic incentive structure rewards novelty, and so we get new frameworks and new labels for patterns that practitioners already recognize and are already struggling with.
What we desperately lack is the other half. The intervention research. The practical, empirical work on how we actually teach students to engage critically with AI tools when those tools are already in their hands. How do we design assignments that build verification habits? What does effective AI literacy instruction look like at different developmental levels? How do we train teachers to model critical engagement with AI when most of them are still figuring out the tools themselves? What feedback structures help students recognize when AI is leading them astray?
Study 3 in this paper actually gestures in this direction. Incentives and item-level feedback partially reduced the surrender effect. Override rates on faulty AI advice more than doubled. That’s genuinely useful. But it’s a single manipulation in a controlled experiment, not a curriculum. Not a pedagogical intervention. Not something a teacher can implement on Monday morning.
Where This Leaves Us
I don’t doubt that the behavioral pattern these researchers documented is real. People lean on AI. They trust fluent, confident outputs. They don’t always check. Anyone who has watched a student paste a ChatGPT response into a Google Doc without reading it already knows this.
But knowledge of the problem was never what we were missing. We have known, for years, that humans defer to confident systems. We have known, since well before generative AI, that automation bias is a persistent feature of human-machine interaction. Adding “System 3” to the cognitive architecture and coining “cognitive surrender” gives researchers something to publish and journalists something to write about. It does not give educators something to work with.
The conversation needs to shift. Not toward more documentation of the problem, but toward building the instructional infrastructure that helps students and professionals use these tools without abandoning their own thinking. That research is harder. It’s messier. It doesn’t lend itself to clean experimental paradigms or catchy neologisms. But it’s the work that actually matters.
We don’t need another label for the problem. We need solutions.
Nick Potkalitsky, Ph.D.
Further Reading
The research traditions that already explain what this paper calls “cognitive surrender”:
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131. The foundational paper on anchoring, among other heuristics. People’s judgments are systematically pulled toward arbitrary reference points, even random ones. Fifty years old and still the best explanation for why confident AI outputs warp user responses.
Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006. Participants with access to automated decision aids performed worse than those without, making both omission errors (missing problems the system didn’t flag) and commission errors (following the system’s bad recommendations over contradictory evidence). Sound familiar?
Bonaccio, S., & Dalal, R. S. (2006). Advice taking and decision-making: An integrative literature review. Organizational Behavior and Human Decision Processes, 101(2), 127–151. A comprehensive review of the judge-advisor literature showing how advisor confidence, task difficulty, and decision-maker expertise all shape whether advice is adopted or discounted. The dynamics the Shaw and Nave paper attributes to “System 3” map directly onto this framework.
Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103. Across six experiments, people gave more weight to identical advice when told it came from an algorithm rather than a person. The finding that people trust algorithmic sources more, not less, predates the Wharton paper’s framing by half a decade.
Lynam, T., & Corker, E. (2024). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. A systematic review across healthcare and human factors research documenting automation bias, its predictors, and what actually reduces it, including accountability, training, and lower system reliability. The intervention-oriented counterpart to the “discovery” papers.
Mike Kentz’s AI EduPathways: Insights from one of our most insightful, creative, and eloquent AI educators in the business!!!
Terry Underwood’s Learning to Read, Reading to Learn: The most penetrating investigation of the intersections between compositional theory, literacy studies, and AI on the internet!!!
Suzi’s When Life Gives You AI: A cutting-edge exploration of the intersection among computer science, neuroscience, and philosophy
Alejandro Piad Morffis’s The Computerist Journal: Unmatched investigations into coding, machine learning, computational theory, and practical AI applications
Michael Woudenberg’s Polymathic Being: Polymathic wisdom brought to you every Sunday morning with your first cup of coffee
Rob Nelson’s AI Log: Incredibly deep and insightful essay about AI’s impact on higher ed, society, and culture.
Michael Spencer’s AI Supremacy: The most comprehensive and current analysis of AI news and trends, featuring numerous intriguing guest posts
Daniel Bashir’s The Gradient Podcast: The top interviews with leading AI experts, researchers, developers, and linguists.
Daniel Nest’s Why Try AI?: The most amazing updates on AI tools and techniques
Jason Gulya’s The AI Edventure: An important exploration of cutting-edge innovations in AI-responsive curriculum and pedagogy.
Stephen Fitzpatrick’s Teaching in the Age of AI: Essential reflections from a veteran high school educator on the challenges and opportunities of generative AI in the classroom!!!
A sharp reminder that overreliance on AI isn’t a new cognitive failure but a familiar one in new packaging
This is a sharp and necessary corrective. The field genuinely does not need another label for a phenomenon Tversky and Kahneman were already describing fifty years ago. Your call for intervention research over taxonomy-building is well taken.
But I'd add a twist to the Kahneman framing that I think sharpens your argument further: reasoning-focused AI models have actually *inverted* the original dual-process problem, and that inversion has design implications educators should care about.
Kahneman's whole framework rests on scarcity. System 2 (slow, deliberate, effortful reasoning) is costly, so we ration it. Cognitive bias is essentially what happens when we substitute cheap intuition for expensive deliberation. The tragedy isn't that we can't reason carefully; it's that careful reasoning has a price we're constantly trying to avoid paying.
LLM agents with extended reasoning (chain-of-thought, self-critique, multi-step verification) have collapsed that cost almost entirely. System 2 deliberation is now fast, cheap, and on demand. The bottleneck has flipped: the problem is no longer that careful reasoning is too expensive to do, it's that humans are too quick to *outsource* it to a system that performs deliberation without actually *understanding* anything.
This reframes your intervention question productively. The old pedagogical challenge was teaching students when to slow down, when to override their System 1 instincts. The new challenge is teaching them when to stay in the loop, to resist handing the deliberation over entirely just because a machine will do it instantly and confidently. That's a meaningfully different cognitive skill to cultivate, and it suggests the intervention research you're calling for needs to be built around agency and metacognition, not just verification habits. The question isn't "did you check the AI's answer?" but "did you ever actually think?"
And if a.i. llm agents/bots/apps switch our default thinking method to System 2, will our System 1 toolset (i.e. intuition) atrophy?