When AI “Analyzes” Your Students: What Is Actually Happening?
Gen AI analytics offer great descriptive content, but is it accurate and reliable? Can we explain to stakeholders how it arrived at particular conclusions?
The fourth part of my series on student AI literacy is taking a little longer to prepare than I thought. Thanks for your patience.
In the meantime, here’s a piece on a crucial topic many school and teacher leaders are grappling with right now: what these new AI-powered analytics tools are actually doing when they claim to “analyze” our students. With major ed-tech vendors pushing hard to get administrative tools into districts for 25-26, it’s worth pausing to understand what’s fundamentally different about AI analysis compared to the systems we’ve used for decades.
This fall, the major ed-tech players are making a big push to get their administrative analytics tools into districts for the 25-26 school year. The pitch is compelling: upload your student data (achievement scores, attendance records, behavior logs) and get instant insights about who’s at risk, what interventions to try, and where to focus your limited resources.
One superintendent told me she sat through a demo where the AI tool produced, in under thirty seconds, a beautifully written report identifying at-risk students, complete with recommended interventions and confident language about “engagement trajectories” and “behavioral patterns.” It would have taken her team days to compile something similar.
She had one question: “How did it know all that?”
The vendor smiled. “Machine learning. The AI analyzes everything.”
She pressed: “But how? What did it actually look at?”
The smile got tighter. “It’s a neural network trained on millions of data points. It identifies patterns humans would miss.”
She left the demo impressed but uneasy. The AI clearly did something. But what, exactly?
And there’s another question lurking underneath: even if we trust what the AI does, should we be routing student data with personally identifiable information through these systems at all?
The data security question matters enormously here. When you used your Student Information System to generate reports, the data stayed in your district’s infrastructure (or with a vendor operating under well-established FERPA frameworks). These new AI tools require uploading student data to external platforms where it becomes the raw material for analysis. Even with enterprise agreements and privacy protections, the question of who has access to your data, and what happens to the insights derived from it, becomes far more complex.
We’ll come back to that. But first, we need to understand what these tools are actually doing when they claim to “analyze” our students.
What Your Old Analytics Actually Did
For decades, districts have used analytical tools. When you ran a report in your Student Information System or built a pivot table in Excel, the process was straightforward:
You decided what to measure (attendance below 80%, GPA drop over 0.5 points)
The system counted and calculated based on your rules
You got results you could verify (show me the 23 students who meet these criteria)
You could trace every decision (why is this student flagged? Because they missed 15 out of 60 days)
It was limited, maybe too limited. A simple threshold might miss students who are struggling in ways that don’t show up in attendance percentages. But it was transparent. You knew exactly what you were measuring and why.
When a counselor asked “Why is Maria on the at-risk list?” you could answer: “Because she’s missed more than 20% of school days.” The counselor might say “Actually, those were all excused medical absences and she’s keeping up with work,” and you’d remove her from the list. The system was a tool. You were the analyst.
What Gen AI Actually Does
Here’s what’s different when an AI “analyzes” your data:
It doesn’t find patterns in your data. It generates text about your data based on patterns from its training.
Let me unpack that:
The AI was trained on vast amounts of text: student records, research papers, early warning reports, educational documents. Through this training, it learned what “at-risk student analysis” looks like as a genre of writing. It learned which data patterns tend to appear alongside which conclusions.
When you upload your student data and ask for analysis, the AI:
Converts your data into numerical representations it can process
Matches patterns in your data to patterns it encountered during training
Generates text that resembles the kind of analysis typically written about similar data
Produces confident, coherent, professional-sounding insights
It’s not reporting what it discovered. It’s predicting what analysis of this data should sound like.
This is why the output feels so polished and authoritative. The AI has processed thousands of analytical reports. It knows the conventions, the language, the structure. It can reproduce that style instantly.
But notice what’s missing:
The reasoning behind specific conclusions
Which data points mattered most and why
Any indication of uncertainty
The ability to distinguish correlation from causation
Awareness of its own limitations
Why This Matters
Let’s take a concrete scenario. Your district has a traditional early warning system that flags students who:
Have attendance below 75%
Failed two or more classes last semester
Had three or more discipline referrals
It’s not sophisticated. It probably misses students who are quietly struggling. But when it flags someone, everyone understands why.
Now you implement the AI tool. You upload the same data. The AI produces a report:
“Analysis indicates 47 students exhibiting concerning engagement trajectories. Predictive modeling suggests these students show behavioral patterns historically associated with academic disengagement. Recommended interventions include attendance monitoring, counseling referrals, and family outreach.”
It sounds authoritative and comprehensive. But ask follow-up questions:
Which 47 students and what specifically triggered their inclusion?
What constitutes an “engagement trajectory”?
Which behaviors are we talking about?
“Historically associated” based on what history?
Why these specific interventions rather than others?
The AI will generate responses to these questions. But those responses are themselves predictions of what explanations should sound like, not traceable logic. You’re not getting closer to understanding the actual reasoning. You’re getting increasingly sophisticated descriptions that maintain the appearance of analysis.
Your old system told you what it counted. The AI tells you what similar analyses typically conclude.
What Changes (And What We Might Lose)
The Ability to Audit Decisions
Traditional analysis: “We flagged everyone below 75% attendance.” AI analysis: “We applied a multi-factor algorithm considering engagement indicators.”
When Maria’s parent calls asking why she’s flagged as at-risk despite having straight As, which system allows you to give a clear answer?
The Learning That Happens Through Analysis
When your team built that early warning system (debating threshold values, discovering seasonal patterns in behavior referrals, noticing how class scheduling affects attendance), they developed institutional knowledge about your students and your context.
The AI completes the same task in thirty seconds. That’s efficient. But your team doesn’t learn anything in the process. When the AI flags an unexpected pattern, they may lack the intuition to evaluate whether it makes sense, because they never developed a feel for what’s typical in your data.
Clear Lines of Accountability
If your traditional system showed bias (perhaps over-identifying certain student populations), you could:
Identify which criteria were triggering the flags
Test whether the bias was systematic
Adjust the specific thresholds
Document what changed and why
If the AI system shows bias, your options are more limited. You can report the problem to the vendor and request that they “retrain the model.” But you can’t see inside the system to understand the source of the bias. You can’t make targeted fixes. You’re dependent on the vendor’s response.
Moving Forward Thoughtfully
These tools represent genuine capability. AI can process complexity and scale beyond what human analysts can manage manually. The question isn’t whether to use them, but how to use them wisely.
Before implementing AI analytics for 25-26, consider asking vendors:
Can you show me the specific reasoning for why this particular student was flagged? (Looking for actual explanations, not general descriptions of the methodology)
Can we run your AI analysis parallel to our existing methods for a trial period? (To see where results align, where they diverge, and build understanding of the differences)
What’s your process when the AI makes an error? (Looking for concrete procedures, not theoretical commitments)
What happens to our data and any derived insights if we end the contract? (Understanding data ownership and portability)
How do you handle questions about liability when AI recommendations affect student outcomes? (Understanding the accountability framework)
What data security measures govern our student information in your system? (Especially regarding PII and whether data is used for model training)
If vendors struggle to provide clear, specific answers, that tells you something important about how well they understand their own systems and how much transparency you can expect.
The districts that will implement these tools most effectively won’t necessarily be the early adopters. They’ll be the ones who take time to understand what “analysis” means in an AI context, maintain their own analytical capabilities alongside these new tools, and insist on partnerships with vendors who can explain their systems clearly.
These are powerful tools. But they work differently than what came before, and that difference matters for how we interpret their outputs and integrate them into decision-making about students.
Nick Potkalitsky, Ph.D.
Check out some of our favorite Substacks:
Mike Kentz’s AI EduPathways: Insights from one of our most insightful, creative, and eloquent AI educators in the business!!!
Terry Underwood’s Learning to Read, Reading to Learn: The most penetrating investigation of the intersections between compositional theory, literacy studies, and AI on the internet!!!
Suzi’s When Life Gives You AI: A cutting-edge exploration of the intersection among computer science, neuroscience, and philosophy
Alejandro Piad Morffis’s The Computerist Journal: Unmatched investigations into coding, machine learning, computational theory, and practical AI applications
Michael Woudenberg’s Polymathic Being: Polymathic wisdom brought to you every Sunday morning with your first cup of coffee
Rob Nelson’s AI Log: Incredibly deep and insightful essay about AI’s impact on higher ed, society, and culture.
Michael Spencer’s AI Supremacy: The most comprehensive and current analysis of AI news and trends, featuring numerous intriguing guest posts
Daniel Bashir’s The Gradient Podcast: The top interviews with leading AI experts, researchers, developers, and linguists.
Daniel Nest’s Why Try AI?: The most amazing updates on AI tools and techniques
Jason Gulya’s The AI Edventure: An important exploration of cutting-edge innovations in AI-responsive curriculum and pedagogy.
Sounds like AI can get us part of the way there - matching the student population against algorithmic indicators. But the rest of the heavy lifting needs to be done by those who can actually see and investigate the many moving parts. Excellent analysis, Nick!
The “amount of variance explained” by each variable in the model may not be great. Have the algorithms been validated? Or is this the Barnum effect? Maybe it “sounds good” like a horoscope. It would make sense to run it several times to see if the output is repeatable. Or test “method A” against “method B” to see if they differ.