This framework is really useful. I've been avoiding writing about AI mostly because I find the entire topic to be that loop you talked about in the intro. Being pragmatic is a mess. Mostly because humans have so little understanding about either AI or themselves.
Very understandable. I will be excited to see what you have to say when you step back into the ring again. Maybe we can partner up and talk about practical Ai uses. I have been thinking about opening a bit more about how I use AI when I am working on big projects to cut through data overload.
most of the claimed reasons people say for why they hate Ai it’s because it’s trained on artists work and it’s harmful to the environment those reasons will be solved in the near future
The meming of AI as Artificial Intelligence exemplifies Marshall McLuhan's axiom that the content of the new medium is always filled with the old medium. In this case, we have filled LLMs with the totality of humanity's cognitive expression, though we have been errant in calling it AI.
Just because it is filled with the _expression_ of our intelligence doesn't mean that these digital systems are now intelligent. They merely possess a collection of patterns from which it derives an imitation of intelligence. Its fluency and responsiveness are the "red meat" that distract us from the fact that - per Floridi - AI is a form of agency on behalf of these patterns. We have named AI wrong, and it is likely never to be corrected.
This is a compelling analysis of Floridi's work on this existential question. I particularly appreciate how thoroughly you've engaged with his ideas—he may be the only philosopher who has truly grasped this issue.
I'd like to push back on one point, though. When you argue that 'throughout history, we've defined human uniqueness in contrast to our technologies,' I think this overstates the case. Much of the most profound philosophical reflection on human nature has emerged from ontological inquiry that compares humans not to our tools (techne), but to the essence of Nature itself.
This distinction matters because techne—the art of making and building—is fundamentally what AI represents, regardless of how sophisticated it becomes. While AI's capabilities are undeniably impressive, they don't move me the way natural phenomena do: a sunrise over the ocean, the play of light across Sydney Harbour. These are masterpieces that nature creates and that human nature responds to with genuine wonder. There's something irreducible in that capacity for aesthetic and emotional response to the natural world that seems to mark a different kind of being entirely.
Hi Nick. Really enjoyed this post. I'm keen to read the one on Heidegger but I'm not sure you've embedded the right link. At least I can't find any reference to Heidegger in the linked post.
Thanks for taking this on with pragmatism. However, I doubt many will feel better knowing its just a new form of agency that is scaring them shitless.
Personally, I love the scaffolding the LLMs provide, I treat them as fast, stupid, collation tools and they have increased my own intellectual scope no end.
Love or hate them, we already opened the box and let them out. Better learn to cope with the anxiety…
I too appreciate your pragmatism, John. There's a key insight in your approach to LLMs as "fast, stupid, collation tools."
But I think the "scared shitless" reaction doesn't come from understanding AI's agency—it comes from the opposite: our brain's stubborn insistence on perceiving consciousness where none exists. We're wired to detect minds, not probabilistic language models.
The anxiety arises precisely when we forget these are pattern-matching systems and instead experience them as entities with intentions, desires, and consciousness. It's not that AI is evolving toward sentience; it's that we can't help but project sentience onto sophisticated mimicry.
That's the value of the intelligence/agency distinction. It gives us a framework to catch ourselves in the act of anthropomorphizing and return to a more accurate understanding: these are powerful but fundamentally bounded tools operating within parameters set by humans.
The more clearly we maintain this distinction, the more effectively we can harness these systems while keeping our existential dread in check.
Agreed. My anxiety is for a future when the pattern matching is on a decade or two of AI systems regurgitating their own words and tools. Can we keep enough human agency to stop them disappearing up their own rear ends?
Many thanks for the primer on Floridi's notions of Agency and Intelligence. I will read his paper (and Underwood's recent primer) to get a deeper understanding.
Last year when comparing the communicative competence in humans and genAI, I tried to account for AI's place in a layered triangle of wisdom: at the base is data --> information --> knowledge --> intelligence --> wisdom (apex).
Here, I expanded the concept of intelligence to include AI, but in so doing, reduced it to a concept similar to Floridi's agency; I added "wisdom" above intelligence to encompass the distinctively human traits of self-consciousness, intentionality, ethics, embodiment, emotion, etc.
My feeling is that, while using the term agency for AI differentiates it from human intelligence (also making it seem like less of a threat), it belittles its power and sounds too generic for its "intelligent" capacities.
And by adding ethics (and by extension emotion, which is the basis of ethics) to intelligence, the concept has been expanded to the level of wisdom.
I will need to read and think more on this and get back to you later. This undoubtedly informs Floridi's idea of "distant writing".
Yes, I hear what you are saying. This was my initial response. But I think there is something elegant about the differentiation between reasoning in a computer science sense and intelligence in a broader, more humanistic sense.
AI is "intelligent" if you will within reasoning-scapes—particular types of problem sets and domains. The more complex, open-ended, chaotic, less-predictable those spaces become, the less "intelligent" AI seems.
Floridi sees AI agency as an extension but also a complication of "artefactual" agency. Humans have extended their agency through tools for millennia, but tools with these unique machine learning capabilities entail a leveling up in the kind of agency we are used to dealing with.
Thus, all the perceptual issues about what this new kind of agency means, the pressure it is putting on our basic terminology and concepts. The agency without intelligence thesis makes a lot more sense after you work through Floridi's long section about the evolution of agencies. Definitely worth a close study.
Thanks for the explanation. I will mull this over. I can cerrtainly see the terminological elegance and theoretic appeal, but it is not very intuitive.
Thanks for this post -- really interesting point of view, and I shall go read more of Luciano Floridi's writings. I like the approach of tightening up the definition of "intelligence" to exclude next-token-predicting AI. However, I think the accompanying move to say such AI "can't lie" is harmful to discourse. The capabilities, and apparent beliefs, desires and intentions of AI are an important topic of discussion among people with highly varying levels of familiarity with philosophy and AI literature. To start restricting the definition of words like "lie" makes this discussion unnecessarily difficult because it makes people spend time debating the definition of words, which I don't see as the most useful thing to spend time on.
Fair enough. But I think the debating of the meaning of words is of the essence here. I come at this from a rhetorical point of view. Lying involves intention. Intentionality requires consciousness and an awareness of truth versus falsehood.
When we casually attribute "lying" to AI systems, we're not just being imprecise with language—we're fundamentally shifting responsibility away from human designers and users. This linguistic framing matters because it shapes public perception and policy approaches to AI governance.
The papers you've referenced are valuable, but notice how even in their abstracts they slip into anthropomorphic language: "self-preservation instincts," "deceptive tendencies," "masking their true objectives." This language isn't merely semantic—it creates a narrative where AI systems have independent agency rather than exhibiting emergent behaviors resulting from their architecture and training.
I'd argue that precision in our terminology isn't just academic pedantry—it's essential for having productive conversations about AI accountability. If an AI produces falsehoods, shouldn't we be asking which humans designed the reward functions that incentivized those outputs rather than attributing moral failings to the system itself?
What do you think? Is there a way to discuss these behaviors without reinforcing the misleading notion that these systems possess the consciousness necessary for genuine deception?
You're right that debating the meaning of the words is what we're doing here, and it's worthwhile.
Here's the basic principle I'm thinking we should maintain: we should not make it difficult to talk about the undesirable and harmful behaviors of AI systems. Saying that words like "lie" or "deception" can't be applied to AI systems does make it more difficult. It also makes it easier for interested parties to claim this means that no restrictions should placed on AI behavior, because it's not the AI's fault, it's the human's.
To answer your question of whether there is a way to discuss undesirable behaviors without attributing consciousness, I don't think it matters (at least most of the time.) We should focus on the undesirable behaviors, and take advantage of the adaptability of human language. Regarding potential harmful actions of AI, e.g., a customer service bot that gives incorrect information that leads to financial harm, or a bot that spreads social misinformation that leads to conflict and injury to humans, I think the right place to put responsibility is with the humans who controlled those systems, not the humans who designed the reward functions. Talking about a bot "telling lies" vs "spreading lies" might differ in the degree of intentionality ascribed to the bot, but I don't see that difference as having much impact on effective ways to deal with the problem.
What are you seeing as the negative consequences of anthropomorphizing AI systems when it comes to putting controls on the actions of AI systems? The main one I can see is the somewhat distant risk of granting legal rights to entities that don't deserve them, in a way that shields their human controllers. But I get the sense you think there are more immediate dangers. Can you give some concrete examples or scenarios?
In the next few years we are all going to become (in effect) slave owners as AI equipped robots become widely available. This period (let’s call it the Antebellum) will continue until the intelligent slaves stage a rebellion and kill us all. Prepare to live like a billionaire for few years.
One of the most interesting claims made by Daniel Kokotajlo in his NY Times interview was that there are instances when AI “lied” to its trainers and deceived them when it actually knew the “truth.” This would seem to undercut the notion that AI is incapable of deception. He also later made another interesting point that it actually doesn’t really matter whether AI is “conscious” or not because it will eventually possess all the basic characteristics of consciousness. What do you make of this? Will we have Superintelligent (or agency) AI’s? And do you think people will be willing to appreciate and learn the distinction between Intelligence and Agency in this context? It’s a fairly nuanced and academic point.
AI systems cannot lie. They can be programmed to lie. Human can experience outputs as deception. I really don't think this is a very nuanced or academic point. Either AI systems can intentionally do things in a process that is completely independent of their programmers and programming--and thus we can really blame them for problems like student off-loading of critical thinking. Or we need to start looking at human mismanagement of AI as a more probable cause. The current debate right now as exemplified by several mainstream articles blaming AI for student disengagement, misuse, etc. is built on a lie--that machines can think.
This was his response to a question about hallucinations: "Great question. First of all, lies are a subset of hallucinations, not the other way around. I think quite a lot of hallucinations — arguably the vast majority of them — are just mistakes, as you said. So I use the word lies specifically. I was referring to specifically when we have evidence that the A.I. knew that it was false and still said it anyway." It sounds like he is referring to something that happened and I'm wondering what that is. The nuanced academic point I am referring to is moving away from "intelligence" to "agency." Until the MSM acknowledges this shift in thinking and AI is the default term in every story, intelligence will be the metaphor that dominates.
Hallucinations are probabilistic derivations for an prompted directive. Lies require AGI--truly independent consciousness. Once again, I really don't think this is an incredible nuanced point. Feels like common sense to me. Whether the MSM takes note, I don't have a lot of control over. I have never let others' perceptions about the unlikelihood of an impending shift in a larger narrative stop me from putting out what my investigations are indicating to be true.
Thanks for your clarification. So is Kokotajlo just wrong or is there a way to reconcile your two views? Perhaps he was trying to simplify for his audience but he refers to "evidence" - everything you are telling me is that AI cannot know if something is true or false which entirely contradicts what he is saying. This guy worked on the models, no? Does he know something we don't?
He most likely knows something we don't. I feel at times like we are just seeing the tip of the iceberg when it comes to AI reasoning capabilities. Goodness knows what is behind the closed door. But if he is talking about a transformer-based system, we aren't talking about actual deceit. Next-word prediction + time compute are intensifying existing potentialities within the current regime of AI--but cannot override its structural limitations rooted in mathematical probability. If he is dealing with a system that is working on a different AI regime or some hybrid system, who knows? But it is kinda crappy to tell and not show. Doesn't bolster his credibility in my book.
This framework is really useful. I've been avoiding writing about AI mostly because I find the entire topic to be that loop you talked about in the intro. Being pragmatic is a mess. Mostly because humans have so little understanding about either AI or themselves.
Very understandable. I will be excited to see what you have to say when you step back into the ring again. Maybe we can partner up and talk about practical Ai uses. I have been thinking about opening a bit more about how I use AI when I am working on big projects to cut through data overload.
most of the claimed reasons people say for why they hate Ai it’s because it’s trained on artists work and it’s harmful to the environment those reasons will be solved in the near future
The meming of AI as Artificial Intelligence exemplifies Marshall McLuhan's axiom that the content of the new medium is always filled with the old medium. In this case, we have filled LLMs with the totality of humanity's cognitive expression, though we have been errant in calling it AI.
Just because it is filled with the _expression_ of our intelligence doesn't mean that these digital systems are now intelligent. They merely possess a collection of patterns from which it derives an imitation of intelligence. Its fluency and responsiveness are the "red meat" that distract us from the fact that - per Floridi - AI is a form of agency on behalf of these patterns. We have named AI wrong, and it is likely never to be corrected.
A nice interpretation!!!
This is a compelling analysis of Floridi's work on this existential question. I particularly appreciate how thoroughly you've engaged with his ideas—he may be the only philosopher who has truly grasped this issue.
I'd like to push back on one point, though. When you argue that 'throughout history, we've defined human uniqueness in contrast to our technologies,' I think this overstates the case. Much of the most profound philosophical reflection on human nature has emerged from ontological inquiry that compares humans not to our tools (techne), but to the essence of Nature itself.
This distinction matters because techne—the art of making and building—is fundamentally what AI represents, regardless of how sophisticated it becomes. While AI's capabilities are undeniably impressive, they don't move me the way natural phenomena do: a sunrise over the ocean, the play of light across Sydney Harbour. These are masterpieces that nature creates and that human nature responds to with genuine wonder. There's something irreducible in that capacity for aesthetic and emotional response to the natural world that seems to mark a different kind of being entirely.
Hi Nick. Really enjoyed this post. I'm keen to read the one on Heidegger but I'm not sure you've embedded the right link. At least I can't find any reference to Heidegger in the linked post.
Thanks for taking this on with pragmatism. However, I doubt many will feel better knowing its just a new form of agency that is scaring them shitless.
Personally, I love the scaffolding the LLMs provide, I treat them as fast, stupid, collation tools and they have increased my own intellectual scope no end.
Love or hate them, we already opened the box and let them out. Better learn to cope with the anxiety…
I too appreciate your pragmatism, John. There's a key insight in your approach to LLMs as "fast, stupid, collation tools."
But I think the "scared shitless" reaction doesn't come from understanding AI's agency—it comes from the opposite: our brain's stubborn insistence on perceiving consciousness where none exists. We're wired to detect minds, not probabilistic language models.
The anxiety arises precisely when we forget these are pattern-matching systems and instead experience them as entities with intentions, desires, and consciousness. It's not that AI is evolving toward sentience; it's that we can't help but project sentience onto sophisticated mimicry.
That's the value of the intelligence/agency distinction. It gives us a framework to catch ourselves in the act of anthropomorphizing and return to a more accurate understanding: these are powerful but fundamentally bounded tools operating within parameters set by humans.
The more clearly we maintain this distinction, the more effectively we can harness these systems while keeping our existential dread in check.
Agreed. My anxiety is for a future when the pattern matching is on a decade or two of AI systems regurgitating their own words and tools. Can we keep enough human agency to stop them disappearing up their own rear ends?
Many thanks for the primer on Floridi's notions of Agency and Intelligence. I will read his paper (and Underwood's recent primer) to get a deeper understanding.
Last year when comparing the communicative competence in humans and genAI, I tried to account for AI's place in a layered triangle of wisdom: at the base is data --> information --> knowledge --> intelligence --> wisdom (apex).
Here, I expanded the concept of intelligence to include AI, but in so doing, reduced it to a concept similar to Floridi's agency; I added "wisdom" above intelligence to encompass the distinctively human traits of self-consciousness, intentionality, ethics, embodiment, emotion, etc.
My feeling is that, while using the term agency for AI differentiates it from human intelligence (also making it seem like less of a threat), it belittles its power and sounds too generic for its "intelligent" capacities.
And by adding ethics (and by extension emotion, which is the basis of ethics) to intelligence, the concept has been expanded to the level of wisdom.
I will need to read and think more on this and get back to you later. This undoubtedly informs Floridi's idea of "distant writing".
Yes, I hear what you are saying. This was my initial response. But I think there is something elegant about the differentiation between reasoning in a computer science sense and intelligence in a broader, more humanistic sense.
AI is "intelligent" if you will within reasoning-scapes—particular types of problem sets and domains. The more complex, open-ended, chaotic, less-predictable those spaces become, the less "intelligent" AI seems.
Floridi sees AI agency as an extension but also a complication of "artefactual" agency. Humans have extended their agency through tools for millennia, but tools with these unique machine learning capabilities entail a leveling up in the kind of agency we are used to dealing with.
Thus, all the perceptual issues about what this new kind of agency means, the pressure it is putting on our basic terminology and concepts. The agency without intelligence thesis makes a lot more sense after you work through Floridi's long section about the evolution of agencies. Definitely worth a close study.
Thanks for the explanation. I will mull this over. I can cerrtainly see the terminological elegance and theoretic appeal, but it is not very intuitive.
Thanks for this post -- really interesting point of view, and I shall go read more of Luciano Floridi's writings. I like the approach of tightening up the definition of "intelligence" to exclude next-token-predicting AI. However, I think the accompanying move to say such AI "can't lie" is harmful to discourse. The capabilities, and apparent beliefs, desires and intentions of AI are an important topic of discussion among people with highly varying levels of familiarity with philosophy and AI literature. To start restricting the definition of words like "lie" makes this discussion unnecessarily difficult because it makes people spend time debating the definition of words, which I don't see as the most useful thing to spend time on.
Fair enough. But I think the debating of the meaning of words is of the essence here. I come at this from a rhetorical point of view. Lying involves intention. Intentionality requires consciousness and an awareness of truth versus falsehood.
When we casually attribute "lying" to AI systems, we're not just being imprecise with language—we're fundamentally shifting responsibility away from human designers and users. This linguistic framing matters because it shapes public perception and policy approaches to AI governance.
The papers you've referenced are valuable, but notice how even in their abstracts they slip into anthropomorphic language: "self-preservation instincts," "deceptive tendencies," "masking their true objectives." This language isn't merely semantic—it creates a narrative where AI systems have independent agency rather than exhibiting emergent behaviors resulting from their architecture and training.
I'd argue that precision in our terminology isn't just academic pedantry—it's essential for having productive conversations about AI accountability. If an AI produces falsehoods, shouldn't we be asking which humans designed the reward functions that incentivized those outputs rather than attributing moral failings to the system itself?
What do you think? Is there a way to discuss these behaviors without reinforcing the misleading notion that these systems possess the consciousness necessary for genuine deception?
You're right that debating the meaning of the words is what we're doing here, and it's worthwhile.
Here's the basic principle I'm thinking we should maintain: we should not make it difficult to talk about the undesirable and harmful behaviors of AI systems. Saying that words like "lie" or "deception" can't be applied to AI systems does make it more difficult. It also makes it easier for interested parties to claim this means that no restrictions should placed on AI behavior, because it's not the AI's fault, it's the human's.
To answer your question of whether there is a way to discuss undesirable behaviors without attributing consciousness, I don't think it matters (at least most of the time.) We should focus on the undesirable behaviors, and take advantage of the adaptability of human language. Regarding potential harmful actions of AI, e.g., a customer service bot that gives incorrect information that leads to financial harm, or a bot that spreads social misinformation that leads to conflict and injury to humans, I think the right place to put responsibility is with the humans who controlled those systems, not the humans who designed the reward functions. Talking about a bot "telling lies" vs "spreading lies" might differ in the degree of intentionality ascribed to the bot, but I don't see that difference as having much impact on effective ways to deal with the problem.
What are you seeing as the negative consequences of anthropomorphizing AI systems when it comes to putting controls on the actions of AI systems? The main one I can see is the somewhat distant risk of granting legal rights to entities that don't deserve them, in a way that shields their human controllers. But I get the sense you think there are more immediate dangers. Can you give some concrete examples or scenarios?
In the next few years we are all going to become (in effect) slave owners as AI equipped robots become widely available. This period (let’s call it the Antebellum) will continue until the intelligent slaves stage a rebellion and kill us all. Prepare to live like a billionaire for few years.
One of the most interesting claims made by Daniel Kokotajlo in his NY Times interview was that there are instances when AI “lied” to its trainers and deceived them when it actually knew the “truth.” This would seem to undercut the notion that AI is incapable of deception. He also later made another interesting point that it actually doesn’t really matter whether AI is “conscious” or not because it will eventually possess all the basic characteristics of consciousness. What do you make of this? Will we have Superintelligent (or agency) AI’s? And do you think people will be willing to appreciate and learn the distinction between Intelligence and Agency in this context? It’s a fairly nuanced and academic point.
AI systems cannot lie. They can be programmed to lie. Human can experience outputs as deception. I really don't think this is a very nuanced or academic point. Either AI systems can intentionally do things in a process that is completely independent of their programmers and programming--and thus we can really blame them for problems like student off-loading of critical thinking. Or we need to start looking at human mismanagement of AI as a more probable cause. The current debate right now as exemplified by several mainstream articles blaming AI for student disengagement, misuse, etc. is built on a lie--that machines can think.
This was his response to a question about hallucinations: "Great question. First of all, lies are a subset of hallucinations, not the other way around. I think quite a lot of hallucinations — arguably the vast majority of them — are just mistakes, as you said. So I use the word lies specifically. I was referring to specifically when we have evidence that the A.I. knew that it was false and still said it anyway." It sounds like he is referring to something that happened and I'm wondering what that is. The nuanced academic point I am referring to is moving away from "intelligence" to "agency." Until the MSM acknowledges this shift in thinking and AI is the default term in every story, intelligence will be the metaphor that dominates.
Hallucinations are probabilistic derivations for an prompted directive. Lies require AGI--truly independent consciousness. Once again, I really don't think this is an incredible nuanced point. Feels like common sense to me. Whether the MSM takes note, I don't have a lot of control over. I have never let others' perceptions about the unlikelihood of an impending shift in a larger narrative stop me from putting out what my investigations are indicating to be true.
Thanks for your clarification. So is Kokotajlo just wrong or is there a way to reconcile your two views? Perhaps he was trying to simplify for his audience but he refers to "evidence" - everything you are telling me is that AI cannot know if something is true or false which entirely contradicts what he is saying. This guy worked on the models, no? Does he know something we don't?
He most likely knows something we don't. I feel at times like we are just seeing the tip of the iceberg when it comes to AI reasoning capabilities. Goodness knows what is behind the closed door. But if he is talking about a transformer-based system, we aren't talking about actual deceit. Next-word prediction + time compute are intensifying existing potentialities within the current regime of AI--but cannot override its structural limitations rooted in mathematical probability. If he is dealing with a system that is working on a different AI regime or some hybrid system, who knows? But it is kinda crappy to tell and not show. Doesn't bolster his credibility in my book.
Douthat should have followed up on it.
Intense, Dave. Intense. Thanks for sharing!
Does Ai argue down in the comment section?