James Somers recently published an interesting essay in The New Yorker titled “The Case That A.I. Is Thinking.” He starts by presenting a specific definition of thinking, attributed in part to Eric B. Baum’s 2003 book What is Thought?, that describes this act as deploying a “compressed model of the world” to make predictions about what you expect to happen. (Jeff Hawkins’s 2004 exercise in amateur neuroscience, On Intelligence, makes a similar case).
Somers then talks to experts who study how modern large language models operate, and notes that the mechanics of LLMs’ next-token prediction resemble this existing definition of thinking. Somers is careful to constrain his conclusions, but still finds cause for excitement:
“I do not believe that ChatGPT has an inner life, and yet it seems to know what it’s talking about. Understanding – having a grasp of what’s going on – is an underappreciated kind of thinking.”
Compare this thoughtful and illuminating discussion to another recent description of AI, delivered by biologist Bret Weinstein on an episode of Joe Rogan’s podcast.
Weinstein starts by (correctly) noting that the way a language model learns the meaning of words through exposure to text is analogous to how a baby picks up parts of language by listening to conversations.
But he then builds on this analogy to confidently present a dramatic description of how these models operate:
“It is running little experiments and it is discovering what it should say if it wants certain things to happen, etc. That’s an LLM. At some point, we know that that baby becomes a conscious creature. We don’t know when that is. We don’t even know precisely what we mean. But that is our relationship to the AI. Is the AI conscious? I don’t know. If it’s not now, it will be, and we won’t know when that happens, right? We don’t have a good test.”
This description conflates and confuses many realities about how language models actually function. The most obvious is that once trained, language models are static; they describe a fixed sequence of transformers and feed-forward neural networks. Every word of every response that ChatGPT produces is generated by the same unchanging network.
Contrary to what Weinstein implies, a deployed language model cannot run “little experiments,” or “want” things to happen, or have any notion of an outcome being desirable or not. It doesn’t plot or plan or learn. It has no spontaneous or ongoing computation, and no updatable model of its world – all of which implies it certainly cannot be considered conscious.
As James Somers argues, these fixed networks can still encode an impressive amount of understanding and knowledge that is applied when generating their output, but the computation that accesses this information is nothing like the self-referential, motivated, sustained internal voices that humans often associate with cognition.
(Indeed, Somers specifically points out that our common conceptualization of thinking as “something conscious, like a Joycean inner monologue or the flow of sense memories in a Proustian daydream” has confused our attempts to understand artificial cognition, which operates nothing like this.)
~~~
I mention these two examples because when we talk about AI, they present two differing styles.
In Somers’s thoughtful article, we experience a fundamentally modern approach. He looks inside the proverbial black box to understand the actual mechanisms within LLMs that create the behavior he observed. He then uses this understanding to draw interesting conclusions about the technology.
Weinstein’s approach, by contrast, is fundamentally pre-modern in the sense that he never attempts to open the box and ask how the model actually works. He instead observed its behavior (it’s fluent with language), crafted a story to explain this behavior (maybe language models operate like a child’s mind), and then extrapolated conclusions from his story (children eventually become autonomous and conscious beings, therefore language models will too).
This is not unlike how pre-modern man would tell stories to describe natural phenomena, and then react to the implication of their tales; e.g., lightning comes from the Gods, so we need to make regular sacrifices to keep the Gods from striking us with a bolt from the heavens.
Language model-based AI is an impressive technology that is accompanied by implications and risks that will require cool-headed responses. All of this is too important for pre-modern thinking. When it comes to AI, it’s time to start our most serious conversations by thinking inside the box.
our thinking isn’t very different than how chatGPT works. notice your tendency to “create words” in your mind without you being able to stop it from happening.
e.g. “the hills are alive with sounds of ….”
i allow myself to guess that (practically) everyone reading this sentence had its *next most likely word* added to its ending.
but reading this article made me thing that the difference is not in quality, but in something else – for us, thoughts is an “inner” thing, while the “us” part is what happens as an outside phenomenon to tis thinking. but this sense of “I” is still happening mostly inside our own body, so we can still think of the thoughts as something that is inside the “I”, whereas for the chat bots, while the thinking process is indeed very much alike what we experience, it is still lacking an “I”, and so the *experiencing of the thoughts* that the chatbot produces is done completely outside of it – that is, it happens inside of us, the users.
in some sense, our consciousness and sense of “I” are the exoskeleton that holds that AI thinking process
In my opinion, the examples given are not entirely accurate. It is true that the models are static, that they do not carry out a series of small experiments, they do not inquire. But thinking in this way is similar to comparing these models to how a human functions. In humans, everything is interconnected, forming a whole. On the other hand, AI models can have separate ‘engines’ for inquiry, searching for new data, reasoning, etc. I think all of this lies ahead of us, and we should not underestimate it.
Great analysis. You are absolutely right to debunk the „ghost in the machine“ narrative from a technical standpoint. However, looking at this through the lens of Bruno Latour („We Have Never Been Modern“), there is a significant blind spot in dismissing the social reaction as merely „pre-modern“.
You seem to conflate intentionality (consciousness, planning) with agency (the ability to effect change). Latour argues that non-human actors don’t need a soul or a brain to have agency. Even if the neural net is static and „dumb“, it forces us to react, adapt, and reorganize our work.
By strictly separating the „technical object“ (code) from the „social subject“ (us), you might be missing the reality of the „hybrid“: AI doesn’t need to „think“ like a human to reshape our world like a powerful actor. Weinstein is wrong about the biology, but he is right to sense the massive agency of this new actor.
Thanks for your thoughtful posts, Cal! I’ve been following this blog for a while and appreciate the insights. You’re spot on the mark for calling out Weinstein for claiming that AI absolutely, for-sure, will gain consciousness.
I sensed something was a bit odd regarding the rest of your interpretation of Weinstein’s quote, so I took a look at the podcast episode. It’s wildly speculative, but also thoughtful in a way that didn’t care about presenting ideas in precise language. Your counter points to his quote are accurate facts, but shift the quote into a very specific, precise meaning that may not have been what Weinstein was communicating.
For example, if we look at the quote through the lens of something like Agentic AI (implemented in software like Perplexity’s Comet Browser and BrowserOS)… That fits his metaphor much better and also introduces behavior that mimics the reactive inner monologue Sommers mentioned. At the core they are still just token prediction models iterating inside a feedback loop. No consciousness, but effectively doing exactly what Weinstein said. Challenging our definition of and ability to assess consciousness.
Since I first knew Gary Marcus, I have had a question I didn’t dare to ask: “What is cognitive AI – the cognitive AI he advocates for?”. Marcus is a cognitive scientist so when he tries to promote that it can be really intelligent, I have to wonder. I even worried (naively) that VCs would listen to him and pour money into something dangerous.
As I googled and asked AI (Microsoft Copilot) to distinguish cognitive AI and genAI or whatever the language prediction they have now, I couldn’t find anything distinct. Even Marcus said (as I remember) that Google had gone that road with cognitive AI, then I tried to find information and still don’t understand the mechanism of cognitive AI. Now Marcus has not talked much about it anymore.
If you think there is enough substance and it’s interesting enough (but not alarming), can you cover it like the one you wrote about ChatGPT’s mind? Not now, of course.
You may want to consider not publishing my comment or editing it before publishing as Marcus currently seems to be sensitive and aggressive. No need to add fuel.