Machines, Intelligence and Subjectivity

By John Hardy

I’ve been following the talk on social media about whether we’ve cracked intelligence, particularly the excitement about AGI, or artificial general intelligence. I think we already have a form of automated intelligence: these systems can solve problems and turn up useful connections that people have overlooked. What bothers me is how easily people move from recognising those abilities to talking about consciousness, as though solving a problem establishes that a computer has an experience of its own.

A language model produces that conversation through matrix calculations and predictions of the next token, a word or fragment of text. It’s next-word guessing, roughly speaking, on an enormous scale. When we call the machinery a “neural network”, we invite a comparison with the brain that I think we take too far. An artificial neuron is a mathematical simplification; we can build larger networks and improve their training through backpropagation while still leaving much of the biology out. Neuroscientists can explain particular mechanisms in real neurons, including electrical signalling and chemical transmission at synapses, and researchers use artificial networks to model aspects of the brain. I’m sceptical of how much we can infer from these partial models about the brain as a whole, especially when we try to explain subjective experience.

Roger Penrose, the mathematician and physicist, set out his case against a computational account of the mind in The Emperor’s New Mind. In his later explanation of the argument, he uses Gödel’s incompleteness theorems to examine what happens when we trust the rules of a formal mathematical system to give us true statements. He argues that we can then recognise certain truths that the system’s rules cannot establish, and concludes that no algorithm can fully capture human understanding. My position is that Penrose is right, and I regard his reasoning as rigorous. Impressive answers from a language model give us no reason to assume we’ve overcome the limits he describes.

Penrose encountered opposition from researchers in AI and the philosophy of mind long before today’s language models. Supporters of a computational account argue that reproducing the relevant pattern of interactions in a brain would also reproduce its mental properties: the right computation would itself be enough to produce a mind. In a 1996 reply to his critics, Penrose objected that some researchers approached his argument assuming it must contain a flaw. I think we have reasonable grounds to suspect motivated reasoning when people start with that assumption. For someone who regards the mind as a computational process, Penrose calls a basic part of their understanding of the world into question; for the tech industry, artificial minds make an attractive story to sell. I think those philosophical commitments and commercial interests help explain the eagerness to dismiss Penrose.

A living brain belongs to an organism whose survival depends on what it does. We can represent those conditions in a computer model, but I want to know how that gives rise to a subject with anything at stake. We may never have a way to settle the question of machine experience either way, however capable our machines become. Anyone claiming that we’ve achieved machine consciousness still needs to explain how they’ve established it from the behaviour we can observe. After all, fluent conversation is precisely what developers train these systems to produce.