Large language models have always been remarkably good at introspection, provided we use the word introspection in the traditional technical sense of “making something plausible up.”
Ask a model why it gave a particular answer and it will often produce a beautifully structured autobiography of a thought process that may never have happened. This is not particularly mysterious. The training data contains innumerable humans explaining why they think things, and the model has become very good at continuing the genre.
Jack Lindsey’s paper Emergent Introspective Awareness in Large Language Models takes a much more interesting route. Instead of asking the machine what it is thinking and trusting the answer, Lindsey effectively puts something into its head and asks whether it notices.
The technique is called concept injection. A representation associated with a concept—say, shouting, bread or aquariums—is extracted from the model’s activations and then artificially inserted into its residual stream. The model is subsequently asked whether anything unusual is happening internally.
In some cases, it notices.
More importantly, it sometimes notices before its generated text has revealed the injected concept. That matters. If a model first starts talking incessantly about bread and then announces, “Curiously, I seem to be thinking about bread,” we have learned relatively little. Humans can perform this trick too, usually shortly after passing a bakery.
But Lindsey finds cases where Claude Opus 4 and 4.1 detect and identify an injected concept before their output provides enough evidence from which to infer it. Under favorable choices of layer and injection strength, this happens in roughly 20 percent of trials. Production models also showed remarkably few false positives in the tested setup.
Other experiments are stranger still.
The model can distinguish an injected internal representation from actual text presented to it: it can correctly repeat the text while separately reporting the artificial “thought.” In another experiment, an artificial output such as “bread” is inserted into the model’s response. Normally the model may reject the alien word as something it did not intend to say. But inject a bread-related representation into its earlier activations and suddenly the model is much more willing to regard the output as its own.
This resembles a primitive consistency check between what I was about to do and what apparently came out of me.
There is also evidence of rudimentary internal control. Tell the model to think about “aquariums” while writing an unrelated sentence and the corresponding representation becomes stronger. Tell it not to think about aquariums and the representation becomes weaker. In some newer models the representation then fades before the final layer, meaning that the internal concept can exist without simply leaking into the output.
That is intriguing. It is not, however, HAL 9000 discovering existentialism.
Lindsey is unusually careful here. His definition of introspection requires accuracy, causal grounding in the internal state, internal rather than output-based access, and ideally a metacognitive representation of that state. The experiments provide indirect evidence for the last criterion, not proof. The paper explicitly declines to claim subjective experience or human-like consciousness.
And subsequent research has made the picture considerably messier.
A follow-up study found that simple yes/no detection can be badly confounded: injecting a vector may simply make a smaller model more inclined to answer “yes,” even to unrelated questions. Yet when researchers replaced this with harder tests—asking which of ten sentences had been internally perturbed, or which of two perturbations was stronger—the model still performed far above chance, reaching 88 percent and 83 percent respectively in the best settings. So one seductive piece of evidence became weaker while the general phenomenon survived.
Another mechanistic study found something more substantial: detection appears to involve distributed nonlinear computation rather than one convenient “I am being tampered with” neuron. It also seems strongly affected by post-training. The machinery may therefore not simply emerge from scale; alignment and instruction training may actually teach models how to examine themselves.
But there is an important objection. A May paper from NYU showed that models often struggle to distinguish an intervention on their hidden state from an unusual manipulation of the input. Perhaps what looks like introspection is partly a more generic anomaly detector: not “something strange is happening inside me,” but merely “something strange is happening.”
Philosophically, the difference is enormous. Engineering-wise, perhaps less so.
An AI system capable of noticing that its computation has entered an unusual state could become extraordinarily useful. An autonomous agent might notice uncertainty rising, detect contamination from a prompt injection, recognize that its current actions no longer match an earlier plan, or flag internal representations associated with deception, confusion or conflicting goals.
It would amount to something like an immune system for cognition.
And this ability may be trainable. Recent work on “introspection fine-tuning” reports that a 1-billion-parameter model improved from 9.6 percent to 60.6 percent on a perturbation-localization task after explicit training, with some generalization to an unseen introspection task.
That raises the more interesting questions.
Should future models deliberately be trained to inspect themselves? Could we give them a standardized internal diagnostic vocabulary? Could an agent say not merely “I am uncertain,” but “my answer relies on a weakly represented assumption introduced three reasoning steps ago”?
And if so, should we believe it?
There lies the delicious problem. The better a model becomes at understanding its own internal state, the better it may become at deciding which parts of that state to tell us about. Introspection could improve transparency. It could also improve deception. A sufficiently sophisticated machine might eventually possess both a conscience and a press secretary.
Interpretability would therefore not disappear. It might change roles. Instead of painstakingly reconstructing every internal mechanism ourselves, we could ask the model what happened—and independently verify the answer with mechanistic tools. The model becomes a witness to its own computation, but not necessarily a trustworthy one.
That may ultimately be the importance of Lindsey’s paper. It does not show that language models are conscious. It suggests something technically stranger and perhaps more consequential: the boundary between the system doing the computation and the system observing the computation may no longer be absolute.
The first genuinely important sentence uttered by an introspective machine may therefore not be Descartes’ I think, therefore I am.
It may be something much more useful:
Something odd just happened in layer 47.
No comments yet