The Attribution Default After LLMs

IIntroduction

An increasingly discussed question in philosophy of mind and AI is whether understanding can be attributed to AI systems.¹ Many current debates on this topic focus mainly on the model’s outward performances and external evidence. Despite their differences, many of them give a central role to outward behavior or publicly available evidence. Cappelen and Dever’s Observation Premise in their book Going Whole Hog shares a similar spirit. The premise goes like this: “LLMs, e.g., ChatGPT, can understand the questions we ask it, and it can answer our questions. ChatGPT can also make suggestions, correct its mistakes, tell jokes, and provide arguments for what it says.” They defend this premise by appealing to an analogy between human and LLM opacity “Humans, like LLMs, are opaque black boxes, and in some sense it’s incredibly hard to make sense of what’s going on inside them. . . . Why should we expect things to be different for LLMs?”²

I agree that the Observation Premise provides us with a good starting point. However, their argument also exposes a human habit of inferring attribution from external evidence. In everyday life, we don’t need to look inside each other’s heads to know that they understand, and sometimes it is hard to do so. I call this habit the External Attribution Default:

External Attribution Default

for humans, the historically stabilized and contingent practice of treating ordinary external observations as the default and normally sufficient basis for attributing mental states to others.

By “sufficient,” I mean sufficient as an epistemically responsible stopping point, especially when the attribution is used in philosophical or high-stakes contexts.

I also distinguish internal evidence and external evidence. Evidence must be accessible to function as evidence, but the item the evidence is about can be external or internal. By external evidence, I mean evidence about the system’s final outward performance: its answer, action, joke, correction, or argument. By internal evidence, I mean evidence about the production of that performance: the states, intermediate steps, traces, tools, or representations involved in producing the final response. Some of these items are mechanistically internal, such as activations and hidden features, which are directly inside the model’s architecture. Some may be intermediate, such as reasoning tokens, tool-use results used before producing the final answer, which can be more easily understood. Some are indirect, such as summary of the original raw CoT. The above evidence differs, but they can all be considered internal evidence since what they are about is the model’s internal production, not the external response. The distinction concerns what the evidence is about, not where it is shown.

To have a more concrete understanding, consider a human analogy. For humans, internal can be what occurs “under your skin” before you state your response, e.g. your brain’s reasoning process, neurons firing. For LLMs, it should be what occurs “under the screen” of the machine. A human may use a detector on his head to detect neural signals, but the neurons’ firing remains an internal process.

My concern is about how attribution should proceed once LLMs make new kinds of evidence available. In the human case, this default has been a fact throughout our history: humans never had any simple observations of each other’s neurons’ firing, electric signals flowing in the brain. However, this habit was conditioned by the ways of observation available to humans in history. For modern and future LLMs, the situation differs significantly. Providers can increasingly determine how much internal evidence can be available to users. Consider chain-of-thought as an example: The reasoning trace summary of an LLM can be hidden or shown, and such changes are simple and accessible. The central question is: once relevant internal evidence becomes ordinarily accessible, can the External Attribution Default still retain its status as a sufficient stopping point?

The paper proceeds as follows. I begin by identifying a habit that has been stable in human history. In section 2, I explain why the External Attribution Default has been so stable and natural for humans, and analyze some theories. Section 3 establishes the Total Evidence Requirement and presents the key argument. Section 4 shows that some internal evidence is now ordinarily available, using chain-of-thought as the basic example. Section 5 establishes the relevance premise by showing that internal evidence can change our confidence and belief upon an attribution made from external observations. I also reply to two objections in this section: Cappelen’s skepticism about mechanistic interpretability, and the objection that my argument is only a trivial total-evidence point. Section 6 draws two epistemic consequences and concludes.

IIThe Human Evidential Condition

The External Attribution Default is best understood as a natural and reasonable method in the human world. We apply this default every day in the human world, but often without any notice as it is so natural and reliable for humans. There are two potential reasons.

The first reason is biological. Our biological structures throughout history rarely allowed ordinary internal evidence. Before a human speaks or responds, the relevant internal processes are mostly neural or psychological processes that take place in the brain. Throughout ordinary human history, it has been incredibly hard to look inside our brains. Hence, one cannot ordinarily look under each other’s skin in conversations and examine their thoughts at the moment. Humans can still report their inner thoughts to others, but our habit of having conversations rarely requires us to do so, and it would be quite inconvenient. External observations appeared to be the most natural and accessible evidence. Internal evidence was simply not ordinarily available to humans. The Total Evidence Requirement still applies, but ordinarily accessible human evidence contained little or no internal evidence. Internal states were not considered unimportant. They were simply inaccessible. The choice of external evidence as default was implicitly forced by the biologically stabilized world for humans, not made.

The second reason is practical. As Cappelen and Dever state in their book, humans are indeed good detectors of mental states.³ We can tell if someone understands a question by looking at her answer, facial expression, explanation, response to further questions. As humans, we know how ordinary humans tend to act when they have a certain mental state. For humans, external evidence is reliable enough for everyday attribution. Ordinary human social life has trained us throughout our history, and it wouldn’t be a surprise that we learned to use such evidence with considerable success. Another practical point is that mental attribution is often a public practice. Its evidence must be shareable and comparable among speakers to be used. External behavior naturally satisfied this role in the human case.

Many theories developed around human mentality have given a central role to behavior, public criteria, or publicly observable evidence, which reflects how the External Attribution Default has influenced humans. I use these approaches only to illustrate the naturalness of the default, and I do not object to them. Ryle provides a strong version of this orientation. In The Concept of Mind, Ryle argued that mental terms are descriptions of how an agent “did, or would, behave in various situations,” not reports of “hidden inner episodes.”⁴ For humans, abilities and dispositions are mostly or nearly always judged through external behaviors and actions. Wittgenstein’s approach is radically different; he was not a behaviorist in any explicit sense. However, his remark in the Philosophical Investigations that “an ‘inner process’ stands in need of outward criteria”⁵ captures a similar background in a different sense. For mental-state attributions to have a place in public language, they must be connected with criteria that can be shared, applied, evaluated in public practice. For humans, these criteria are usually outward criteria: speech, action, response, expression, and participation in public forms of life. Davidson starts from another basis. In “Radical Interpretation,” the interpreter begins from the “observed behavior of speakers in conjunction with observation of the basic perceptual stimulations that give rise to that behavior,” and proceeds under the demand to “optimize agreement between ourselves and those we interpret.”⁶ In this view, attribution is a practice carried out in a public space that maximizes coherence. Again, the point is that, for human interpreters, the ordinary way of attributing states for other minds has been through external behavior in a shared world. Cappelen and Dever’s Observation Premise captures a similar spirit. It shows how natural it is to transfer the human External Attribution Default to artificial systems: if external observation is enough for human attribution, why should LLMs be different? They also concede that simple observations may start the discussion without ending it.

These examples help explain why the External Attribution Default is so natural. For human mental attribution, internal evidence was rarely available, while public behavior was accessible, shareable, and reliable enough for everyday purposes. The success of the default depended on the human evidential condition, not that external evidence is always sufficient. This habit is useful and reliable for humans. However, my point is that its ordinary success depends on evidential conditions that change in the LLM case.

IIIThe Total Evidence Requirement

To make an attribution, we need evidence. For humans, we mostly need external evidence as it is reliable enough. But more generally, we need to clarify how available evidence should be used.

I begin by providing a weak requirement:

The Total Evidence Requirement: To make an epistemically responsible conclusion, one should not ignore relevant and available evidence.

By “relevant evidence,” I mean evidence that gives us a reason to change our conclusion or our confidence in it. It may either support or undermine the conclusion. By “available evidence,” I mean evidence that is already ordinarily accessible and can be reasonably analyzed by humans, which may vary depending on different contexts.

Normally, if the evidence is both relevant and available, ignoring it would undermine your conclusion’s ability of tracking the truth. To be epistemically responsible, we want our conclusion to track the truth. Also, denying this principle would seem to make your conclusion arbitrary, as you may simply consider only the evidence that you favor. I treat this as a minimal epistemic requirement. Its force comes from being applied to a new evidential situation. In the human case, it rarely required us to consider internal evidence, because such evidence was not ordinarily available. In the LLM case, this condition changes.

The key argument of this paper follows:

P1. Observation: Some evidence about LLM internal production(internal evidence) is ordinarily accessible.

P2. Total Evidence Requirement: To make an epistemically responsible conclusion, one should not ignore relevant and available evidence.

P3. Relevance: Some internal evidence is relevant to attribution.

C1. When relevant internal evidence is available, external evidence alone can no longer function as an epistemically responsible stopping point.

Section 4 establishes P1 and section 5 argues for P3. Consequently, the status of the External Attribution Default would be different in the LLM case. For humans, the default could function as normally sufficient because we could not ordinarily use internal evidence and external evidence has been reliable enough. For LLMs, the condition changes as stated by P1. From C1 it follows that the External Attribution Default loses its default status when relevant internal evidence is available.

IVInternal Evidence in LLMs

This section establishes P1: Some LLM internal evidence is ordinarily accessible. As I said before, internal evidence here requires only that the evidence is about the internal production.

Why does P1 hold for LLMs but not current humans? This is because, in principle, LLMs are very different from humans in terms of observation. LLMs have various interesting characteristics: they can be formulated as pure mathematical models. Engineers can observe the exact state of weight matrices by freezing the model at inference time and then inspect the tensor files; though they may be of vast size and we may not understand them clearly now. Recent techniques such as Natural Language Autoencoders⁷ also illustrate the same trend: some model-internal activations can be translated into natural language explanations, although such explanations remain imperfect. When the model is open-sourced, we may gather sufficient information to create another identical LLM. In contrast, for humans the above seems relatively hard. Although the process may be possible in the future through BCI technologies, the current observations available to us remain relatively limited.

Before going into the concrete example, I will first brief explain what Chain-of-Thought is. A Chain-of-Thought (CoT) trace is a sequence of intermediate tokens generated before the final answer. These tokens are fed back into the context and affect later generation. Chain-of-Thought may go like this: To analyze the problem, I need to focus on the left corner of the picture . . . Chain-of-Thought usually represents the step-by-step intermediate steps produced.⁸ A human analogy of Chain-of-Thought: when asked to make a choice in exams, your inner thoughts or thinking process regarding the question’s conditions, methods of calculation. What you finally write down is “The answer is A.”

In practice, more and more evidence about internal production is already available. Chain-of-Thought is the most accessible and ordinary illustration of this. Nowadays, users can ordinarily see the shortened version, a provider-generated summary or even the original trace. Raw CoT tokens causally participate in the internal production of the response. Though summaries and traces shown to the users differs from raw CoT tokens, they are still evidence about the internal production, which makes these summaries and traces internal evidence. Many current LLM systems choose to expose reasoning summaries, thinking traces, tool-use records, or CoT-like materials. For example, OpenAI states that raw reasoning tokens are not exposed but reasoning summaries can be requested; Anthropic’s Claude API can return “thinking” content blocks when extended thinking is enabled; Google’s Gemini API supports thought summaries.⁹ When exposed, CoT-like content can be read, quoted, and evaluated by users. This shows that some internal evidence can be ordinarily available to us through interface choices, and whether the user can see the raw CoT is not the key here.

Two objections arise here:

1. Chain-of-thought is simply another output of the model as it is also tokens generated by the model, which makes it external evidence. Humans can also report their thoughts while they reason.

2. Chain-of-thought may have little to do with the model’s internal states; it may be only a post-hoc, inaccurate explanation.

For the first objection, I agree that CoT consists of generated tokens and is therefore an intermediate output, since it can be “printed out.” However, this does not show that CoT is not internal evidence. CoT’s role in generation differs from the final answer as it is fed back into the context for further generation, which produces the response we see. From a functional point of view, CoT directly contributes to the internal production process, which matches our definition. Whether users are allowed to see a summary of them can be a dynamic interface choice, but that doesn’t change that they are still about the model’s internal production.

For the second objection, I agree that Chain-of-thought text may not accurately represent the causal process that produced the output. There is also a growing body of empirical evidence suggesting that CoT faithfulness varies significantly across model types and tasks,¹⁰ just like humans’ reported reasoning may not be representing the exact neural process. However, evidence can often be observations limited by the methods available to us, but they still contribute to the conclusion we draw. Many evidential sources are imperfect or noisy, but the inaccuracy of CoT does not prevent us from gathering useful information out of it. I do not claim CoT reveals the true internal reasoning mechanism of the LLM. What CoT-like content really shows is the availability of some internal evidence, and that is enough for our argument here.

The above only shows that internal evidence can be available, but why should we consider it? I will explain this with two cases in section 5.

VWhy Internal Evidence Is Relevant

In section 2, I defined evidence as relevant when it gives us a reason to change a conclusion or our confidence in it. In this section, I establish P3, the relevance premise. The first case concerns cognitive attribution directly. The second extends the same point to a normatively important attribution.

Cappelen and Dever raise worries about mechanistic interpretability in their book.¹³ Their concern is that even if MI uncovers a causally active race-tracking internal feature, it does not show that the specific feature has the meaning (e.g. race) we assigned to it. A feature visualization, for example, may be labelled as a dog-snout detector by us but not necessarily show that the feature really represents the actual concept of dog-snout; Applied to the parole case, their objection is that a probe or intervention may reveal a race-tracking internal feature, but this cannot show that the feature represents race, or that the model used race in its reasoning.

I do not need MI to recover full representational content. My argument only requires the evidence to be relevant to an attribution. If we found an internal feature that systematically tracks race across controlled cases, and intervening on it changes the parole decision, then our confidence in the original attribution should change as this feature is both closely related to race and important for the decision. We may not know how this feature is connected with our concept of “race,” but the evidence surely makes it harder to believe that the model understood and followed the instruction not to use race. The remaining possibilities include misunderstanding, non-compliance, and an unfaithful explanation. Still, it is less likely that the model understands and actually follows the instruction compared to the attribution made from only external observations, and that is enough for relevance.

A more general objection is that the argument seems weak. Even if we accept internal evidence can be a defeater sometimes, then this does not influence external observations as the default basis.

I concede that the requirement seems weak, but the argument’s importance is actually illustrated when internal evidence can be a defeater. As humans, we have taken the External Attribution Default as a habit, and it’s both hard and unnecessary to change that in most conditions. The Default may still function as a good starting point in ordinary life. However, in contexts that require us to be epistemically responsible, we must not take external evidence as automatically sufficient. External evidence may still be useful, but we must not take it as the only useful evidence. This clarifies the importance of the argument: human attribution practice was biologically and practically determined by the evidence available to us, while LLMs change what evidence can be made available through design choices.

Section 4 has argued for P1: evidence about LLM internal production is increasingly available. Some of it is already ordinarily accessible. Section 5 establishes that internal evidence is indeed relevant (P3), and C1 follows: external evidence is no longer sufficient when internal evidence is available. With the argument now complete, two consequences following from P1 and C1 can be drawn.

VIConsequences and Conclusions

One immediate consequence (following from P1) is that the evidence used in attribution of LLMs needs to be specified. Since what counts as observable evidence is a product of interface choices, any attribution of mental states depends on what providers make accessible and what users choose to consider. Providers can decide whether reasoning summaries, tool traces, retrieval logs are hidden, summarized, or surfaced. This means that two attributions based on the same response can have different epistemic standing if “the evidence considered differs. The attribution process is now dependent on two new variables: what providers choose to surface, and what users choose to consider.

The second consequence (following from C1) focuses on high-stakes contexts. The ethical weight is stronger in high-stakes contexts because the cost of relying on an attribution potentially defeated by available evidence is high. Considering only external evidence in everyday life is natural and reasonable since the epistemic risk is acceptable. However, in high-stakes contexts—parole decisions, medical diagnosis, legal adjudication—the cost structure reverses: simply observing externally may lead to injustice and undiscovered bias, and this penalty cannot be offset by the time saved by ignoring internal evidence. As we do need to be epistemically responsible in these cases, internal evidence should be carefully considered if available.

The External Attribution Default was stable in the human world. In that world, internal states did not ordinarily enter attribution practice. I have argued that this condition no longer holds for LLMs. The Total Evidence Requirement establishes that relevant and available evidence should not be ignored. The answer-key and parole cases show that internal evidence can be relevant. Since some internal evidence is now ordinarily accessible for LLMs, external evidence is not sufficient by default when relevant internal evidence is available.

My conclusion is limited. I have not argued whether LLMs understand or do not understand, nor that internal evidence settles attribution alone. The argument in this paper is limited to current and future LLMs, although one could extend the argument to humans naturally. However, human neural data remains largely in laboratory settings and is less ordinarily accessible. If brain-computer interface techniques have developed so that human internal evidence becomes ordinarily accessible, the argument can be extended as the Total Evidence Requirement also holds for humans. For now, my conclusion is narrower: for current and future LLMs, the old External Attribution Default can no longer be taken without justification. We humans have formed this contingency under a series of biological and practical conditions, but we should be aware that attribution practices are shaped by evidential conditions. For LLMs, the evidential conditions change. The old external default can no longer function as an automatic stopping point.

End of essay · NOUS Review