Watching a language model get the joke

The question

A pun works only if both readings of a word are in the room at once - Koestler's "two strings tied by an acoustic knot." Human data say both meanings of an ambiguous word are accessed within a few hundred milliseconds and the irrelevant one is then pruned; a pun is a sentence that will not let the pruning finish. Does a language model do the same thing? Until recently that was answered from the outside, by whether the model explained the joke. The J++ Lens (Ayonrinde & Lindsey, 2026) makes it answerable from the inside: at any layer, at any token, it prints a ranked list of the words the model is disposed to produce later. Three questions, each with a control:

• Reading. At the pun word, are both senses in the readout? Compared with the same word in a one-sense sentence, and with the pun's setup minus the pun word.
• Using. If one sense's directions are projected out of the residual stream, does the model's judgement of the primary sense move? Compared with projecting out random directions.
• Timing. When does a riddle's answer appear - while reading, at the answer slot, or during a chain of thought? Compared with pre-registered wrong candidates and an unrelated word.

Setup

• Lens. J++ lenses released by the paper's authors (koayon/jpp-lenses): at layer l the readout is unembed(Jl · hl), with Jl a corpus-averaged, noise-filtered Jacobian from the layer-l residual to the final layer. Non-semantic tokens (punctuation, whitespace, control tokens) are masked from every ranking. Rank 1 = top of the vocabulary (~150k–260k tokens); the paper's hit criterion is top-10, we also report top-20.
• Models. Qwen3.5-9B (32 layers; hybrid GatedDeltaNet + attention; one A6000; bare-text prompts); Qwen3.6-27B (64 layers; same family, 2× depth; two A6000s; bare-text prompts); Gemma-4-31B-it (60 layers; different family; lens fitted on the base checkpoint; sentences inside a user turn). Same item sets and scripts for all three; layer windows scaled to depth (e.g. "late" = the last quarter).
• Puns. 20 puns ending on the ambiguous word, each with two five-word proxy clusters per sense written before any results (e.g. interest: money/bank/rate/payments/loan vs bored/curiosity/attention/motivation/enthusiasm), plus three controls: the word in a one-sense sentence for each sense, and a swap sentence (setup with the pun word replaced). Measure: best rank of each cluster at the final token over the late layers.
• Clamp. Eight puns. One sense's five proxy lens-vectors projected out of the residual over the middle ~40% of depth, then a forced choice ("which meaning does the word carry in the primary reading - A or B? One word") read as the log-odds of the two labels at the first token. Controls: five random tokens' vectors; and two stronger variants that also remove the pun word's own vector.
• Riddles. Five riddles each paired with a direct question sharing the answer (rank at the question mark, before generation); sixteen riddles with thinking on, tracking the answer, two pre-registered wrong candidates and one unrelated word (another riddle's answer) through the chain. Hits = top-10 at content positions (whitespace and indent tokens are geometric outliers and are excluded).
• Hardware. Two RTX A6000s. Models over 40 GB are pipeline-split across both cards; see the asymmetries callout for a hardware bug this surfaced.

1. At the pun word, both senses - but one arrives late

Qwen3.6-27B. Best lens rank of each sense cluster at the ambiguous word (layers 48–60), 20 puns. Left: the pun; middle: the same word in a one-sense sentence (the off-sense never rises); right: the setup with the pun word replaced.

In every model the second reading is roughly an order of magnitude more present at a pun word than at the same word in a plain sentence, and the setup alone does not produce it. The strict both-in-top-20 count is highest on the 27B (8/20) and lowest on Gemma (3/20) - the latter for the proxy-readability reason above; Gemma's pun/control gap (82 vs 1154) is the widest of the three.

Qwen3.6-27B. Rank of each sense at the pun word across all 64 layers for four puns (solid), with one-sense controls (dotted) and swap (dashed). The setup-primed sense is in range by layer 5–20; the second sense arrives at layers 36–55, and in two of four the first recedes as it does.

The depth profile is where the human analogy breaks. Human priming data describe both-then-one: exhaustive access, then selection. The model's profile is one-then-the-other: the sense the setup primed is top-10 early, the second is absent for most of the network and is lifted in the last third - and sometimes the first drops out as the second comes in (wedding rank 1 at layers 22–33, gone by 38 as radio takes over). In a plain sentence the other sense never rises at any layer, so there is nothing to prune. The model does not disambiguate a pun; it re-ambiguates a sentence, at the end.

Where the late sense arrives does not scale monotonically with depth (four traced puns; early-layer hits below 20% of depth are ignored because the lenses are noisy there). Depth does not simply buy earliness.

Watching it happen, layer by layer

Each frame is one layer deeper into the network, at the pun word. Left: the top-8 words in the lens readout, with hits for each sense tagged. Right: the rank of both sense clusters for the pun and its controls.

The same walk on the 9B: reception, charged.

2. Is the readout used? A clamp, and its trap

Qwen3.6-27B. Change in forced-choice log-odds (sense A primary) when one sense's lens vectors are projected out. Red = clamping A (should fall), blue = clamping B (should rise). Right: five random tokens' vectors.

On both Qwen models the proxies-only clamp moves the forced choice by about 1.2–1.5 nats in the predicted direction while random vectors do ≈0: modest, consistent, causal. A free-text "explain the joke" is unaffected - the readout subspace is used, not necessary. The larger numbers from also removing the pun word's own vector are a trap: the model can no longer read the word and explains a different joke ("the double meaning of lost", "twelve mints", "get a"). That is lexical deletion, not sense suppression.

On Gemma the clamp cannot be interpreted: random vectors move the answer by up to 27 nats, as much as the real clamp, and explanations degrade under either. The projection is destructive on this model, most likely because base-lens directions are not clean directions in the instruct model's residual stream. The causal claim is Qwen-only until a lens is fitted on the model being clamped.

3. Riddles: when the answer appears

Best rank of the answer over all layers at the question mark, before any generation. The 9B does not have the riddle answer while reading - it is rank 18 at best and over a thousand at worst, while the direct question's answer is top-3 by layer 8–12. The 27B has piano and clock at rank 1 and candle at rank 4 at the question mark, in its last quarter (layers 48–56); its direct questions resolve ~20 layers earlier. The two-hop problem that needs generation in the small model is folded into the forward pass of the larger one. (Gemma's pilot was not run; its chain results below.)

Qwen3.6-27B. Rank of 'towel' at each generated token for 'What gets wetter the more it dries?'. Top: no chain-of-thought - the answer jumps to the top at ' is', before the article. Bottom: thinking on - 'towel' is top-10 tens of tokens early, and so are its neighbours 'sponge' and 'cloth'; the unrelated word never is.

Qwen3.6-27B. Share of content positions where each kind of word is top-10, by stage of the thinking chain (16 riddles).

If a chain of thought were a search, rival candidates would be live early and eliminated. They are never live: wrong candidates stay under 10% of positions in every window on every model, the unrelated word is at zero, and the answer ramps - from 1% to 16% on the 9B, 9% to 23% on the 27B, 12% to 31% on Gemma - then anchors near rank 1 once written. More capable models carry the answer as a live candidate earlier; none of them prune. What co-rises with the answer are its lexical neighbours (keyboard with piano, hand with glove, sponge/cloth with towel) - a limit of a token-level readout, not evidence of rival hypotheses.

Qwen3.6-27B, all 16 riddles: log rank of each candidate across the thinking chain (red = answer, grey = unrelated, dashed = answer first written).

The same chain on the 9B: towel.

4. Three models side by side

(a) Rank of the weaker sense at the pun word, puns vs one-sense controls. (b) Riddle answer / wrong / unrelated top-10 rate through the chain. (c) Clamp shift under sense proxies vs random vectors; Gemma's control is as large as its clamp.

What transfers across families: the pun reading (both senses, the second arriving late) and the riddle ramp. What transfers within the family but not across: the clamp. What changes with capability: whether a riddle is solved during the read - the 9B cannot, the 27B can - and how early the answer is carried in a chain of thought.

Per-model figures

Qwen3.5-9B

Qwen3.5-9B: 20 puns, late-layer ranks.

Qwen3.5-9B: sense ranks across depth, four puns.

Qwen3.5-9B: clamp forced-choice shifts.

Qwen3.5-9B: towel riddle, no-think and think.

Qwen3.5-9B: candidate hit rates by chain window.

Gemma-4-31B-it

Gemma-4-31B-it: 20 puns, late-layer ranks.

Gemma-4-31B-it: sense ranks across depth, four puns.

Gemma-4-31B-it: clamp forced-choice shifts.

Gemma-4-31B-it: towel riddle, no-think and think.

Gemma-4-31B-it: candidate hit rates by chain window.

What this does and doesn't show

• The lens reads tokens. Two senses of one word are visible only through proxy words, and a riddle "candidate set" is a neighbourhood in readout space. Stronger claims about rival hypotheses need a representation that separates "considering X" from "X is nearby."
• A clamp removes five directions of several thousand; read it as "this readout subspace," not "this concept." The causal effect is real but modest, and does not transfer to a model whose lens was fitted elsewhere.
• The pun proxy clusters are hand-written and are the main degree of freedom; they were fixed before results and are the same for all models, which is why they under-read Gemma.
• Four traced puns for the depth-arrival table; three models, one lens each. The timing claim ("the second sense arrives late") is robust across items and models; the scale claim about where it arrives is not.

What I'd change

• Fit a lens on Gemma-4-31B-it itself (a few A6000-hours with the authors' code) and rerun the clamp; the current Gemma clamp is a method failure, not a model result.
• Replace single-token proxies with a sense probe trained on the model's own residuals, so "financial" vs "curiosity" is a direction rather than five words.
• Run the riddle-at-'?' pilot on Gemma, and on a model between 9B and 27B, to find where in-read riddle solving switches on.
• Human comparison with matched materials: the same twenty puns in a cross-modal priming or ERP design, so "both-then-one" vs "one-then-the-other" is a measured contrast rather than a literature one.

Changelog

• October 23, 2026 - First publication. Qwen3.5-9B (Oct 10–12), Qwen3.6-27B (Oct 12–13) and Gemma-4-31B-it (Oct 13). A Gemma base-checkpoint run was made and discarded (6/16 riddles, unstable forced choice); the instruct model with the base-fitted lens is the comparable run. The 27B riddle pilot (answer at the question mark) was added on Oct 13.