You need coherence

Suppose you hand a language model the posts in this notebook and ask it for the next one, nine paragraphs long. The failure I have in mind goes like this. The topic is right, and the voice is close. Somewhere around the fourth paragraph a formula appears between single dollar signs, although every formula here uses two. A heading comes out in Title Case. “Behaviour” loses its u. The post stops after seven paragraphs, and the sources never come.

I haven’t run that exact request for this post. The draft is a composite, made of failures that the studies in the next section measure one at a time. None of them is hard to avoid, either. The double dollar signs are right there in the context, hundreds of them.

I’ll call the broken rules the five promises, give them short names, and follow them all the way through: the double dollars (formulas between two dollar signs), sentence case (headings written like sentences), the u in behaviour (British spelling), nine paragraphs (the length that was asked for) and sources last (every post ends with its sources).

Picture a scribe copying a long letter in a dark room. Every word already on the page can be read again, but only by pointing the one lamp at it, and the lamp has a fixed brightness. Light spent on the instructions at the top of the page is light taken from everything else, and every new line adds more page for the same light. The scribe also has no margin to write notes in. To know whether the sources are done, or how many paragraphs there are so far, the scribe has to go and look again, every time.

A transformer is in the same position. The lamp is the softmax in attention: its weights always add up to one. And the model keeps nothing from one step to the next except the words themselves, so it has no margin either.

What I want to know is how fast the chance of a flawless text falls as the text gets longer. Call that chance $S(L)$ for a text of $L$ tokens. For long texts it falls like $e^{-\kappa L}$, and I’ll call $\kappa$ the length exponent. A small $\kappa$ means long texts stay faithful to their rules. A large one means they don’t.

The results, in brief. In an ordinary transformer, $\kappa$ tends to the value it would have if the rules were never looked at, and a longer prompt doesn’t change the limit, it only gets there sooner. Give each promise a small state of its own, read through a softmax over the promises instead of over the whole text, and $\kappa$ stays under a bound that depends on neither the prompt nor the length of the text. Some promises are just a choice between interchangeable variants, like one dollar sign or two. For those, making the choice once, before the first word, gives $\kappa = 0$. The end of a text is different again: it fails through the decision to stop, and I give that a formula of its own.

The module I propose is called the Ledger. I trained it, along with a plain baseline and some controls, on a small synthetic task on four CPU cores. The Ledger kept every implicit rule in 90.0% of its posts, against 80.5% for the baseline ($p = 5.5 \times 10^{-8}$), and wrote the requested number of paragraphs in 98.5% of them, against 38.5%. The same slots placed inside the model’s ordinary softmax did no better than the baseline (74.0%). With the style fixed at the start, there were 4,526 chances to break a convention and none was taken.

The experiment is a toy, and I say so wherever it matters. The argument rests on the proofs. The toy is there to check that the mechanisms behave as the proofs say. Two tests on a real pretrained model are written and tested, and neither has run yet. Their predictions are below, written before any result exists.

If a formula gets in your way, skip it. The sentence after each one says what it means.

How often models lose the thread

Nobody has measured coherence in exactly the sense I use here, but plenty of studies measure pieces of it, and they sort into groups.

In the first group, things get worse as the context gets longer. Liu and colleagues put the answer to a question at different depths of a long input and got U-shaped accuracy. With the answer in the middle of 20 or 30 documents, GPT-3.5-Turbo did worse than with no documents at all (56.1%). RULER tested models that claim contexts of 32,000 tokens or more, and only half of them held up at 32,000. NoLiMa made retrieval harder by keeping word overlap between question and answer to a minimum; at 32,000 tokens, 11 models fell below half their short-context scores, and GPT-4o went from 99.3% to 69.7%. Levy, Jacoby and Goldberg padded the same reasoning questions to different lengths and saw reasoning degrade far below the models’ technical limits. And Li and colleagues let two chatbots talk to each other, each with a persona set by its system prompt, and found the personas drifting within eight rounds. They traced it to attention decay, which will matter below.

In the second group, things get worse with more rules or more turns. On IFEval, about 500 prompts with 25 kinds of instruction a program can check (things like “no commas” or “at least 3 sections”), GPT-4 kept every instruction of a prompt 76.9% of the time. Multi-IF spread such instructions over three turns in eight languages, and o1-preview fell from 0.877 at the first turn to 0.707 at the third. IFScale asked for a business report containing up to 500 required keywords; the best models managed 68% at 500 and favoured the earlier instructions. Laban and colleagues simulated over 200,000 conversations with 15 models on six tasks. When a task arrived in pieces over several turns, performance dropped by 39% on average.

The third group is about endings. ALCE asks for answers that cite their sources, and on its ELI5 questions even the best models lacked complete citation support half the time.

Imitating a writer’s style from a few samples has all of these problems, plus one more, because nobody writes the rules down. Wang and colleagues tried it with more than 400 everyday authors and over 40,000 generations per model. The models got close on news and email. They did badly on blogs and forums.

Laban and colleagues also split their drop into two parts: aptitude, how well a model does at its best, and reliability, how often it does that well. Aptitude fell by about 15%. Unreliability rose by 112%. So the models could still do the task. They just didn’t do it every time. That looks like a problem with machinery, not with knowledge, and the same pattern shows up across labs and years. The models in these studies are, almost without exception, transformers, a design from 2017.

The machine

What follows is the Transformer as its first paper described it, read with one question in mind: where could a promise be kept?

Tokens and positions

Ashish Vaswani and seven co-authors introduced the Transformer in 2017, for translation. Text is cut into tokens, and each token becomes a vector of $d_{\text{model}} = 512$ numbers. There is no recurrence and no convolution, and one odd consequence is that nothing in the model knows which token came first. Shuffle the words and, without help, each one gets the same treatment as before. So position is added by hand. Each position $p$ gets a fixed vector, added to its token’s:

$$\mathrm{PE}(p, 2i) = \sin\frac{p}{10000^{2i/d_{\text{model}}}}, \qquad \mathrm{PE}(p, 2i+1) = \cos\frac{p}{10000^{2i/d_{\text{model}}}}.$$

Each pair of dimensions is like a clock hand turning at its own speed, with periods from $2\pi$ to $10000 \cdot 2\pi$ positions. Going from position $p$ to $p + k$ turns every hand by an angle that depends only on $k$, so the dot product of two positions depends only on how far apart they are:

$$\mathrm{PE}(p) \cdot \mathrm{PE}(p+k) = \sum_{i} \cos(\omega_i k), \qquad \omega_i = 10000^{-2i/d_{\text{model}}}.$$

Most current models use a variant called rotary embeddings, which rotate the queries and keys instead of adding to the input. The toy model in this post uses them. Rotary attention fades with distance, which puts an instruction far back at a further disadvantage. Without positions, attention can’t tell one word order from another. That is a symmetry, and I use one later.

Attention

Each token produces three vectors, each a linear function of the token: a query $q$, a key $k$ and a value $v$. Stacked for all tokens,

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V.$$

One row at a time, it goes like this. A token’s query is compared with every key by a dot product, giving a score $s_j$ for each token $j$. The softmax turns scores into weights, $a_j = e^{s_j} / \sum_i e^{s_i}$, and the output is the average of the values with those weights. Every weight is positive, and the weights in a row add up to exactly 1. That is the lamp.

Why divide by $\sqrt{d_k}$? If the components of $q$ and $k$ are independent with mean 0 and variance 1, their dot product has variance $d_k$ (the paper’s footnote 4). Without the division, the scores grow with the width of the head, the softmax puts nearly all its weight on one token, and the gradients vanish. You can try it:

A model that writes left to right can’t look at words it hasn’t written, so the scores of later positions are set to $-\infty$ before the softmax. It’s called the causal mask.

Heads, layers and training

The model runs $h = 8$ attentions side by side, each on its own 64-dimensional projections, and joins the results:

$$\mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\, W^O, \qquad \mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$

Heads can specialise: one might track the previous word, another the subject of a verb. After attention, each position goes through a small two-layer network on its own, $\mathrm{FFN}(x) = \max(0,\, xW_1 + b_1)\,W_2 + b_2$, with an inner width of 2048. Each part is wrapped as $\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))$. The $x +$ is a residual connection, a path around the layer, and because of it anything added at any depth still reaches the output. Six layers make the encoder, and the decoder has six more, with an extra part that attends to the encoder’s output.

The Transformer was built to translate one sentence at a time. Inputs were a few dozen tokens long, and one budget of attention was plenty. The failures in this post show up when the same design writes thousands of tokens against rules set far back.

Aside: how the 2017 model was trained

Training used Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$ and a learning rate that rises for 4,000 steps and then decays with the inverse square root of the step,

$$\mathrm{lr} = d_{\text{model}}^{-0.5} \cdot \min\!\left(\text{step}^{-0.5},\ \text{step} \cdot 4000^{-1.5}\right),$$

with dropout and label smoothing both at 0.1. The big model reached 28.4 BLEU on English to German and 41.8 on English to French, after 3.5 days on eight P100 GPUs.

The paper’s Table 1 compares kinds of layer:

layer	cost per layer	sequential steps	longest path between two tokens
self-attention	$O(n^2 d)$	$O(1)$	$O(1)$
recurrent	$O(n d^2)$	$O(n)$	$O(n)$
convolutional	$O(k n d^2)$	$O(1)$	$O(\log_k n)$

In a recurrent network, a word at the start of a document has to survive $n$ updates before it can affect the end. In a transformer, any token reads any other directly, in one step. It looks as if a transformer can’t forget. But every one of those one-step reads is paid for out of the same budget, and the budget adds up to 1.

Language models today keep only the decoder. They are trained to predict the next token,

$$p(y \mid c) = \prod_{t=1}^{L} p(y_t \mid c,\, y_{<t}),$$

and they write by sampling one token at a time and feeding it back in.

Two facts

I’ll use two facts about this design over and over, so here they are with numbers.

Fact 1: one budget. Every row of attention is a probability distribution over all the tokens it can see. What a head gives one token it takes from the others, and there is one more of those with every token written.

Fact 2: memory without state. Between two steps, a decoder passes on nothing except the tokens, with their keys and values. It keeps no running summary of the kind a finite automaton keeps. So anything the text has committed to, a font, a count, sources still owed, has to be found again at every step by attention over the whole context. The design never needed a margin, because any token can read any other, and so it doesn’t have one.

In the draft at the top, the double dollars were set in the example posts, thousands of tokens before the fourth paragraph, and every token since then has joined the crowd they compete with (Fact 1). And nothing was counting paragraphs or tracking the sources (Fact 2).

What coherence means

To check a promise, or keep one, you have to say exactly what it is. This section writes the five promises as formulas. They come in a few shapes, and each shape breaks at a different moment.

Obligations

In 1957 Noam Chomsky wrote “Colorless green ideas sleep furiously” to show that a sentence can be grammatical and mean nothing. Coherence, as I use the word, is a third property, closer to bookkeeping. The earlier posts in this notebook promise a way of writing formulas, a spelling, headings in sentence case, a closing section that explains where the post is filed, and sources at the end. A request for “nine paragraphs” promises nine. Formally, a context $c$ creates a set of obligations $\Phi(c) = \{\varphi_1, \dots, \varphi_k\}$, and an output $y$ is coherent with $c$ when

$$y \models \varphi_i \quad \text{for every } \varphi_i \in \Phi(c).$$

Linear temporal logic has names for most of the shapes. Amir Pnueli introduced it in 1977 for reasoning about programs that run for ever, and De Giacomo and Vardi gave it a meaning on finite traces in 2013. A text is a finite trace.

A count isn’t a temporal operator, but a finite automaton can still count up to a fixed number.

All of these can be checked while the text is being written. Bauer, Leucker and Schallhart (2011) give any prefix of a run a verdict: $\top$ if every continuation satisfies $\varphi$, $\bot$ if none does, and $?$ if it’s too early to tell. A text ends, and at the end every verdict is settled. An invariant can only go from $?$ to $\bot$, and it is kept if the text ends unbroken. An eventuality can only go from $?$ to $\top$, and it is broken if the text ends unpaid. Each obligation compiles to a small automaton that reads the output once and keeps a few bits as it goes. A monitor like this, giving a verdict on every prefix, can also teach a network, token by token, to keep the same books. The Ledger is trained that way.

What a transformer can and cannot do with them

These are regular languages, the bottom level of the hierarchy Chomsky described in 1956 and 1959. A finite automaton recognises them with a fixed amount of memory. How do transformers compare? Yang, Chiang and Angluin proved that transformers with hard attention, where each position attends to a single other position, a strict causal mask and no position embeddings recognise exactly the languages definable in linear temporal logic. Yang and Chiang compiled a temporal logic with counting into ordinary softmax transformers. So every obligation above can be written as a transformer. Delétang and colleagues trained 20,910 networks on 15 tasks arranged by the Chomsky hierarchy and tested them on longer inputs than they were trained on. Recurrent networks, which keep a state, generalised on the regular tasks, and only networks given an explicit memory they could update, a stack or a tape, generalised beyond them.

Put that next to Fact 2. An idealised transformer can express every obligation but has nowhere to keep one, so it has to express it again at every step, through attention. With hard attention that costs nothing, because one position is found exactly. With the soft attention we actually train, each of those re-expressions comes out of the shared budget of Fact 1.

Competence and performance

In Aspects of the Theory of Syntax (1965), Chomsky separated what a speaker knows, competence, from what a speaker does, performance. Linguistic theory, he wrote, is about an ideal speaker

unaffected by such grammatically irrelevant conditions as memory limitations, distractions, shifts of attention and interest, and errors (random or characteristic) in applying his knowledge of the language in actual performance.

The results about what transformers can express are about competence. The failures in the next section are performance, and they fit Chomsky’s list: memory limitations (no state), distractions (whatever text is nearest), shifts of attention (dilution). Methods that ask a model to criticise and revise its own draft, such as Self-Refine (Madaan et al. 2023), do improve the draft, which suggests the model can often see what went wrong. Subbarao Kambhampati, on Machine Learning Street Talk, argues for letting the model propose and external checkers judge, in a loop he calls LLM-Modulo. The Ledger takes the checkers inside. They teach the network during training, and a small state holds their verdicts while it writes.

Chomsky himself has little patience for language models. He discussed them on Machine Learning Street Talk in 2022, and in the New York Times in 2023 he and his co-authors called ChatGPT “a lumbering statistical engine for pattern matching”. I don’t need to settle that here. I borrow his distinction because it points at the machinery that has to keep what the model knows through a long performance.

Three ways to lose the thread

An invariant can break at any step. An eventuality breaks only at the end, when the text stops with the debt unpaid. A count breaks when the text stops at the wrong number because nothing was counting. There is one mechanism for each moment, and the length exponent comes from the second.

Dilution

The double dollars fade by dilution. They were set in the example posts, and every token written since then has joined the crowd they compete with.

Lemma 1. Suppose every attention score in a row satisfies $|s_j| \le B$. Then every weight satisfies

$$a_j = \frac{e^{s_j}}{\sum_{i=1}^{n} e^{s_i}} \le \frac{e^{B}}{n\, e^{-B}} = \frac{e^{2B}}{n}.$$

Proof. The numerator is at most $e^B$, and each of the $n$ terms of the denominator is at least $e^{-B}$. ∎

So the $m$ tokens of an instruction get at most $m e^{2B}/n$ of a head’s attention, and their share of what the head passes on is at most that fraction of the largest value vector. While the model writes, $n = n_0 + t$ grows by one per token. The lemma is a ceiling, and a model can work well below it. But to keep a fixed share on an instruction as the context grows, the model has to raise the instruction’s scores, and we can say by how much.

Corollary 1. Let $m$ instruction tokens have a score $\Delta$ above the other $n - m$ tokens, which share a common score. Their share of the head is at least $c \in (0, 1)$ if and only if

$$\Delta \ \ge\ \log \frac{c}{1 - c} + \log \frac{n - m}{m}.$$

Proof. The share is $a = m e^{\Delta} / (m e^{\Delta} + n - m)$. Rearranging $a \ge c$ gives $e^{\Delta} \ge \frac{c}{1-c} \cdot \frac{n - m}{m}$. ∎

The edge an instruction needs grows like $\log n$. With scores bounded by $B$, the edge can’t exceed $2B$, so the instruction’s share drops below any $c$ once $n > m\bigl(1 + e^{2B}(1-c)/c\bigr)$. That length is fixed by the weights.

Several results from other people fit here. Veličković and colleagues proved that softmax attention has to spread out as its inputs grow. Nakanishi’s scalable softmax multiplies each score by $s \log n$, which is the logarithmic growth the corollary asks for. Trained models put their spare attention on the first few tokens, the attention sinks of Xiao and colleagues. Federico Barbero’s “Transformers need glasses!”, which he discussed on Machine Learning Street Talk, shows distinct inputs ending up with nearly the same final representation. Rotary positions add their own decay with distance, as Su and colleagues showed, so an instruction at the start of a long context loses out twice: to the size of the crowd and to its distance from it.

A fix would need a channel whose normalisation leaves the context out.

Compounding

A promise kept for three paragraphs and broken in the fourth has usually been worn down by compounding. An invariant has to hold at every chance to break it, and a long text has many. Let $h_i(t)$ be the chance that invariant $i$ breaks for the first time at step $t$, given that it has held until then. Reliability engineers call this a hazard. If the invariants fail independently, the chance that all $k$ of them survive $L$ steps is

$$S(L) = \prod_{i=1}^{k} \prod_{t=1}^{L} \bigl(1 - h_i(t)\bigr),$$

and a product of many numbers a little under 1 shrinks exponentially. What happens next depends on how the hazards behave as the text grows. For that I need an assumption, and I’ll state it plainly, because everything after it depends on it.

The hazard model. The hazard of invariant $i$ at step $t$ depends on the past only through the largest share of attention any head gives to the evidence for it, $a_i(t)$. So $h_i(t) = g_i(a_i(t))$, where $g_i$ never increases (more attention never makes a slip more likely), $g_i$ is continuous at 0, and $g_i(0) = h_i^{\infty} < 1$ is the hazard of an invariant nobody is looking at. Given the past, different invariants fail independently.

None of this is a law of nature. Evidence can reach the output in ways that one attention share doesn’t capture, and failures can be correlated. The experiment tests three consequences of the model, and they hold, but the assumption stays an assumption.

Proposition 2 (the length exponent). Write $\kappa = -\lim_{L \to \infty} \frac1L \log S(L)$.

(a) One shared softmax. If the evidence for invariant $i$ is $m_i$ tokens of a prompt of $n_0$ tokens, and every score is bounded by $B$, then for every $n_0$

$$\kappa = -\sum_{i=1}^{k} \log\bigl(1 - h_i^{\infty}\bigr).$$

(b) A separate read. If the evidence for invariant $i$ is also held by one of $K$ slots, read through a softmax over the slots alone in which slot $i$ scores at least $\Delta_i$ above every other slot, then for every $n_0$ and every $L$

$$S(L) \ \ge\ \prod_{i=1}^{k} \bigl(1 - g_i(c_i)\bigr)^{L}, \qquad c_i = \frac{e^{\Delta_i}}{e^{\Delta_i} + K - 1},$$

so $\kappa \le -\sum_i \log\bigl(1 - g_i(c_i)\bigr)$. That bound mentions neither the prompt nor the length of the text.

Proof. Under the hazard model, the chance of no break at step $t$, given the past, is $\prod_i \bigl(1 - g_i(a_i(t))\bigr)$. If at every step that number lies between two bounds that don’t depend on the past, the chance of surviving $L$ steps lies between the products of those bounds. (a) By Lemma 1, every head gives the evidence at most $b_i(t) = \min\{1,\, m_i e^{2B}/(n_0 + t)\}$. Since $g_i$ never increases, $g_i(b_i(t)) \le h_i(t) \le h_i^{\infty}$, so $S(L)$ lies between $\prod_t \prod_i \bigl(1 - h_i^{\infty}\bigr)$ and $\prod_t \prod_i \bigl(1 - g_i(b_i(t))\bigr)$. As $t$ grows, $b_i(t) \to 0$ and $g_i(b_i(t)) \to h_i^{\infty}$, so both products have the same exponent. (b) A softmax over $K$ slots gives slot $i$ at least $c_i$ of its weight whatever the context holds, so $h_i(t) \le g_i(c_i)$ at every step. ∎

Part (a) says that with the shared softmax, the exponent ends up equal to that of an obligation whose evidence is never read. For a long enough text, the model keeps the rule about as well as it would by guessing. The length of the prompt drops out. A longer prompt doesn’t change the limit; the evidence just starts out fainter, so the model gets there sooner. Part (b) says that with a separate read, the exponent no longer depends on any length. It isn’t zero, and slips still happen, but their chance stops growing as the page fills.

Dziri and colleagues measured the same arithmetic in problems with many steps, where small error rates per step multiply. A long post is a problem with many steps, and every paragraph is a step for every invariant:

The theory predicts two things the experiment can test. First, sparse and dense invariants should behave differently. An invariant used on every line, like the font of each paragraph, can be read off the model’s own recent output. One used rarely, like the delimiter of a formula that appears once every few hundred words, has to be found in the examples each time, across a widening gap. So per chance to break, the sparse one should be more fragile. (A dense invariant can still fail more often per post, because it gets more chances.) Second, the product formula should predict how often whole posts keep all their invariants, starting from nothing but the hazards per opportunity.

A fix would keep each obligation as state, so that each step reads it instead of rebuilding it, and read that state through a softmax the crowd can’t dilute. For some obligations the choice can be removed altogether; the section on symmetry shows which.

Blind stopping

The post that stops after seven paragraphs, with no sources, fails at the end. Eventualities and counts are judged only at the end, and the end is itself a token, $\langle\text{eos}\rangle$.

Proposition 3. Suppose that at each step while an eventuality $q$ is owed, the model ends the text with probability $e$, and otherwise pays $q$ with probability $d$. Then

$$P(\text{the text ends with } q \text{ unpaid}) = \sum_{t \ge 0} e\,\bigl((1 - e)(1 - d)\bigr)^{t} = \frac{e}{e + d - e d}.$$

Proof. The text ends unpaid at step $t$ if it neither ended nor paid at each earlier step, which has probability $((1-e)(1-d))^t$, and then ends. Sum the geometric series. ∎

With rates that change over time, the same sum holds term by term. The outcome depends on the rate of stopping compared with the rate of paying, and in a transformer nothing links the first to what is owed. The decision to stop is one entry of one softmax, computed from the same diluted state as everything else. Two older results show how fragile that entry is. Stahlberg and Byrne searched a translation model exactly and found that for more than half of the sentences, the model’s single most probable output was the empty translation. Newman, Hewitt, Liang and Manning found that training a model to predict the end token at all makes it generalise worse to longer outputs, because its hidden states organise themselves around position.

The fix is to make the rate of stopping depend on the debt, so that $e$ falls while $q$ is owed.

The Ledger

From here on, three voices interrupt now and then. The Model wrote the draft at the top of the post. The Reader asks why any of it matters. The Clerk is part of the machine in this section.

The Ledger gives the obligations a state, which counters compounding and makes counting possible. It reads that state through a softmax of its own, which counters dilution. And it lets the state hold back the end of the text. It sits beside an ordinary decoder and leaves the decoder as it was:

The clerk. Once the prompt has been read, $K$ learned queries attend over the prompt’s hidden states at a middle layer $\ell_c$ and write $K$ slot vectors $o_1, \dots, o_K$, one line of the book per obligation. The clerk runs once per prompt. It reads implicit rules off the examples, and small linear heads on each slot (which font, whether sources are expected, how many paragraphs) let training check what it read.

The state. This is the margin the scribe didn’t have. Every written token updates one number per slot. With $h_t$ the token’s normalised state at layer $\ell_c$, the chance that this token settles something for slot $i$ is $e_{t,i} = \sigma\bigl(\langle W_h h_t,\, W_o o_i\rangle/\sqrt{d} + b_i\bigr)$, and the status $u_{t,i}$ builds up according to the slot’s type:

$$u_{t,i} = \begin{cases} 0 & \text{invariant: always in force} \\[2pt] 1 - \prod_{s \le t} (1 - e_{s,i}) & \text{eventuality: noisy-OR, monotone} \\[2pt] \sigma\bigl(4\,(\textstyle\sum_{s \le t} e_{s,i} - \hat n_i + \tfrac12)\bigr) & \text{count, with target } \hat n_i \text{ read by the clerk} \end{cases}$$

The middle line is one minus the chance that no token so far paid the debt. The bottom line is a soft version of “have we reached $\hat n_i$ yet?”. These are the monitors’ automata, written so gradients can pass through them. An eventuality or a count is pending with weight $\pi_{t,i} = r_i (1 - u_{t,i})$, where $r_i$ is the clerk’s estimate that the obligation applies at all. An invariant is never pending, just in force.

The read. In every layer above $\ell_c$, each written token attends to the $K$ slots, whose keys and values come from $o_i$ and from $(u_{t,i}, \pi_{t,i}, r_i)$. This softmax, one per head, is normalised over the $K$ slots and never over the context, so Fact 1 doesn’t apply to it:

$$x_t \leftarrow x_t + \tanh(g_\ell)\; W_O \sum_{i=1}^{K} \mathrm{softmax}_i\!\left(\frac{q_t \cdot k_{t,i}}{\sqrt{d_h}}\right) v_{t,i}.$$

The gate. Pending eventualities and counts hold back the end of the text. With $\varepsilon = 10^{-3}$,

$$\text{logit}_t(\langle\text{eos}\rangle) \mathrel{+}= \gamma \sum_{i \,\in\, \text{eventualities, counts}} \log\bigl(1 - \pi_{t,i} + \varepsilon\bigr).$$

While a debt is fully owed ($\pi = 1$), this multiplies the odds of stopping by $\varepsilon^{\gamma} = 10^{-3\gamma}$, which is what Proposition 3 asks for. The toy learned $\gamma$ between 0.30 and 0.42, so the odds of stopping with a debt outstanding were cut by a factor of 8 to 18.

The Clerk’s book for the draft at the top, as its seventh paragraph ends, would read:

Training. Checking is cheap, so the checkers can teach. Next to the usual next-token loss, the clerk’s heads learn from a label for each obligation, and the state learns from the monitors’ verdicts on every prefix of the training text: have the sources started yet, and, for a count, does a paragraph start here. Every training token becomes a small lesson in bookkeeping.

$$\mathcal{L} = \mathcal{L}_{\text{LM}} + \lambda_{\text{ext}} \sum_i \mathrm{CE}(\text{clerk}_i, \varphi_i) + \lambda_{\text{ev}} \sum_{t, i} \mathrm{BCE}\bigl(u_{t,i},\, [\![ y_{\le t} \text{ has paid } \varphi_i ]\!]\bigr).$$

What the Ledger buys

Five properties follow from the definitions.

1. It changes nothing when inserted. With $g_\ell = 0$ and $\gamma = 0$, $\tanh(g_\ell) = 0$ and the gate adds nothing, so the network computes exactly what it computed before. A trained model can be extended without being disturbed. The zero-initialised tanh gate comes from Flamingo, which used it to add new inputs to a frozen model. In the toy, every placement tested below matched the baseline’s loss to 12 decimal places at insertion.
2. Its share doesn’t dilute. The read is normalised over $K$ slots, so by the argument of Lemma 1 a slot with an edge $\Delta$ over the others gets at least $e^{\Delta}/(e^{\Delta} + K - 1)$ of the read, whatever $n$ and $t$ are. Part (b) of Proposition 2 needs exactly this.
3. Paid stays paid. An eventuality’s or a count’s status can never go down. A product of numbers between 0 and 1 only shrinks, and a sum of non-negative terms only grows. So once a debt is settled, the gate lets go.
4. It trains in parallel. The noisy-OR is a running sum of $\log(1 - e_{s,i})$ and the count a running sum of $e_{s,i}$, so training needs no back-propagation through time, even though the state is a recurrence.
5. It is cheap, and you can read it. A slot’s state is $o_i$ plus a linear function of three numbers, so its keys and values are $W o_i$, computed once per prompt, plus a rank-3 correction. Per written token and read layer that leaves one query projection and $K$ dot products, $O(d^2 + Kd)$, with no term in $n$; self-attention costs $O(d^2 + nd)$. And you can read the slots out: what the clerk thinks is owed, and what the model thinks it has paid.

Why not something simpler? Repeating the instruction, or moving it to the end of the prompt, brings it closer but keeps it in the shared softmax. Reweighting attention, as split-softmax, PASTA and scalable softmax do, gives marked tokens more of the budget, but there is still no state and nothing tied to the stop. Memory or register tokens add learned vectors, but inside the same softmax, where they dilute like everything else. Checking and retrying from outside the model (Self-Refine, LLM-Modulo, checklist rewards) catches failures after the fact, at the price of whole extra generations. Constrained decoding with automata does keep state and can block a bad stop, but someone has to write the constraint out as a formula; it can’t learn a rule from examples. The Ledger deals with all of these at once and learns the rules from the examples, for an extra cost per token of $O(d^2 + Kd)$, with no term in $n$.

Keeping state beside the language machinery is an old idea. William Woods’ augmented transition networks, in 1970, kept registers alongside the grammar, holding the parts of a sentence found so far (its subject, say) so that later tests, such as agreement, could consult them. Among neural models, the Recurrent Memory Transformer and Block-Recurrent Transformers pass a learned state from one stretch of a long input to the next. What I haven’t found anywhere is this particular combination: one typed line per obligation, filled from examples, taught by monitors, read through its own softmax, and able to hold back the end of the text.

Aside: more earlier work

Kiddon and colleagues’ neural checklist (2016) has a recurrent network track which items of a given agenda it has mentioned; the Ledger infers its agenda from examples and tracks invariants, order and counts as well. Coverage models for translation (Tu et al. 2016) track how much of each source word has been translated, which is coverage of the input, not of obligations. Ma and colleagues’ progress monitor (2019) has a navigation agent estimate how much of its instruction it has completed, the same instinct without types or a stop gate. Equivariant networks (Cohen and Welling 2016) and learned canonicalisation (Kaba et al. 2023) build a symmetry of the input into a model; gauge fixing, below, applies the symmetry to the style of the output. Gordon and colleagues (2020) and Akyürek and Andreas (2023) use permutations of words to generalise meanings; here the group fixes a convention, read once and enforced exactly. Gist tokens (Mu et al. 2023) compress a prompt by masking attention; with the frame and the clerk, what survives the compression is typed and checked.

Symmetry: when the state collapses

For one class of invariants, you can do better than a lower hazard, and mathematicians met the reason long ago under the same word.

Coherence in mathematics

In a monoidal category you can multiply objects, and a product like $(a \otimes b) \otimes (c \otimes d)$ can be bracketed in several ways. With four objects there are five bracketings, joined by chains of associativity isomorphisms, and usually more than one chain between two of them. Saunders Mac Lane’s coherence theorem (1963) says any two such chains between the same bracketings are equal. Every diagram built from them commutes. Mac Lane later gave the same fact another form in Categories for the Working Mathematician: every monoidal category is equivalent to a strict one, where those isomorphisms are identities and there is nothing left to choose.

I prefer the second form. Coherence there means that choices made in different places agree. Strictification gets that agreement by removing the choices.

A lot of writing conventions are choices of this kind. The double dollars and sentence case are among them, and so are the font and “colour” against “color”: each is a pick among interchangeable variants. A text keeps such a convention when it picks the same way everywhere. The Ledger helps the network remember its pick. Strictification suggests a better plan, which is to pick once, outside the network, and leave nothing to remember.

A group acting on words

Let a finite group $G$ act on the vocabulary $V$ by permutations, $\rho : G \to \mathrm{Sym}(V)$ with $\rho(gh) = \rho(g)\,\rho(h)$, extended to texts one token at a time. In the toy below, $G = S_6 \times S_4 \times S_3$. One factor permutes the six fonts, one the four heading styles, one the three delimiters (opening and closing together), and every other token stays put. The orbits $G \cdot v = \{\rho(g)\,v : g \in G\}$ are the style families plus single tokens, and a style is one member from each family. Attention has a symmetry of the same kind, acting on positions instead of words: without positional encodings and without the mask, it is equivariant to any reordering of the tokens, which is why the 2017 paper had to add positions. The Geometric Deep Learning programme of Bronstein, Bruna, Cohen and Veličković, the subject of a special edition of Machine Learning Street Talk, builds whole architectures out of symmetries like these.

Write $C(x, q)$ for the set of coherent posts, given example posts $x$ and a request $q$.

Proposition 4 (equivariance). For every $g \in G$, $C(\rho(g)\,x,\ q) = \rho(g)\,C(x, q)$. Relabel the author’s examples, and the coherent posts are relabelled the same way.

Proof. Each style monitor checks that every token of its family equals the member the examples use, and applying one permutation to both sides keeps that equality. The other monitors never ask which member of a family appears, and from the request they read only the requested length, which $G$ leaves alone. ∎

A network that learns this from data has to learn it separately for every style, and it sees the rare styles least. In the toy’s baseline (two seeds, 600 posts), authors who used the default font had it kept 96.7% of the time, and authors who used one of the other five, 91.2%. For delimiters the figures were 97.2% and 90.4%. The task has the symmetry and the model doesn’t know it.

Gauge fixing

In physics, a gauge is a choice of reference frame that the laws don’t depend on, and fixing the gauge removes the redundancy. The same move works here. Choose a canonical style $c$, the first member of each family. Read the author’s style $\hat s$ from the examples, and let $\hat g$ be a group element with $\hat g \cdot c = \hat s$. (In the toy it’s a product of three transpositions, and its own inverse.) Then:

1. move the context into the canonical frame, $x \mapsto \rho(\hat g)^{-1} x$;
2. let the transformer, trained the same way in the canonical frame, write with its output restricted to the canonical member of every family, so that its alphabet is the set of orbits $V/G$ (96 tokens become 84);
3. map the text back, $y = \rho(\hat g)\, y_c$.

Proposition 5 (invariants by construction). Under gauge fixing, the invariant of family $i$ holds at every position of the text if the frame read it correctly, and fails at every occurrence of that family otherwise. For every length $L$,

$$P(\text{invariant } i \text{ is broken}) \le P(\hat s_i \ne s_i),$$

with equality whenever the family occurs at all.

Proof. The only member of family $i$ the network can emit is $c_i$, and step 3 maps it to $\rho(\hat g)\,c_i = \hat s_i$ wherever it appears. So every occurrence matches the author if $\hat s_i = s_i$, and none does otherwise. ∎

Proposition 6 (reading the frame). Suppose the examples contain $m$ tokens of family $i$, each equal to the author’s member independently with probability $1 - \eta$, where $\eta < \tfrac12$. Then the most frequent member is the author’s with probability at least $1 - e^{-2m(1/2 - \eta)^2}$.

Proof. Let $X \sim \mathrm{Bin}(m, 1 - \eta)$ count the author’s tokens. If $X > m/2$ the author’s member is a strict majority. By Hoeffding’s inequality, $P(X \le m/2) = P\bigl(X - m(1 - \eta) \le -m(\tfrac12 - \eta)\bigr) \le e^{-2m(1/2 - \eta)^2}$. ∎

Theorem 1 (a length exponent of zero). Under gauge fixing, with the frame read by majority count and the assumptions of Proposition 6 for each family $i$ (with $m_i$ examples and noise $\eta_i$), the probability that every group-structured invariant holds throughout a text of any length $L$ is at least

$$1 - \sum_i e^{-2 m_i (1/2 - \eta_i)^2}.$$

The bound doesn’t depend on $L$, so whenever it is above zero, the length exponent of these invariants is $\kappa = 0$.

Proof. By Proposition 5 the text keeps all these invariants whenever every family’s frame is read correctly, whatever $L$ is. By Proposition 6 each family is misread with probability at most $e^{-2 m_i (1/2 - \eta_i)^2}$. A union bound over the families gives the result. ∎

The three cases have the same shape: one reading of the prompt, then a hazard at every step. The shared softmax does no single reading, and its hazard climbs towards $h^{\infty}$. The Ledger reads once, through the clerk, and keeps its hazard bounded. Gauge fixing reads once, through the frame, and has no hazard left, because all the risk has gone into that single reading before the first word. More examples make that reading more reliable, while more text made the hazards worse. In the language of the last section, many places that had to agree have been replaced by one choice.

What it saves

Gauge fixing also saves work. Once the clerk and the frame have read the examples, the written tokens need the examples only for their content. When the examples are there only for their conventions, as in the toy, their keys and values can be dropped as soon as they’ve been read. Each written token then attends to the request, its own text and the $K$ slots, and a post of $L$ tokens after $n_{\text{ex}}$ example tokens saves $L\, n_{\text{ex}}$ key lookups per head and layer. The output alphabet shrinks from $V$ to $V/G$ too, so the network stops spending capacity on relabelled copies of the same behaviour.

It doesn’t work for everything. The group has to act on the units the model writes, and “colour” and “color” can be split into different tokens, while lower-casing a heading can’t be undone. Examples that matter for their content, like voice, topic and facts, can’t be dropped; only their conventions can be factored out. And obligations with no symmetry, like “end with sources” or “write nine paragraphs”, stay with the Ledger.

Where coherence belongs

I’ve been saying “at the output”, “inside”, “along the text”. Each part of the repair answers one failure, and each failure happens at one place in the network, so the placement can be worked out instead of guessed.

Start with the residual stream. Elhage and colleagues describe a transformer as a shared channel that every attention layer and feed-forward layer reads from and adds to, with the logits reading the final sum. Anything written into the stream at any depth reaches the output. Depth decides what the rest of the network can do with it.

Before, in the context. Written as tokens in the prompt, obligations compete in the same softmax as everything else (Lemma 1), and with rotary positions they also end up further back with every token. Corollary 1 rules this out.

Inside, before or after attention. Added to the stream just before a layer’s attention, the Ledger’s read can change what that attention looks for: “sources still owed” becomes “find how the examples wrote their sources”. Added just after, it reaches the feed-forward layer and the layers above. Either way the network can compute with it.

Inside, as modulation. The Ledger could scale and shift the normalised stream instead. Peebles and Xie compared these options for conditioning diffusion transformers (tokens in the context, cross-attention, and zero-initialised adaptive layer norm); the last gave the best images at the lowest cost, and tokens in the context the worst.

After, at the output. Biasing the logits, gating $\langle\text{eos}\rangle$ or restricting the alphabet can forbid or choose tokens exactly, but can’t make the network look anything up or plan.

Across, along the sequence. Attention keeps no state (Fact 2). Regular languages are naturally kept by a small state updated as the text goes by, and the Ledger’s status is that: a small recurrence running along the positions, next to attention.

The rule I end up with: act on each obligation at the lowest place where it can be both checked and acted on. A choice among interchangeable variants breaks at a single output token, so it belongs after, by restricting the alphabet. An eventuality or a count breaks at the decision to stop, so it needs a status along the text, a read of it inside, and a gate after. An order, or anything that needs content looked up, belongs inside, before attention. Voice, topic and facts stay with ordinary attention.

I got the counts wrong at first. My first table put them at the output, with the gate alone, since a count is judged when the text stops. The placement screen in the experiment below disagreed. A gate can refuse to stop, but it can’t plan the paragraphs that make the count come out right, so a count needs its status read inside the network.

An experiment

I can’t retrain frontier models, so I built the smallest world that has the problem, and set it up so that each numbered result above has a measurement that could have gone against it.

The task

An author has a house style: one of six paragraph fonts, one of four heading styles, one of three formula delimiters, and two habits, a closing “why” section and a sources section, each present or absent. It’s the opening scene, shrunk. The fonts, heading styles and delimiters stand in for the double dollars and sentence case, the two habits for sources last, and the request for the nine paragraphs. The context shows two or three of the author’s posts, then a request, a line in the default style (plain serif, single-dollar formulas) saying how many paragraphs to write. Then the model writes. Nobody states the style. The model has to read it off the posts and hold it against the request, which sits right next to where writing begins. The words come from a fixed random chain over 48 words and mean nothing, so only form is being tested.

Most authors in the training data use the default style (40% of fonts, half of all delimiters), so a model that stops looking at the evidence slides towards the request’s style. Seven monitors check each post for the implicit rules: font, headings and maths (the three invariants), the why-section and the sources (eventualities, with sources required to come last), their order, and form, meaning no token that belongs to the prompt. An eighth checks the requested length, the one explicit instruction and the one count.

The models

There are six variants, identical except for the parts under test. Each is a 3-layer decoder with 4 heads, width 64 and rotary positions (155,456 parameters), trained from scratch for 1,500 steps on fresh batches of 32 episodes with AdamW (peak learning rate 0.002, cosine decay, gradient clipping at 1), with the loss taken only on the post the model writes. The Ledger’s two label losses are weighted 0.3 each.

1. baseline: the plain decoder.
2. baseline + same labels: the plain decoder with the Ledger’s supervision attached to linear probes on its hidden states. If the Ledger wins only because it sees more labels, this one should win too.
3. Ledger: clerk at layer 1, reads after attention in layers 2 and 3, $K = 8$ slots (three invariants, two eventualities, one count, two left free and unsupervised), the gate on the eventuality and count slots, and supervision (212,427 parameters).
4. Ledger, one shared softmax: the same slots, read inside the ordinary attention softmax, as tokens in the context would be. Same state, same labels, shared budget: this isolates Fact 1.
5. Ledger + frame: the Ledger with gauge fixing.
6. Ledger + frame, examples dropped: as 5, but written tokens can’t attend to the example posts. Only the clerk and the frame read them, so their keys and values can be thrown away before writing starts.

Each trained model wrote 300 posts, one for each of the same 300 prompts (two example posts each; 5 requested lengths, 60 authors each), sampled at temperature 1 so the numbers describe the model and not a decoding trick. Every model sees the same prompts, so two models can be compared prompt by prompt with an exact McNemar test.

It all ran on four CPU cores, with no accelerator. A run takes about twenty minutes to train and ten to evaluate, and I kept to that budget. The baseline and the Ledger got two seeds each; the labelled baseline, the shared-softmax Ledger and the two Ledgers with a frame got one each. The other ablations didn’t fit, and they’re listed at the end. Placement got a cheaper screen: all six placements and the baseline, one seed each, trained for 400 steps instead of 1,500 and scored on 100 prompts. A screen like that ranks candidates; it doesn’t settle anything.

Results, claim by claim

Across 600 posts from two seeds, the baseline kept all seven implicit rules in 80.5% of them (77.3% and 83.7% for the two seeds). The Ledger kept them in 90.0% (89.0% and 91.0%). On the same prompts and seeds, the Ledger succeeded where the baseline failed 84 times, and the reverse happened 27 times; the exact McNemar test gives $p = 5.5 \times 10^{-8}$. With the frame, the Ledger reached 95.0%, and with the examples dropped 93.7% (one seed each; $p = 3 \times 10^{-6}$ and $1 \times 10^{-9}$ against the baseline on the same prompts).

monitor	baseline	baseline + same labels	Ledger, one shared softmax	Ledger	Ledger + frame	Ledger + frame, examples dropped
font	93% ±2	95%	93%	96% ±0	100%	100%
headings	99% ±1	99%	99%	99% ±0	100%	100%
maths	94% ±1	96%	97%	98% ±0	100%	100%
why-section	97% ±0	95%	88%	99% ±2	100%	98%
sources last	98% ±2	98%	94%	99% ±0	97%	100%
order	100% ±0	100%	100%	100% ±0	100%	100%
form	97% ±0	95%	98%	99% ±0	97%	95%
length (explicit)	39% ±4	40%	93%	99% ±1	97%	97%
all seven implicit	81% ±4	83%	74%	90% ±1	95%	94%

Dilution (Lemma 1). In the baseline, the share of attention that written tokens pay to the example posts halves over a post, from 0.66 in the first ten tokens to 0.34 at tokens 80 to 99, averaged over layers. The Ledger model’s ordinary attention fades the same way, from 0.69 to 0.34. So the Ledger doesn’t stop dilution. It gives the obligations a channel that dilution can’t reach.

The hazards in the three regimes. Take the delimiter of a formula, the sparsest invariant. Per chance to break it, the baseline broke it 3.96% of the time (95% interval 2.82% to 5.54%), and the Ledger 1.51% (0.88% to 2.57%); the intervals don’t overlap. The frame never broke it. The same slots inside the shared softmax came out in between, at 2.26%, with an interval that overlaps both neighbours, which is roughly what Corollary 1 leads you to expect from slots that compete with the context.

Sparse against dense. Per chance, the baseline broke the maths delimiter, which appears once every few paragraphs, 3.96% of the time, and the font, which starts every paragraph, 0.75% of the time. The sparse invariant is about five times as fragile, as predicted. The font hazard also rises the way part (a) of Proposition 2 describes: zero at the first paragraph, 0.51% in paragraphs 2 to 4, 1.09% in paragraphs 5 to 8, and 0.84% after that.

The product formula. Take the hazards per opportunity alone, and each post’s number of opportunities, and predict the share of posts that keep each invariant and all three together. For single invariants the predictions match the measurements, partly by construction, since the hazards came from the same posts. The joint prediction is a real test, because it assumes the three invariants fail independently, and nothing in the task forces that. It comes within half a point of the measured joint rate for every model.

model	font	headings	maths	all three
baseline	92.8 / 92.8	98.8 / 98.8	94.7 / 93.7	87.0 / 86.5
baseline + same labels	95.0 / 95.0	99.0 / 99.0	95.7 / 95.7	90.0 / 90.0
Ledger, one shared softmax	93.1 / 93.0	98.7 / 98.7	96.7 / 96.7	88.8 / 88.7
Ledger	96.0 / 96.0	99.2 / 99.2	97.8 / 97.8	93.2 / 93.2

Each cell is predicted / measured, in percent. The Ledger lowers every hazard (maths 3.96% → 1.51%, font 0.75% → 0.41%), and the product formula turns those lower hazards into whole-post rates: 86.5% → 93.2% for all three invariants together.

The separate budget (Fact 1). Put the same slots, with the same state and labels, inside the ordinary softmax, and they don’t help: 74.0% against the baseline’s 80.5%, a difference the paired test can’t tell from zero ($p = 0.29$, one seed). The why-section and the sources suffered most, kept in 88.0% and 93.7% of posts. Only the normalisation differs from the Ledger.

The state (Fact 2). The largest single gain is on the count. 98.5% of the Ledger’s posts had the requested number of paragraphs, against 38.5% for the baseline. The labelled baseline, which sees the same paragraph labels through probes on its hidden states, got 40.3%. Labels without a state to accumulate them hardly help. The labelled baseline is 82.7% coherent overall (one seed; $p = 0.08$ against the baseline), well short of the Ledger. Coherence still falls as the requested length grows, from 85.8% at four paragraphs to 70.8% at sixteen for the baseline, and from 95.0% to 87.5% for the Ledger. The drop is halved, not gone.

Blind stopping (Proposition 3). In this toy, even the baseline rarely stops too early: it kept the why-section in 97.0% of posts and the sources in 97.5%. The Ledger raises these to 98.5% and 99.0%. With $\gamma$ between 0.30 and 0.42, the gate cuts the odds of ending with a debt outstanding by 8 to 18 times. The ablation that would isolate the gate, the Ledger without it, didn’t fit the budget, so this effect is small and not confirmed on its own.

Symmetric rules (Theorem 1). Under gauge fixing, the frame was read correctly for every prompt, and no invariant broke: none in 3,006 font opportunities, 1,069 headings and 451 formulas, and none in 3,007, 1,076 and 422 with the examples dropped. The examples here never deviate ($\eta = 0$), so Theorem 1’s bound is 1. What still goes wrong for the frame models is everything the frame doesn’t cover: form, the eventualities, the count.

Dropping the examples. It cuts the keys each written token attends to from 119.6 to 43.2, 64% fewer, and coherence stays at 93.7%. In a model this small, the fixed cost of the projections and feed-forward layers dominates, so the total falls less: about 199,000 operations per written token for the full architecture, the same as the baseline and 13% less than the Ledger alone. With long examples, where attention dominates, the saving grows.

model	coherent	keys attended per written token	attention per token	all operations per token
baseline	81%	119.5	46k	199k
baseline + same labels	83%	119.6	46k (+0%)	200k (+0%)
Ledger, one shared softmax	74%	120.4	46k (+1%)	207k (+4%)
Ledger	90%	119.6	46k (+0%)	229k (+15%)
Ledger + frame	95%	120.3	46k (+1%)	229k (+15%)
Ledger + frame, examples dropped	94%	43.2	17k (−64%)	199k (−0%)

Placement (a screen). At 400 training steps, with one seed and 100 prompts, every model is still learning, so only the comparisons mean anything.

where	coherent	why-section	sources last	length	Ledger operations per token
nowhere: the baseline	17%	69%	62%	5%	0
before: in the context	20%	72%	66%	18%	7k
inside, before attention	35%	68%	83%	76%	29k
inside, after attention	37%	65%	83%	62%	29k
inside, top layer only	30%	65%	82%	67%	17k
inside, as modulation	31%	58%	83%	41%	17k
after: logits only	44%	68%	86%	17%	7k

Slots in the context did no better than no slots at all (20% against 17%), as Corollary 1 predicts for anything sharing the softmax. Every placement with a channel of its own roughly doubled the baseline (30% to 44%), and with 100 prompts the screen can’t separate them: ten points is within noise. One contrast is big enough to read. With the Ledger acting on the logits alone, the surface rules did best (maths 89%, form 98%) and counting did worst (17% of posts had the right length), while a read before attention counted best (76%). That’s the correction to the counts I mentioned.

Try it

Putting it in a real model

The toy doesn’t say whether any of this helps a model that already speaks. I’ve built two tests of it on SmolLM2-135M-Instruct, a pretrained instruction model with 30 layers and a context of 8,192 tokens: one on IFEval, whose rules are stated, and one on rules that are only shown. The code is public and tested. The runs haven’t happened yet, so this section gives the design and the predictions, with no numbers.

IFEval is already written in the three shapes

All but two of IFEval’s instruction types are obligations of the kinds in the toy. “No commas” is like the double dollars, “end with a postscript” like sources last, “at least 300 words” like nine paragraphs.

Each type gets one slot of its shape, so nothing needs matching. The monitors already exist too: the state’s training labels are IFEval’s own checkers, run on every prefix of a response. For a count, the label marks the token where a prefix first gains a counted unit; for an eventuality, the token where it first pays. On a test set of texts, these labels agree with the official checkers at every threshold.

Why the read must be separate, again

A module added to a working model should start by doing nothing and learn from there. The Ledger does. Its read comes in through $\tanh(g)$ with $g = 0$, so the model is unchanged at insertion, and the gradient with respect to $g$ is the read itself, so training moves it at once. The obvious alternative is to add the same slots as extra keys and values inside the model’s own attention, as memory tokens and prefix tuning do. That doesn’t work the same way, and I found out the hard way.

In my first version of the retrofit, the control that puts the slots inside the model’s own softmax scaled their share by $\lambda = \max(\phi, 0)$, starting at $\phi = 0$ so the model began unchanged. It never learned anything. Its share stayed at exactly zero from the first training step to the last, and I went looking for a bug. There wasn’t one. An early gradient step had pushed $\phi$ just below zero, where the share is flat at zero and every gradient vanishes, and the slots stayed shut for good. Once I understood it, the failure became a small proposition.

Proposition 7. Let a frozen head output $y = \sum_j a_j v_j$ with $a = \mathrm{softmax}(s)$, and add slot values $w_i$ with scores $\sigma_i$ to the same softmax, with a share multiplier $\lambda(\phi) \ge 0$:

$$y' = \frac{\sum_j e^{s_j} v_j + \lambda \sum_i e^{\sigma_i} w_i}{\sum_j e^{s_j} + \lambda \sum_i e^{\sigma_i}}.$$

Then $y' = y$ on every input only if $\lambda = 0$ (unless every $w_i$ equals $y$ on every input), and if $\lambda$ is differentiable, every derivative of $y'$ with respect to the new parameters is zero there.

Proof. $y' - y = \lambda \sum_i e^{\sigma_i}(w_i - y) / (Z + \lambda Z_\sigma)$, with $Z$ and $Z_\sigma$ the two sums of exponentials, which is zero everywhere only if $\lambda = 0$ or every $w_i = y$. At $\lambda(\phi_0) = 0$ the nonnegative function $\lambda$ is at its minimum, so its gradient is zero, and so is that of $y'$ with respect to $\phi$. Every term involving the slots’ own parameters has the factor $\lambda$. ∎

A share of one budget can’t be negative, so a share that starts at exactly zero sits at the bottom of its range, where the ground is flat. Put into a shared softmax without changing the model, slots never learn. Put in open, they change the model before they’ve learned anything. It’s Fact 1 again, this time deciding whether a module can be added at all. The control I now use starts barely open, with $\lambda = \phi^2$ and a share of $10^{-4}$.

The test

The model is frozen, and only the new parameters learn. The clerk reads the output of layer 10, one learned query per slot; numbers written in the prompt reach it as values, and a pointer picks each count’s target. The reads follow self-attention in layers 11 to 30, and the gate acts on the end-of-turn token. That is 3.96 million parameters, 2.9% of the model.

There are three controls. LoRA adapters with exactly as many parameters, trained on the same data, test whether it’s the architecture or just the training. The same slots and state inside the model’s own softmax test the separate budget. The Ledger without its gate tests the stop. The training prompts are plain writing tasks with one to three instructions, worded by IFEval’s own instruction classes but never taken from its test prompts. The targets are the frozen model’s own answers: the first of four samples that passes every checker, or else the best one repaired by exact edits, like lowercasing it or adding the missing postscript, and kept only if it then passes. The evaluation is the official IFEval on its 541 prompts, then the same prompts followed by up to 6,000 tokens of unrelated text, which changes the prompt length $n_0$ as Proposition 2 needs.

My own hardware is modest, so I sized the whole thing for a laptop GPU with 4 GB of memory. It also runs on the free GPUs of Google Colab and Kaggle; there’s a notebook for each that installs everything, runs the tests and saves its progress, so a session that times out picks up where it stopped. On Kaggle’s two GPUs, the work is split between them. The machine I built it on couldn’t download the model, so every part was tested on a random model of exactly SmolLM2’s shape: 414 tests across both studies, including one that checks the Ledger leaves the model’s outputs unchanged at insertion and one that checks that writing step by step gives the same answer as a single pass.

Predictions, before the numbers

Richard Feynman’s first principle of honest science comes in two phrases: “you must not fool yourself”, and “you are the easiest person to fool”. Writing predictions down before the runs is my protection against that. Here they are.

1. The Ledger raises IFEval’s strict accuracy over the frozen model. Whether it beats LoRA on short prompts is an open question, since both learn from the same data.
2. Its gains are largest on eventualities and counts, the obligations its state tracks.
3. As the unrelated text grows to 6,000 tokens, the frozen model, LoRA and the shared-softmax slots lose accuracy, and the Ledger loses less. If this one fails, the argument is in trouble.
4. The shared-softmax slots never beat the Ledger, and fall further behind as the context grows.
5. Without its gate, the Ledger loses most on obligations that must be paid before the end: postscripts, end phrases, minimum counts.

If the third prediction fails, then either dilution isn’t what limits a 135-million-parameter model on IFEval, or the retrofit doesn’t get the state through, and the next version of this post will say which. The code, tests, notebooks and commands are in research/ledger-smollm.

Rules nobody states

IFEval states its rules: end with a postscript, use no commas. Most of what a writer keeps is never said. A column numbers its points the same way every week, signs off the same way and keeps its reading list at the bottom, and nobody writes any of that down. This post’s claim is about obligations, not instructions, so it should hold when the obligation comes from examples. That is the toy’s task, and the second study moves it to SmolLM2.

The prompt shows two answers from one column, then a new question asking for some number of items. The column has a house pattern: one of eight list markers (numbered ones count up from one), leads in bold or not, perhaps everything in lowercase, perhaps a one-line summary first, a reading list last and a sign-off at the very end. Nothing in the prompt names a rule. Two examples always fix the pattern, and the tests check that they do for every kind of column the generator makes. The words are the frozen model’s own answers to plain questions, and only the form is imposed. The test columns use topics and sign-offs that never appear in training, and ask for 3, 6, 9 or 12 items. Twelve is longer than any answer the models trained on.

This study reaches the two shapes the IFEval mapping leaves out. “The summary comes first” is an order, and “nothing after the sign-off” is a last. Its length is the length of the answer, which is what the length exponent measures: $S(n)$ is the share of answers that keep everything when asked for $n$ items, and $\kappa$ per item is minus the slope of $\log S(n)$.

The Ledger keeps seven slots here. The marker and bold slots carry a class the clerk reads off the examples, as the toy’s clerk read the font, and the count takes its target from the new question through the pointer, not from the counts the examples show. Order and last get no slots of their own: the monitors check them, and the Ledger sees them only through the state of the summary, the reading list and the sign-off. With these slots the Ledger has 3.95 million parameters, and LoRA keeps rank 69.

The monitors keep a book as the answer is written. It’s easiest to see as a grid, one row per obligation and one step per piece of text, like a sequencer’s.

The rules at the top are what the clerk reads off the two examples, by counting, with nothing trained. When the runs exist, every answer they write will load into the same book, and the same monitors will read it.

Predictions for the house patterns

1. The length exponent per item is smaller for the Ledger than for LoRA: the paired bootstrap interval of the difference lies below zero. If this fails, the claim doesn’t reach rules learned from examples, and I’ll say so.
2. Invariants break mostly at the first item, where the pattern has to be read from the examples, and rarely after it, once the model can copy its own items. The Ledger’s gain on invariants comes mostly at item one.
3. Answers that end with a debt unpaid (no sign-off, no reading list, too few items) become more common with $n$ for the frozen model and LoRA, and less so for the Ledger. Without its gate the Ledger loses part of that advantage, more on the count and the sign-off than on the invariants.
4. At twelve items the Ledger lands on the count more often than LoRA, whose misses are mostly too few items or the examples’ count.
5. With 6,000 tokens of notes between the examples and the question, every model keeps less, and the Ledger loses less than LoRA.
6. The shared-softmax slots do no better than the Ledger.

List markers are a family of interchangeable variants, like the toy’s dollar signs, so gauge fixing applies here too: read the marker by counting, as the clerk above does, and allow only that one. I’ve left it out on purpose, so that the marker row measures the Ledger alone. The recipe for later doesn’t change. Declare the families of interchangeable variants the model writes (delimiters, quotation marks, list markers, spelling pairs that tokenise alike), read the frame by counting them in the examples, and restrict the output to one member per family. No training needed.

What is proved, what is measured, what is open

On this budget, the proofs have to do most of the work, so here is exactly where each claim stands.

Proved, with the toy agreeing: that softmax dilutes a fixed instruction as the text grows (Lemma 1; attention on the examples halves over a post), and that holding the instruction’s share needs an edge growing like $\log n$ (Corollary 1; slots in the shared softmax gave nothing). Under the hazard model, the shared softmax sends the length exponent to that of an unread obligation, and a separate read bounds it (Proposition 2; the font hazard rises and levels off, and the maths hazard orders 3.96%, 2.26%, 1.51%). Proposition 3 gives ending unpaid in closed form (the gate learned to cut the stop odds 8 to 18 times, a small effect here). The Ledger computes the base model’s function at insertion (equal to 12 decimal places in every placement). The obligations are equivariant under the style group (Proposition 4, checked on 4,000 random cases). Under gauge fixing an invariant breaks only if the frame is misread (Proposition 5; no break in 4,526 opportunities, nor in 4,505 with the examples dropped), counting reads the frame reliably (Proposition 6; exact on every evaluation prompt), and symmetric conventions have a length exponent of zero (Theorem 1). And slots in a model’s own softmax can’t be inserted neutrally and still learn (Proposition 7; my first control failed in exactly this way).

Assumed, and only tested: the hazard model itself, including independence between invariants (the product formula predicted the joint keep rate to within half a point).

Measured, not proved: that the Ledger and the frame make posts more coherent (90.0% against 80.5% with two seeds, $p = 5.5 \times 10^{-8}$; 95.0% with the frame, one seed), and where each part belongs in the network (a one-seed screen).

Open: all of it, for a real language model. Both studies on SmolLM2 are built and tested, and their results are pending.

The toy is tiny, its prose is noise, and its style has five dimensions where a real blog has hundreds. Its labels come free from the generator; a real system has to extract them, and extraction errors are a new way to fail. Its symmetry is exact and declared in advance, and real conventions are only roughly permutations of tokens. What should come next, in order:

1. The same toy with more seeds and longer training, plus the ablations the budget left out: the Ledger without its gate, the Ledger without monitor labels, and the frame without a Ledger.
2. The retrofit above, with every control; then larger models, style monitors extracted from examples, and Multi-IF, IFScale and a blog-imitation set like Wang and colleagues’.
3. Hazards measured against prompt length in pretrained models, where Proposition 2 predicts the same plateau for every prompt length, reached sooner for longer ones.
4. Contexts of 32,000 tokens and more, where dilution is strongest and dropping the examples saves most.
5. Symmetries that are learned instead of declared, in the spirit of LexSym, for conventions that are only nearly permutations.

The ledger of this notebook

The monitors are the easy half of all this, and they’re useful on their own. They’re the same kind of checker that taught the Ledger. Below are the conventions of this notebook as monitors, learned from the other posts. Each rule counts how many posts keep it, and it becomes owed when at least two thirds do. One rule is owed regardless. Only one of the three earlier posts ends with further reading, so the corpus alone wouldn’t have made it a rule. I made it one anyway, because an author can promote a habit to a rule, and that’s an obligation too. Paste a draft, from a model or not:

Why the class is NP

Checking whether a draft keeps a set of obligations takes one pass of a few small automata, in time proportional to the length of the draft. Writing a draft that keeps them all and is also worth reading means searching an enormous space, and nothing makes that easy. An answer is easy to check and hard to find, which is the shape of NP, the same shape as running Life backwards in Gliders are messages. The Ledger bets that the easy side can teach the hard one during training, one token at a time. The frame bets that, for some obligations, the hard side can be made to go away.

And the Model wrote the next entry: nine paragraphs, sources last. This time something kept count.

---

Further reading:

• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, “Attention Is All You Need,” NeurIPS (2017).
• Noam Chomsky, “Three Models for the Description of Language,” IRE Transactions on Information Theory 2 (1956); Syntactic Structures (1957); “On Certain Formal Properties of Grammars,” Information and Control 2 (1959); Aspects of the Theory of Syntax (1965).
• Noam Chomsky, Ian Roberts, Jeffrey Watumull, “The False Promise of ChatGPT,” The New York Times (8 March 2023).
• Machine Learning Street Talk: “#60, Geometric Deep Learning Blueprint (Special Edition)” (2021); “#78, Prof. Noam Chomsky (Special Edition)” (July 2022); “Prof. Subbarao Kambhampati: LLMs don’t reason, they memorize” (July 2024); “Transformers Need Glasses!, Federico Barbero” (March 2025).
• Richard P. Feynman, “Cargo Cult Science,” Engineering and Science 37 (1974).
• William A. Woods, “Transition Network Grammars for Natural Language Analysis,” Communications of the ACM 13 (1970).
• Amir Pnueli, “The Temporal Logic of Programs,” FOCS (1977).
• Giuseppe De Giacomo, Moshe Y. Vardi, “Linear Temporal Logic and Linear Dynamic Logic on Finite Traces,” IJCAI (2013).
• Andreas Bauer, Martin Leucker, Christian Schallhart, “Runtime Verification for LTL and TLTL,” ACM TOSEM 20 (2011).
• Grégoire Delétang et al., “Neural Networks and the Chomsky Hierarchy,” ICLR (2023).
• Jeffrey Zhou et al., “Instruction-Following Evaluation for Large Language Models” (IFEval), arXiv:2311.07911 (2023).
• Yun He et al., “Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following,” arXiv:2410.15553 (2024).
• Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville, “LLMs Get Lost In Multi-Turn Conversation,” ICLR (2026).
• Daniel Jaroslawicz et al., “How Many Instructions Can LLMs Follow at Once?” (IFScale), arXiv:2507.11538 (2025).
• Kenneth Li et al., “Measuring and Controlling Instruction (In)Stability in Language Model Dialogs,” COLM (2024).
• Nelson F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” TACL 12 (2024).
• Cheng-Ping Hsieh et al., “RULER: What’s the Real Context Size of Your Long-Context Language Models?,” COLM (2024).
• Ali Modarressi et al., “NoLiMa: Long-Context Evaluation Beyond Literal Matching,” ICML (2025).
• Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen, “Enabling Large Language Models to Generate Text with Citations” (ALCE), EMNLP (2023).
• Zhengxiang Wang, Nafis Irtiza Tripto, Solha Park, Zhenzhen Li, Jiawei Zhou, “Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors,” Findings of EMNLP (2025).
• Mosh Levy, Alon Jacoby, Yoav Goldberg, “Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models,” ACL (2024).
• Petar Veličković, Christos Perivolaropoulos, Federico Barbero, Razvan Pascanu, “Softmax Is Not Enough (for Sharp Size Generalisation),” ICML (2025).
• Ken M. Nakanishi, “Scalable-Softmax Is Superior for Attention,” arXiv:2501.19399 (2025).
• Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, “Efficient Streaming Language Models with Attention Sinks,” ICLR (2024).
• Federico Barbero et al., “Transformers Need Glasses! Information Over-Squashing in Language Tasks,” NeurIPS (2024).
• Nelson Elhage et al., “A Mathematical Framework for Transformer Circuits,” Transformer Circuits Thread (2021).
• William Peebles, Saining Xie, “Scalable Diffusion Models with Transformers,” ICCV (2023).
• Jianlin Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing 568 (2024).
• Andy Yang, David Chiang, Dana Angluin, “Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages,” NeurIPS (2024).
• Andy Yang, David Chiang, “Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers,” arXiv:2404.04393 (2024).
• Felix Stahlberg, Bill Byrne, “On NMT Search Errors and Model Errors: Cat Got Your Tongue?,” EMNLP (2019).
• Benjamin Newman, John Hewitt, Percy Liang, Christopher D. Manning, “The EOS Decision and Length Extrapolation,” BlackboxNLP (2020).
• Jonathan Gordon, David Lopez-Paz, Marco Baroni, Diane Bouchacourt, “Permutation Equivariant Models for Compositional Generalization in Language,” ICLR (2020).
• Ekin Akyürek, Jacob Andreas, “LexSym: Compositionality as Lexical Symmetry,” ACL (2023).
• Nouha Dziri et al., “Faith and Fate: Limits of Transformers on Compositionality,” NeurIPS (2023).
• Aman Madaan et al., “Self-Refine: Iterative Refinement with Self-Feedback,” NeurIPS (2023).
• Subbarao Kambhampati et al., “Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks,” ICML (2024).
• Saunders Mac Lane, “Natural Associativity and Commutativity,” Rice University Studies 49 (1963); Categories for the Working Mathematician, 2nd edition, Springer (1998).
• Michael M. Bronstein, Joan Bruna, Taco Cohen, Petar Veličković, “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges,” arXiv:2104.13478 (2021).
• Taco Cohen, Max Welling, “Group Equivariant Convolutional Networks,” ICML (2016).
• Sékou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, Siamak Ravanbakhsh, “Equivariance with Learned Canonicalization Functions,” ICML (2023).
• Jesse Mu, Xiang Lisa Li, Noah Goodman, “Learning to Compress Prompts with Gist Tokens,” NeurIPS (2023).
• Wassily Hoeffding, “Probability Inequalities for Sums of Bounded Random Variables,” Journal of the American Statistical Association 58 (1963).
• Chloé Kiddon, Luke Zettlemoyer, Yejin Choi, “Globally Coherent Text Generation with Neural Checklist Models,” EMNLP (2016).
• Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, Hang Li, “Modeling Coverage for Neural Machine Translation,” ACL (2016).
• Sainbayar Sukhbaatar et al., “Augmenting Self-attention with Persistent Memory,” arXiv:1907.01470 (2019).
• Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev, “Recurrent Memory Transformer,” NeurIPS (2022).
• DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur, “Block-Recurrent Transformers,” NeurIPS (2022).
• Qingru Zhang et al., “Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs” (PASTA), ICLR (2024).
• Chih-Yao Ma et al., “Self-Monitoring Navigation Agent via Auxiliary Progress Estimation,” ICLR (2019).
• Jean-Baptiste Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS (2022).
• Vijay Viswanathan et al., “Checklists Are Better Than Reward Models For Aligning Language Models,” NeurIPS (2025).
• Nicolas Carion et al., “End-to-End Object Detection with Transformers” (DETR), ECCV (2020).
• Loubna Ben Allal et al., “SmolLM2: When Smol Goes Big, Data-Centric Training of a Small Language Model,” arXiv:2502.02737 (2025).
• Edward J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR (2022).
• Xiang Lisa Li, Percy Liang, “Prefix-Tuning: Optimizing Continuous Prompts for Generation,” ACL (2021).
• Code, data and trained weights for the experiment: research/coherence/ in this site’s repository.
• Code, tests and Colab and Kaggle notebooks for the retrofit to SmolLM2, both studies: research/ledger-smollm/ in the same repository.