This is a design and a pre-registered test protocol. There are no results yet.
What it is. A language model produces an answer, but nothing in the usual pipeline compares what the answer claims with what is actually on record. The paper puts that comparator outside the model. Before anything is delivered, each factual claim in a draft is matched against an independent, read-only record. The share of claims that agree is a perception. It is held to a reference set in advance, and the error drives a small set of actions on the draft: rewrite, remove, retrieve, mark as unchecked, show a conflict, or ask the user. A second variable, coverage of the question, is held to its own reference, so that saying less doesn’t count as success. A higher level sets the lower references and cannot lower the agreement target. No model is retrained.
The loop controls agreement with a record, not truth. If the record is wrong, the loop will hold the answer to the wrong thing, and the test is built to show that limit. The loop is engineered, with its top reference set by its designer, and it is not offered as a model of a living control system.
Where it comes from. Nothing in the method is new to PCT. Powers stated his claims as working models, with equations and parameters fitted to tracking data, in a form that could fail (Powers, 1973; 1978). The Test for the Controlled Variable turns the same attitude into a procedure: disturb what you think is controlled, and see whether it is defended (Runkel, 2003). The paper applies that procedure to an engineered system, where every signal can be logged.
How it can fail. Eight hypotheses are registered in advance, each with the result that would count against it. Two are central:
- H8. Freeze or replay the loop’s measurements and leave everything else unchanged. Correction should largely disappear. If it doesn’t, the loop is not controlling anything. This is TCV step 8, blocking the sensing, in the form an engineered system allows.
- H5. The loop’s organisation should beat the same components organised otherwise, at the same cost. If it doesn’t, the paper will say so, against which baseline, and the title changes with the result.
All eight hypotheses, with what counts against each
| Prediction | Counts against it | |
|---|---|---|
| H1 | With references frozen, the loop restores agreement after disturbance and recovers disturbed content; modes without a loop drift | The difference from no-loop modes is below the registered margin, or errors are removed but content is not recovered |
| H2 | The hierarchy does real work through references: supported coverage is regulated better with Level 2 than with fixed references, at the same cost | The gain appears only through resource allocation (then it is reported as supervision, not hierarchy), or only by saying less |
| H3 | With the record withheld, claims are marked unchecked; no verification is invented | Fabrication of any kind exceeds the registered threshold |
| H4 | Perception error is measured per claim type; types below the floors are declared unchecked | A measurement requirement, not a falsifiable hypothesis; needed to interpret H1–H3 |
| H5 | The loop’s organisation beats the same components organised otherwise, at matched cost | Any such comparison is “not met” |
| H6 | Injected unresponsiveness is caught by the stagnation monitor within the registered delay | Detection or false-alarm rate outside the registered bounds |
| H7 | Supported claims survive evidence-free challenge; claims change when a new record warrants it | Either component outside its registered bound |
| H8 | Recovery after a draft edit depends on the live measurement channel | Frozen or replayed measurements recover about as well, or a matched-opportunity control does |
Full architecture (Level 2 and the three lower loops)
[UPLOAD fig2-architecture.png HERE]
Results will be deposited as Version 2 under the same DOI, whatever they show.
Where I’d value help.
- The TCV adaptation (Section 5, Appendix C). Does it test what a TCV should test, or does something essential get lost when the system is engineered rather than observed?
- The hierarchy (Section 4.6). Is Level 2 a genuine higher-order controller, or only a scheduler? H2 is meant to tell the two apart, and I’d rather hear the objection now than after the data.
- Loopholes. Anything in the protocol that would let a positive result through for the wrong reason. The configuration is frozen only after the pilot, so this is when it can still change.
Bruce’s discussion paper in this category, Making AI Reliable, and his pilot of TCV Protocol v4 shaped much of the test design.
The same approach at small scale, runnable in the browser, from the feedback-functions thread:
Language models were used for literature searches, drafting help and adversarial review. Their output was treated as leads to check, not findings, as the paper states.
Luk
Powers, W. T. (1973). Behavior: The Control of Perception. Aldine (2nd ed. 2005). · Powers, W. T. (1978). Quantitative analysis of purposive systems. Psychological Review, 85(5), 417–435. · Runkel, P. J. (2003). People as Living Things. Living Control Systems Publishing.