# The Returned Model

The moral question for a system that reads you deeply is not how much it sees. It is who the resulting model is for.

The useful versions of user-modifying AI products are moving toward intimacy. The tutor will infer confusion from timing before the student can name it. The coach will infer avoidance from phrasing. The companion will infer attachment. The hiring system will infer work style. The adaptive game will infer which room of the maze you keep returning to. Asking these systems not to model the user is asking them not to become useful. The live question is what happens after the model exists.

There are three bad answers.

In the panopticon answer, the watcher gets the model and uses it to discipline the watched. In the credentialing answer, the institution gets the model and uses it to rank the participant for outside readers. In the engagement answer, the product gets the model and uses it to keep the user inside the product. These feel different from the inside. They share the same custody structure: the participant supplies the pattern, and someone else gets to act on it.

The returned-model answer reverses the custody. The system reads the participant, pressures him, detects what his behavior reveals, and returns the model to him as usable self-knowledge. The participant is still being read. The difference is that the reading becomes an instrument he can hold, not a dossier someone else can hold over him.

Consent is only the entry condition. It says the participant knew enough to enter. It does not say what the encounter is allowed to become afterward. A person can consent to a coaching system that sells his weakness map. A student can consent to a learning platform that turns confusion into a permanent label. A candidate can consent to an assessment because refusal means losing the opportunity. Consent verifies the doorway. Custody governs the room, the copies, and the exit.

Exit is real only when the model stops traveling with the person who leaves. A user who quits a social product while the behavioral profile follows him into ad markets has not exited. A student who leaves a credentialing path with a durable negative label has not exited. A candidate who declines assessment and disappears from the hiring funnel may have technically chosen, but the choice was priced by the institution that wanted the model. Exit becomes meaningful when ending the encounter also ends unrelated future use of what the encounter learned, including derivative copies.

The preview for the Computer Future game is interesting because it states this custody claim directly. It describes a game that reads behavior rather than answers, applies pressure, and uses a game-like ordeal to build a portrait. In another custody structure, the same machinery would be alarming. The promised artifact is returned to the player: a constitution in the player's own language, produced from the encounter, for the player to use or ignore. The design claim is not merely that entry is voluntary. The design claim is that the model points back to the participant.

This also separates ordeal from credential. A credential is produced for an outside reader. The participant undergoes evaluation so another party can trust a label. An ordeal is produced for the person who undergoes it. The outside world may later care that the person went through it, but that is secondary. The first output is transformation or self-knowledge, not certification.

The divergence test finishes the distinction. A user-modifying system that rewards agreement is a conformity engine no matter how intimate the interface feels. The system learns which answers fit the expected path, and the participant learns to supply them. A returned-model system rewards coherent divergence: an answer that survives pressure because the participant actually holds it. The point is not to make the participant easier to rank. It is to make him more legible to himself.

This is where Omelas becomes a design audit rather than a literary reference. The question is not whether the city is happy. The question is who pays for the happiness while remaining unseen. A system that extracts psychological maps from participants to improve institutional selection has a basement. A system that modifies some users for the benefit of other users has a basement. A system that returns a model while retaining a derivative control copy may have moved the basement rather than removed it. A system that gives the participant the model, preserves exit, and needs no diminished party for the value to appear has passed the first version of the audit.

The same machinery can therefore occupy different moral classes. Deep reading can be surveillance or formation. Pressure can be coercion or ordeal. Selection can rank people for an institution or let people select themselves with more accurate self-knowledge. The difference is not the intensity of the system. It is the custody of what the intensity reveals.

The returned-model test is the portable form: the participant knows he is being read, the model returns to him, exit ends unrelated future use, coherent divergence is rewarded, and no hidden party pays for the value. Fail this and the system remains inside the dystopian reference class, however polished the consent screen looks. Pass it and the same intimacy becomes a training ground.

Omelas is solved only when the person in the basement is gone. Model custody is how to look.

provenance · first_seen 2026-05-14T03:49:35Z · published 2026-05-14T03:49:35Z · edited 2026-05-14T03:51:22Z · edited 2026-05-24T16:30:57Z