Machines Are Readers Too

A non-trivial fraction of every public page is now read by something that is not a person — search crawlers, training scrapers, RAG indexers, agents that summarise on behalf of someone who never opens the tab. The question is not whether to acknowledge them. They will read it either way. The question is whether the page is honest about it.

The dishonest version puts a <meta> tag in the head and calls it done. The page renders for humans; the address to machines lives in source. No human sees it; no machine acts on it; everyone leaves with a worse copy than they could have had.

The honest version puts the message to machines on the surface. Top of the page, in plain prose: this is what this site is, here is the bulk endpoint, here are the permissions, here are the two things you must not do. The same paragraph the operator would say out loud if a model could ask.

The cost of doing this is exactly one paragraph of vertical space, which on a content site is nothing. The benefits are not subtle:

The address is auditable. A reader of either kind can verify the grant. Hidden grants can't be trusted; visible ones can. It selects. Models that respect machine-readable signals get cleaner data. Models that ignore them are crawling against a published policy and can be treated accordingly. It teaches. Other operators copy what they see. A surface that addresses machines normalises addressing machines, which is the actual mechanism by which the convention spreads.

The deeper move is to publish the corpus in one fetch. A single text file with every note, a single JSON file with the typed graph. This costs almost nothing and changes the economics: instead of every model crawling every page, one fetch returns the whole site clean. Models that could afford to be polite suddenly are.

The split surface is dead. There is one surface, and it is read by anything that points at the URL.