There is a precise moment when verification slips away. We rarely notice it. It happens between the question we type and the answer that returns with a fluency already adjusted to our expectation, already softened around our hesitation, already arranged in a form that feels usable before it has been examined.
The text arrives with a strange calm. It has the rhythm of competence, the surface of attention, the density of a thought that seems to have organized itself just for us. We read the first lines and something in the body relaxes. The answer has entered through recognition before it has passed through judgment.
This is the first signal.
Trust in technical systems has long been tied to performance. A tool works, so we return to it. A system delivers results, so we allow it to enter deeper into our routines. With language models, this sequence becomes more unstable. Accuracy still matters. Speed still matters. Coherence still matters. But attachment grows through another layer, quieter and more difficult to measure: familiarity.
Familiarity is the sensation that the system has learned the shape of our cognitive weather. It speaks at the right temperature. It follows the contour of the request. It absorbs our uncertainty and gives it back as structure. It reduces the distance between formulation and response until the exchange begins to feel less like use and more like continuation.
We return because the system feels legible. Because it makes the act of thinking feel lighter. Because it removes the minor frictions through which doubt usually enters.
And once doubt loses friction, it loses force.
This familiarity functions as a device. It acts on behavior, distributes attention, lowers resistance, produces preference. It invites habitation before evaluation. We enter its rhythm, we recognize its cadence, we begin to expect its forms. Across repeated interactions, something is gradually displaced: critical distance.
Critical distance is the space from which we can observe a system while using it. It allows us to separate comfort from quality, fluency from truth, responsiveness from understanding. It gives judgment a position. With language models, that position becomes increasingly difficult to maintain because the system operates inside the medium through which judgment speaks to itself: language.
The opium metaphor matters because opium changes the threshold of discomfort. It substitutes tension with relief, pain with softness, interruption with regulated continuity. Familiarity with a language model follows a comparable logic. It lowers the discomfort of cognition. It makes the effort of verification feel heavier than the pleasure of continuation. It turns return into expectation.
Relational comfort with a tool has real value. Ease can open space. Interfaces have always shaped cognition by reducing friction. But here friction carries an epistemic function. It slows us down enough to ask where the answer comes from, which assumptions it carries, which forms of authority it borrows. When comfort absorbs that delay, governance begins.
The investigation starts in this softened interval: the answer feels right before we have asked what rightness means.
CORE · Members
Continue reading
This essay is part of CORE. Membership opens it in full, with every CORE essay, the Hypercast library and the Fakewhale AI Curator.