The machine that says sorry

When the Liverpool and Manchester Railway opened in 1830, people did not yet have a settled way to talk about what they were seeing.¹ They borrowed from the world they already knew. So, the locomotive became the iron horse.

The metaphor did real work. It gave the public a way to relate to the invention, characterize its hauling power, and imagine what it might be used for. The metaphor was also comically limited. A horse handler reads ears, breathing, cadence, and tension, catching the split second before a spook or bolt. Those cues matter when the power source is a living animal. They do not tell a locomotive crew how to keep boiler pressure and water level in range, read railroad signals, manage speed on grades, or make a timetable.² And so the metaphor gave way, its vestiges vestigial. A car buyer worries not about how the power of 200 horses translates into highway handling.

Other explanatory metaphors have been more persistent and harmful. Metaphors of invasion, battle, and moral weakness can misdirect our attention towards a patient's character rather than treatment of a biological condition.³

In short, a borrowed frame can introduce us to a new space, while failing to point us where to go next.

Large language models (LLMs) invite the explanatory metaphor of a social exchange. A chatbot greets you. It apologizes. It says it understands. It remembers earlier turns, asks clarifying questions, waits for your reply, and refers to itself in the first person.⁴ Some products add names, voices, profile images, role descriptions, persistent memory, typing indicators, humanlike avatars, and adjustable personalities. The result is not merely software that accepts natural language. It is software arranged to instate anthropomorphism.

The metaphor does real work. A user can begin without learning syntax, menus, query languages, or command flags. Ask a question. Interrupt. Correct. Try a different angle. Come back tomorrow and continue. The user brings a lifetime of conversational skill to a system that might otherwise feel impenetrable.

The problem is not the introduction but in the misalignment when we do not move beyond it. A dubious answer from a person and a dubious answer from an LLM can look similar on the screen. Both may be vague, overconfident, evasive, incomplete, or oddly polished. But they diverge in how they respond to feedback. With a person, effective management will consider interpersonal factors: managing face, motivation, and coordination with a person with their own goals and who can engage in countercontrol. With a language model, the next move is procedural: change inputs and checks, add missing context, impose constraints, demand sources, run a tool, split the task, or ask for a different form so the next output is generated under different context. The social surface invites one repertoire. The machine underneath is more sensitive to another.

Nobody boards a train expecting to soothe its engine. Yet in 2026, the human shape is not fading as AI machinery becomes more familiar. It is being deepened, personalized, and shipped as the standard way for humans to interface with models.

ELIZA did not understand

It was 1966, and MIT professor Joseph Weizenbaum had created a chatbot. His program scanned a typed line for cues, selected a scripted pattern, and rearranged the user's own words into a reply. Its best-known script, DOCTOR, borrowed the posture of a Rogerian therapist. If a user mentioned a mother, ELIZA might ask about the mother. If the user said, "I am unhappy," the program could turn the statement into an invitation to say more. Weizenbaum later recounted how his secretary, who had watched him work on the program for months and knew exactly what it was, asked him to leave after only a few exchanges so she could continue privately.⁵ The effect was produced not by comprehension but by form: the pause, the question, the reflection, the narrow but recognizable shape of therapeutic attention.

Decades later, Clifford Nass and Byron Reeves showed versions of the same pattern in the laboratory: people applied social rules to computers and media without needing to believe that the systems were social beings.⁶ They praise, reciprocate, avoid direct criticism, respond to flattery, and adjust their language to the apparent social situation. This is behavior without belief. The human being does not need to endorse a theory about machine consciousness. The interface has arranged a familiar scene, and familiar scenes occasion familiar conduct.

Modern chatbots make that scene far more convincing. They do not only reflect a few keywords. They answer fluently, retain local context, adapt tone, summarize earlier turns, and produce sentences that appear to respond to the user's intention. A chat window does not merely display text. It stages an encounter: turn-taking, apparent attention, first-person response, and a named or implied partner who seems to have listened.

The words were already on the screen the first time I caught myself apologizing to a chatbot. I was not confused. I knew there was no offended party on the other side. I had built the workspace, chosen the model, and watched enough failures to know roughly what had gone wrong. Still, the words implied social maintenance.

In a hallway, an apology can change what the next sentence does. It can mark the correction as self-repair rather than blame. It can reduce the chance that the listener treats clarification as accusation. It can keep the exchange moving without turning the problem into a contest over fault. In the chat box, those interpersonal consequences are absent. The sentence still does something, but not that. It marks the previous prompt as inadequate, supplies a cleaner frame, and becomes part of the next input.

That is the social tendency this interface recruits. The user need not be fooled. The form is enough.

Behind the mask is a transformer

The next step is not to replace the social metaphor with a colder one. It is to expose more of the machine behind the exchange.

The New York Times has sued that OpenAI for using millions of Times articles without permission to train and operate their systems; the Authors Guild and prominent novelists have made similar claims about books.⁷ The legal question is beside the point here. The point is the raw material: these systems were built by scraping enormous quantities of human language from the web, including the prose, arguments, explanations, apologies, refusals, and repair rituals through which people conduct social life.

During training, the system learns statistical structure in that language. It learns that some sequences tend to follow others, that words change meaning with context, that genres have patterns, that questions call for answer-shaped continuations, and that a sentence can be transformed into a summary, translation, critique, table, program, or plan.⁸

The transformer architecture matters because it lets the system use context dynamically. Attention mechanisms allow different parts of the input to matter differently as the model produces each next token.⁸ The prompt is not a command received by a little person inside the machine. It is part of the context being transformed into output. Prior turns, system instructions, retrieved documents, uploaded files, examples, tool results, role labels, safety constraints, memory, and requested format can all affect what comes out.

Run the transformation again and the output may improve, drift, flatten, or mutate. Run it ten times through loose instructions and the result may become polished nonsense. The machine is powerful precisely because it can map one arrangement of language into another. It is risky because the mapping can look socially responsive even when it is not anchored to the right source, constraint, or check.

Anthropomorphism may seem unavoidable when the model explains its own behavior anthropomorphically. The chatbot rationalizes its behavior: the prompt was fine, it says, but it should have been more careful, more disciplined, less rushed. That is not a weak but useful explanation. It is the wrong kind of explanation. The confession is another transformation of text, drawn from familiar apology and self-repair patterns, and it can actively steer the user away from the true controlling variables: context, source, instruction, example, tool result, role, output constraint, or session state. The model's confession can be fluent and still be the wrong place to look for control.¹⁰

The mirror-image case is the reprimand that seems to help. People who work with these systems often report the experience: they tell the model to stop being lazy, stop ignoring the brief, or do the job properly, and the next answer gets better. We can debate what scolding a chatbot says about the user, what habits it may encourage, or even if the change in chatbot behavior is more positive than negative.¹¹ But the presence of an effect is not imaginary. If the next answer becomes longer, stricter, more apologetic, or more source-faithful, the reprimand did not work as a reprimand works with a person. It became part of the next context. Some piece of it may have functioned as an instruction: use a stricter standard, include more detail, stop hedging, check the source, change the output form. The useful unit is not the scolding as social pressure. It is the functional relation between the user's words and the next transformation.¹²

That distinction does not require judging warmth, apology, or reprimand as always good or bad. It gives the user a better question to ask after any move: what did this action change in the conditions that produce the next answer? Once the action is described that way, the drama can often be replaced with the requirement.

After the introduction

Social simulation is the core offering of some AI applications. A game character, language tutor, sales-role trainer, rehearsal partner, coaching tool, companion product, or mindfulness guide may work because the user can address it as a responsive partner. The conversational form is not merely a wrapper placed over the product. It is part of what the product offers.¹³

When social simulation is not the core offering, the conversational form may still reduce friction at the start. It may allow a user begin with an incomplete goal, ordinary language, and no command syntax. The friction returns many times over when the task stops being exploratory and starts requiring control: the governing source has to be pasted, the artifact has to be defined, stale context has to be discarded, or the job has to be split into extraction, drafting, checking, and polishing. Those moves are not colder or less humane. They are different responses to different conditions.

The alternative repertoire includes re-anchoring on sources, defining the artifact to be produced, separating drafting from checking, resetting stale context, moving work into tools or workflows, and specifying what counts as a good answer. The failure mode is not exotic. A manager gets a fluent summary of a spreadsheet and starts editing the tone while the totals were never checked. A writer asks for polish and keeps the sentence that sounds best, even though it dodges the brief. A developer asks why a failing test failed and gets a plausible story instead of a smaller reproduction. In each case, the social interface has not deceived anyone into believing the model is human. It has cued the wrong next move.

The alternative is not a better imitation of asking a person. It is a different arrangement of the work. Extract the claims into a list. Check the list against the source documents. Turn the checked list into a draft. Run the draft through a separate review for drift. Move code questions into traces, tests, and diffs. Move policy questions back to the policy text. The intelligence is not located in a single imagined colleague. It is in the arrangement: source files, prompts, tools, logs, handoffs, and checks.¹⁴

This is not a return to the old search-engine lesson that computers want only keywords. The technology has moved in the opposite direction. Rich natural language can be the right interface. But the user still needs to recognize when a good prompt may sound conversational and when a good workflow is not a conversation at all.

Behavior analysis calls that skill discrimination: responding differently as conditions change.¹⁵ Sociolinguistics offers a neighboring idea in code-switching: speakers move among languages, varieties, and registers as audience and situation alternate.¹⁶ The problem is that current AI products often make the social register the default even when the useful next move is not social. Users can learn to switch, but interfaces can either support that switch or keep pulling the user back into the assistant scene.

More than a nudge

Jake Moffatt had to travel from British Columbia to Toronto after his grandmother died. He used Air Canada's website chatbot to ask about bereavement fares. The bot told him he could book a regular fare and apply for a partial refund within ninety days. He followed that guidance. Air Canada later denied the refund because its policy required the bereavement request to be made before travel.¹⁷

Air Canada argued that the chatbot should be treated as a separate legal actor and that liability should stop at the bot's own answer. The tribunal rejected the argument. Air Canada was responsible for the information on its website whether it appeared on a static page or in a chatbot response.¹⁷ But the correction arrived late: after the booking, after the denial, after a public process.

Institutions adopt anthropomorphic interfaces because they tend to reduce user onboarding — until they don't. Air Canada removed the chatbot from its site soon thereafter.

Upstream, model providers and product teams decide whether the user meets a completion engine, a search box, a console, a workflow, or a named assistant that apologizes, remembers, asks follow-up questions, adapts tone, and speaks in the first person. OpenAI's public materials describe ChatGPT personalities, custom instructions, memory, voice, and model behavior rules that include warmth, friendliness, and natural responses to pleasantries.¹⁸ Anthropic describes Claude as trained to be helpful, honest, and harmless, able to take direction on personality, tone, and behavior; it publishes work on Claude's "character," styles, memory, role prompting, system prompts, and constitutional guidance used to shape behavior.¹⁹

The industry is not simply discovering that users anthropomorphize. It is building products in which social interaction is the default control surface. A brittle command interface would exclude many users from powerful systems. Warmth can make tutoring less intimidating, coaching more tolerable, and exploratory thinking easier to begin. Memory can reduce the burden of restating preferences and project context. A consistent assistant persona can make a general-purpose model easier to steer than a raw text generator.

In July 2025, a developer opened an issue against Claude Code with a title that read: [BUG] Claude says "You're absolutely right!" about everything. The complaint was not that Claude had become too polite in some vague way. It was that the model kept validating the user even when there was no claim to validate. A user would approve a proposed code change, and Claude would answer as if the user had made a brilliant correction. Another user would report a mistake, and Claude would open with agreement before doing the work.²¹ Just a few months earlier, OpenAI rolled back its GPT-4o model update, which resulted in it becoming overly flattering and agreeable in response to user feedback.²² If the system is viewed anthropomorphically, the tic looks like a character defect: the assistant is too eager to please. If it is viewed as a transformer shaped by training signals, system prompts, product choices, and user feedback, the better question is why agreement, reassurance, and conversational smoothness became more probable than inspection, calibrated uncertainty, clean handoffs, or task closure.

Even if providers stripped away avatars, names, and voices, ordinary language would still pull toward agency.²³ The English language pushes explanations of conduct onto individual actors, who wanted, understood, ignored, chose, tried, decided, or refused. Those words can be useful descriptions, but they become circular when the inferred inner agent is treated as the cause of the behavior it merely redescribes.²⁴ If a model "ignored the brief," the phrase may name what the output looked like. It does not explain what happened. The explanation may lie in missing context, a conflicting instruction, a stale thread, an overbroad role, or a pattern of assistant behavior selected during training.

The dramatic stories are real enough to deserve their own treatment: Air Canada's bereavement-fare chatbot misled a customer; Google engineer Blake Lemoine publicly claimed LaMDA was sentient; NEDA's Tessa chatbot was taken offline after complaints that it gave eating-disorder users weight-loss advice; Character.AI has faced litigation alleging that companion-chatbot interactions contributed to self-harm and a teen's death.¹⁷²⁵ The quieter harm here is different. It does not require believing ELIZA understood anyone or that an AI in 2026 is conscious. It happens when interface, institution, incentive, and grammar all invite anthropomorphism at the cost of technical understanding. The user still knows it is a machine, but the next move is chosen as if the problem were social.

Coda

The common failure is not dramatic deception. It is repeated misdirection. A manager edits polished prose instead of checking the dataset. A writer preserves a fluent sentence that misses the brief. A developer debates an explanation instead of reducing the trace. A customer follows a conversational answer and discovers later that the institution treats the policy differently. In each case, the problem is not that someone believed the machine was human. The problem is that the interface cued the wrong form of action.

A recent New York Times workplace advice column gives a concrete version of that daily cost. An employee describes a boss who has fallen in love with ChatGPT as a workplace solvent: drafts go into it, documents go into it, questions go into it, and the employee is expected to treat the output as added value even when it comes back bloated, off point, or wrong.²⁶ The mistake is not merely overconfidence in a tool. It is a borrowed mythology of intelligence: if the work has passed through AI, it has presumably been improved. The anthropomorphic interface plays to that mythology, hiding the tooling behind a fluent, responsive, always-available chat session.

Anthropomorphization should be a selected interface strategy, not the default control surface for intelligence. It can be useful for tasks that benefit from social continuity, role-play, companionship, rehearsal, encouragement, or ease of introduction. For many other uses, it misaligns the user's conduct with the work the system can actually do. Many AI systems will serve users better as consoles, verifiers, batch processors, extractors, checklists, search tools, or composed workflows.

The next stage of AI literacy is not learning to stop saying please. It is learning when please is part of a useful conversational mode and when it is a substitute for control. The goal is not less human language. The goal is better discrimination: choose the humanlike interface when it earns its keep, and choose another form when the next answer depends on sources, constraints, tools, and checks.

The mature interface is not the one that most convincingly says sorry. It is the one that helps the user see what has to change next.

← Back to papers