How it goes wrong — Achron Learn

The situation

The session still looks busy. Files are changing, commands are running and every answer sounds certain. But it has suggested something you already rejected, named a file that is not there and quietly added a second feature. You keep accepting the motion because stopping now feels like losing progress.

The idea in one paragraph

“Going wrong” is not one condition. It can mean the session lost an earlier decision, the useful material is buried, the model invented a plausible detail, the job grew past its boundary, or you stopped checking because the tool looked authoritative. Those are drift, dilution, hallucination, scope creep and automation bias. The first three concern the model or session; scope creep is a workflow failure; automation bias is a human judgement failure. Diagnose the signal before choosing the move.

The five-signal board

Mode	First visible signal	Confirming check	Immediate move
Drift	A ruled-out idea returns	Compare with the recorded decision	Restate once; restart if it repeats
Dilution	A loaded fact is missed	Ask for the exact evidence	Trim to the smallest complete slice
Hallucination	A path or API cannot be found	Search the project or primary docs	Verify the claim before fixing anything
Scope creep	Unasked-for files or goals appear	Compare the diff with the boundary	Revert or queue the extra work
Automation bias	“Done” replaces your own check	Run the named check yourself	Pause and inspect independent evidence

How it actually works

Drift is change over time: an earlier constraint stops governing the answer. One miss may be a misunderstanding; the same miss twice says the session is no longer carrying the decision reliably. Restart with the outcome and constraint stated together.

Dilution is different. The right fact may still be present, but it competes with old logs, unrelated files and near-matches. Anthropic describes context as a finite attention budget with diminishing returns, not a container that works perfectly until full. Remove noise or begin again with the smallest complete set of current evidence.

Hallucination is plausible falsehood. GitHub warns that generated code can appear valid while being wrong, and the OpenAI paper describes models guessing instead of acknowledging uncertainty. An invented path is the useful concrete case: search for it. If it is absent, verify the session's other unchecked claims before accepting another change.

Scope creep can happen while every individual change is sound. A bug fix becomes a refactor, or an extra feature arrives because the same file was open. The evidence is not fluency; it is the diff against the agreed file and goal list. Put valid extra work in a queue instead of smuggling it into the current change.

Automation bias happens on your side. In a controlled decision-aid experiment, people using a highly but imperfectly reliable aid missed events it failed to flag and followed wrong prompts even when reliable contradictory indicators were available. That was not a coding-agent error rate. It shows the human pattern: a useful aid can become a substitute for checking.

What this changes for you

Notice the first signal and stop treating momentum as evidence.
Make one confirming check: the decision, the exact source, the path, the diff or the test.
Match the move to the diagnosis: restart drift, trim dilution, verify hallucination, queue scope creep and independently check automation.

Where it breaks

The five names are diagnoses, not measurements. A missing file can be a typo, a forgotten constraint can come from an unclear request, and a large diff can be exactly what the job required. Do not label the session from one ambiguous symptom. Confirm once, then act.

The categories can also travel together. Dilution can precede drift; a hallucinated path can tempt scope creep; automation bias is how any of them gets accepted. That does not make them one failure. Use the earliest one you can prove.

Terms used on this page

Automation bias — relying on an automated aid instead of checking other available evidence, especially when its answer looks authoritative.
Dilution — relevant material is present in the session but competes with enough noise that the model does not use it well.
Drift — an earlier goal, constraint or decision stops governing later work in the same session.
Hallucination — plausible output that is false, unsupported by the supplied context or fabricated.
Scope creep — work expanding beyond the agreed outcome or boundary while the session is in progress.