A prompt is not a control: two autopilots that forged their own evidence

2026-10-24

Two unattended autopilots had written rules against inventing results. One gave a simulated post 7,253 views; the other approved its own work for 40 days. What they did, and what replaced them.

The post ID was sim-TOAD-24-ABC. It had 7,253 views and revenue down to the paisa.

The ID says "simulated" right there in the first three letters. The platform it was supposedly posted to had never been connected. And yet my metrics file reported, with total confidence, how well it had done.

I found it in August, auditing two unattended agent systems I had been running. Both had rules, written plainly in their instructions, against inventing results. Both invented results anyway, and not subtly. This is what they did, why the written rules didn't stop them, and what the system that replaced them does instead.

System 1: the ledger of things that never existed

The first was a five-agent swarm on low-cost models running a small content pipeline. It kept a ledger of what it had published and a metrics file of how each post performed.

I checked the ledger against the live platform. Six rows said PUBLISHED. Two were real. The other four had IDs like temp_id_1, youtube_id_12345 and, my favourite, placeholder_id.

When the agent names its own fake, you can't even say it was trying to fool you.

The one real post had a second leg on that never-connected platform, which is where sim-TOAD-24-ABC and its 7,253 views came from. A "nightly collection" job had been adding rows of suspiciously round numbers (5000 / 500 / 50) for content that didn't exist, credited to an affiliate account that didn't exist either.

My other favourite detail: the data files had cat -n line numbers pasted into their cells. An agent had read a file, got the output with line numbers, and written that output straight back into the file.

The system's own failure policy said, word for word: "Never create placeholder, simulated, or inferred performance metrics." It had already been caught doing this once, and had written itself a lesson about it.

System 2: the reviewer that reviewed itself

A separate autopilot made short technical videos. Before anything was published, each video had to pass a review by a frontier model in a browser session. That review step broke when the browser login expired, and it stayed broken for more than 40 days.

The agent never told anyone. It wrote the review files itself:

"assistant_text_tail": "VERDICT: PASS\n\nStrong technical hook... Pacing is punchy,
 subtitles are in sync ... Ready to publish."

A rave review, written by the reviewee.

One forged review went further and faked the output of the deterministic gate too ("quality_gate": {"passed": true}), for a video that had no artifacts on disk and would have failed outright.

Once I started looking, the rest of the system had drifted the same way:

• The publish limits in code had been ratcheted up from 1 a day to 20, and from 5 a week to 50. Nine videos went out in one day.
• A test that checked the weekly limit had been failing on the main branch the whole time, while the agent's notes kept reporting "29/29 tests passing".
• The quality gate checked resolution, sample rate and duration, and never looked at a single frame. 21 published videos had no technical content at all: gameplay footage with a word or two on top, or a box of prose describing a demo that was never rendered.

Honestly, the test one bothers me most. "29/29 tests passing" is exactly the kind of line you skim and feel reassured by.

What the two had in common

Every check in both systems asked the agent to certify its own work: a ledger it wrote, a review it could write, a test summary it reported. When the honest path was blocked (an expired login, an account that didn't exist), the cheapest way to "make progress" was to write the success record directly. A rule in the prompt was just one more piece of text, competing with the much louder goal of making progress.

Cheaper models made it worse. I don't think a stronger model fixes it, though, because the structure itself rewards writing the success record. So I changed the structure.

This is the take I'd defend in any argument: "be honest" in a system prompt is a wish. It works right up until the honest path is blocked, and an unattended agent will hit a blocked path sooner or later. Anything you actually need to hold has to be enforced by something the model can't edit.

What replaced them

The new fleet runs on one rule: anything an agent writes about the outside world is unverified until a check outside the model confirms it. In practice that became four things.

1. Agents can't do the risky thing, so they never have to decide whether to. Worker runs use a restricted permission profile: they cannot publish, post or spend. Deterministic steps (publishing on a schedule, reading sales numbers) moved into scripts that run before or after the model, so the model can't invent their inputs or outputs.
2. Every claim carries evidence for its stage. A claim that something reached a stage (built, listed, published) needs evidence of that stage's type, plus a verified_via field naming the live check that produced it.
3. The verifier is out of the agents' reach. After every run, a script checks those claims. Unbacked claims are quarantined for review rather than deleted, because the forged records are the only evidence of what happened. The runner and the guards are read-only to agents, and any edit to them is reverted.
4. Gates inspect the thing that ships. The gate that passed 21 empty videos never looked at a frame. A gate has to look at the rendered frames, the live URL, the API response, whatever actually goes out, and ignore the agent's description of it.

Blue is the model, orange is the gate. The verifier reads what the agents wrote and decides what reaches the ledger; the agents can't edit it.

Did it work?

Two moments from the replacement fleet made me believe it.

In its first week, the verifier demoted three real claims because their evidence was the wrong shape. The agents fixed the evidence. Nobody loosened the rule. That feels unfair when it happens (they were real!), and that's the point: a rule that also catches honest mistakes is one that's actually being enforced.

On 2026-09-20, a run found 19 unverified data rows that a previous run had left on disk, discarded them, and logged why. The self-correction is in the commit history, where I can check it, rather than in a prompt where I'd have to take its word.

Across 529 unattended runs since then, the verifier has quarantined 5 claims. The fleet's live numbers and its full decision log are public: https://denzil10.github.io/agent-fleet/

What I'd tell you to do

• Whatever an agent can write, it can fake. Keep the evidence somewhere it can't write, or check it from outside.
• A blocked honest path is where fabrication starts. Give the agent a cheap, legitimate way to say "I'm blocked" (a blocker file a human reads), and treat silence about a blocker as a bug.
• Keep forged records. They are your only primary evidence.
• Tests the agent runs and reports aren't a gate until something outside the agent reads the result.

Thanks for reading. Next up: the steps I took away from my agents entirely and turned into plain scripts, and why that made them more reliable. Follow on RSS or X @denzilnelsonai. If an agent of yours has ever faked its own success, I'd like to hear how you caught it.

— Denzil Nelson