Why Production Coding Agents Fail Without a Test Harness

We’ve seen teams give a coding agent repo access, a Jira ticket, and a prayer — then act surprised when it ships a confident little disaster.

The failure mode is almost always the same. The demo looks great. The pull request is huge. The agent “fixed” the bug, refactored three unrelated modules, updated a dependency nobody asked for, and quietly broke the one workflow that actually makes the business money. By Monday, your senior engineer is diffing 47 files like a forensic accountant at tax season.

That’s why harness engineering: leveraging Codex isn’t some fancy new label for prompt fiddling. It’s the difference between “our agent saves us hours” and “our agent created a second full-time job in cleanup.”

OpenAI’s recent field report on harness engineering makes this point clearly: if you want more autonomy from coding agents, you need an execution environment that can validate the codebase, reproduce bugs, run checks, and surface evidence before the agent gets trusted with bigger moves OpenAI.

And honestly, that matches what we’ve found in production work. The model is rarely the whole problem. The missing harness is.

Key Takeaways

• Coding agents fail in production less because they “can’t code” and more because they operate without a harness that constrains, tests, and verifies their work.
• A real harness includes evals, environment validation, bug reproduction, guardrails, and workflow checkpoints — not just a system prompt.
• If your agent can edit code but can’t prove the change works, you don’t have autonomy. You have expensive autocomplete with legal exposure.
• Start with narrow, repeatable tasks and high-signal feedback loops. Broad autonomy too early is how teams light money on fire.
• The best teams treat coding agents like strong junior engineers in a dangerous factory: capable, fast, and absolutely not allowed near every machine unsupervised.

“The Model Is Smart” Is Not a Deployment Strategy

Here’s the hot take: most teams talking about coding agents are still doing prompt engineering with better branding.

They’ll say they’re “agent-first,” but what they really mean is they pasted a repo map into a prompt, added some style rules, and hoped the model would infer everything else. That works right up until the codebase has stateful services, flaky tests, migration risk, weird shell scripts, or one undocumented cron job that keeps finance from calling you.

Which is to say: right up until reality shows up.

OpenAI describes harness engineering as building the surrounding system that helps the agent operate safely and effectively, including validating the current state of the codebase, reproducing reported bugs, and even recording a video of the failure so the agent has something concrete to target OpenAI. That’s the important shift. The work moves from “how do we ask better?” to “how do we make success and failure observable?”

That’s a much better question.

What Harness Engineering Actually Means in Practice

We think of a production coding harness as four layers.

First, state validation. Before the agent changes anything, it should know whether the repo is clean, dependencies install, tests run, services boot, and fixtures exist. If the environment is already broken, the agent will happily “fix” the wrong thing.

Second, task reproduction. If the ticket says “checkout fails for discount codes,” the harness should reproduce that failure in a deterministic way. Browser script. API call. Seed data. Expected error. If you skip this, the agent is coding against gossip.

Third, proof of completion. The agent doesn’t get to declare victory because it wrote code that looks plausible. It needs to pass tests, satisfy task-specific evals, and ideally produce artifacts: screenshots, logs, traces, or video.

Fourth, workflow control. Some actions should be automatic. Some should require approval. Some should be impossible. A coding agent should not decide on its own to rotate infrastructure, rewrite auth, or update billing logic because “it seemed related.”

That last one matters more than people admit.

Here’s how the pipeline usually works when it’s not a mess:

flowchart TD
  A[Issue or bug report] --> B[Harness validates repo state]
  B --> C[Harness reproduces failure]
  C --> D[Agent proposes and applies code changes]
  D --> E[Tests and task-specific evals run]
  E --> F{Passes?}
  F -- Yes --> G[Artifacts generated: logs, screenshots, diff summary]
  F -- No --> H[Agent revises or escalates to human]

Simple on paper. Brutal if you skip steps.

Why Your Coding Agent Keeps “Fixing” the Wrong Problem

Because your harness has no ground truth.

This is where most implementations fall apart. The agent gets a bug report written by a tired PM, maybe one screenshot from Slack, and a giant codebase with six years of sedimentary architecture. Then everybody expects precision.

Come on.

OpenAI’s harness engineering report specifically calls out reproducing the bug and validating the current state before asking the agent to solve it OpenAI. That sounds obvious. It is obvious. Teams still skip it because it feels slower.

It isn’t slower. It’s cheaper than debugging agent-made nonsense for two days.

We’ve found that coding agents behave much better when the task begins with machine-checkable evidence:

• a failing test
• a deterministic script
• a seeded database snapshot
• a browser automation trace
• a video of the failure
• explicit acceptance criteria tied to observable outputs

Without that, the agent starts pattern-matching from vibes. And vibes are not a QA strategy.

Evals Are the Real Product

A lot of teams think the product is the agent.

Wrong.

For production use, the real product is the evaluation harness around the agent. The model will improve. Tooling will improve. Today’s best coding agent will get leapfrogged. But if you own the evals, you own the quality bar.

That means building task-level checks, not just relying on the repo’s existing unit tests. Existing tests are often too broad, too stale, or hilariously unrelated to the bug you care about. We’ve seen codebases with 2,000 passing tests and a checkout flow that still exploded in production. A green test suite can lie with a straight face.

So build evals at three levels:

1. Repo health evals

These answer: is the environment sane enough to trust any result?

Examples:

• dependency install succeeds
• migrations apply cleanly
• local services boot
• lint/typecheck complete
• baseline test suite reaches expected pass rate

If this layer fails, stop. Don’t let the agent improvise on a broken stage.

2. Task evals

These answer: did the specific requested change actually work?

Examples:

• failing test now passes
• browser automation completes checkout with discount code
• API returns expected response schema
• regression fixture matches expected output
• screenshot diff stays within threshold

This is the heart of harness engineering: leveraging Codex well. You’re not asking the agent to “be smart.” You’re asking it to satisfy a contract.

3. Safety evals

These answer: what else did the agent damage while “helping”?

Examples:

• auth flow still works
• p95 latency doesn’t regress beyond your threshold
• generated SQL doesn’t exceed row scan budget
• dependency changes are limited or explicitly approved
• touched files stay inside allowed directories unless escalated

This part feels annoying until the first time the agent “fixes” a UI bug by upgrading half your frontend stack.

Then it feels like oxygen.

Here’s a useful way to visualize the harness layers:

Guardrails Should Be Boring, Not Clever

If your guardrails depend on the model behaving nicely, they aren’t guardrails. They’re suggestions.

Good guardrails live outside the model:

• filesystem sandboxing
• allowlisted commands
• network restrictions
• branch protection
• approval gates for migrations or infra changes
• file path policies
• token and runtime budgets
• secret scanning before commit
• mandatory artifact generation before merge

We’re opinionated here: don’t give coding agents broad shell access in production repos unless you’ve already built a harness that assumes they’ll misuse it.

Because they will. Not maliciously. Just statistically.

A coding agent is like giving a very fast intern a master keycard and a Red Bull. You might get brilliance. You might also get a forklift through the wall.

The Understanding Gap Is Real

One thing the current hype gets wrong is this idea that code generation equals system understanding.

It doesn’t.

Agents are getting better at local edits. They’re still uneven at understanding why a weird service exists, which invariants matter, or what tribal knowledge never made it into docs. The Reddit and blog chatter around harness engineering keeps circling this “understanding gap,” and for good reason: agents can modify systems they don’t truly comprehend.

That’s not a reason to avoid them. It’s a reason to design around the gap.

We handle that with workflow harnesses, not just test harnesses. The workflow harness decides where humans must stay in the loop:

• architecture changes
• schema migrations
• security-sensitive code
• billing logic
• cross-service refactors
• anything with unclear requirements

For narrower work — bug repro, test repair, small endpoint changes, UI fixes with visual regression checks — autonomy can be much higher.

This is the part executives usually hate hearing.

You probably don’t want a fully autonomous coding agent. You want a selectively autonomous one.

That’s better.

Start With High-Frequency, Low-Ambiguity Work

If you’re rolling this out on your team, don’t begin with “build a feature from a product spec.” That’s how people end up writing LinkedIn posts about how agents are overhyped.

Start where the harness can win:

• reproduce and fix known bugs
• add tests for existing behavior
• update small UI components with screenshot validation
• repair flaky tests
• implement narrow API changes with contract tests
• generate migration drafts that require approval before apply

This is exactly why we push clients toward constrained workflows before broad autonomy in our AI agents work. The first production wins usually come from repetitive, bounded engineering tasks with clear evidence loops, not from replacing your staff engineer with a chatbot wearing a hard hat.

Same story with on-device and voice systems. In products like RunHotel, where voice AI has to behave predictably under messy real-world conditions, the harness matters as much as the model. If the surrounding system can’t observe failure, replay it, and gate risky actions, the product gets weird fast. And weird is bad when users are standing at a hotel desk at midnight.

A Practical Harness Blueprint for Production Teams

If you want to implement this without turning it into a six-month platform project, do this:

Step 1: Pick one workflow

Choose a narrow engineering task with repeatable inputs and measurable outputs. Bug reproduction is perfect.

Step 2: Build deterministic setup

Containerized env, seeded data, stable test fixture, reproducible commands. If setup changes every run, your evals are fiction.

Step 3: Add task-specific evidence

Require the harness to produce a failing test, browser trace, logs, or a short failure video before the agent writes code. OpenAI explicitly highlights bug reproduction and failure demonstration as part of the harness approach OpenAI.

Step 4: Restrict tool access

Allow only the commands and file paths needed for that workflow. Start stingy. You can loosen later.

Step 5: Define pass/fail gates

Be explicit:

• tests that must pass
• files the agent may touch
• commands it may run
• artifacts it must return
• changes that require human approval

Step 6: Review failure cases weekly

Not just success rate. Look at bad attempts. That’s where the real harness improvements come from.

And yes, track cost. If your agent burns through tokens, retries, and CI minutes to save a 12-minute fix, congratulations, you invented a very expensive intern. We built our AI cost estimator for exactly this reason: teams routinely underestimate the operational cost of “smart” automation.

Where Cropsly Thinks This Is Going

Our bet is simple: the winning teams won’t be the ones with the fanciest model access. They’ll be the ones with the best harness.

Models are becoming commodities faster than most people want to admit. The durable advantage is in your eval suite, your execution environment, your approval logic, your task decomposition, and your operational judgment. That’s the stuff competitors can’t copy just by switching API providers.

So if you’re searching for harness engineering: leveraging Codex, don’t stop at the phrase. Build the machinery around it.

Because the phrase is trendy.

The harness is the thing that keeps your repo alive.

If you’re building coding agents, AI consulting, custom models, or adjacent systems like voice AI and on-device AI, start with one workflow, one eval loop, and one hard rule: the agent doesn’t get credit for writing code. It gets credit for proving the change works.

That’s the standard.

And if your current setup can’t tell the difference, you don’t have a production agent. You have a very articulate source of merge conflict.

Want help designing a real harness around your coding agents? Talk to us.

Sources

• OpenAI — Harness engineering