The evaluation interface, and why it's underbuilt

The evaluation interface (V) is the component most teams think they have because they have logs. They almost certainly don't.

The distinction sounds pedantic until you try to do anything with the data. Logs answer "what happened?" The evaluation interface answers "did the agent advance toward its goal?" The first is for humans reading after the fact. The second is for code reading at scale.

L vs V: a sharp distinction

From the survey (source: 2604.agent-harnesses-survey.pdf §2.2):

"A system with only L can tell you that a tool was called and when; a system with V can tell you whether that tool call advanced the agent toward its goal."

Both intercept the execution stream. The difference is what they emit.

You can have a fully-featured L (every call logged, every error audited) and still have no V. You can also have a primitive L (just print statements) but a strong V (a structured trajectory file the benchmark harness reads directly). They're orthogonal.

What a structured trajectory contains

The minimum useful shape:

{
  "session_id": "...",
  "task": { "id": "...", "goal": "..." },
  "steps": [
    {
      "step": 1,
      "phase": "act",
      "tool": "Read",
      "args": { "file_path": "auth.py" },
      "result_hash": "sha256:...",
      "ok": true,
      "latency_ms": 14,
      "tokens": { "in": 1200, "out": 30 },
      "state_delta": { "recent_files": ["auth.py"] }
    },
    ...
  ],
  "final_state": { "passes": true, "files_touched": [...] }
}

A consumer of this can answer all of:

• Did the task succeed? (final_state.passes)
• How many tool calls did it take? (steps.length)
• What did it cost? (sum of tokens)
• Where did it spend its time? (latency_ms histogram)
• Did the agent ever try to call a non-existent tool? (failed registry validations are typed errors here, not log lines)
• Two harnesses on the same task: did they take the same path? (compare step sequences)

A print statement in a log file can express none of these without parsing.

Why "raw traces" beats "summary"

This is one of the most counterintuitive empirical results in the literature, and it shows up twice independently.

Meta-Harness's ablation (source: 2603.meta-harness.pdf):

Summaries don't recover the missing signal. Even the best candidate found with summaries fails to match the median candidate found with raw trace access.

AHE arrives at the same place from the coding-agent side (source: 2604.auto-evolution-coding-agents.pdf): the same Evolve Agent under-performs on raw, unstructured traces and succeeds on a structured corpus that lets it drill down to whatever step it needs.

Both papers point to the same explanation:

This is why V matters. A good V is precisely the surface that lets something downstream (a human debugger, a benchmark, a meta-harness optimizer) navigate the trace, not summarize it.

Pass/fail isn't enough: the verification stack

One thing a structured trajectory should carry is what its checks actually proved. The 2026 Code-as-Harness survey calls this the verification stack: a layered set of checks where each layer declares what it verifies, what it cannot verify, and what confidence it contributes. (source: 2605.code-as-harness.pdf, §5.2.2)

No single layer is sufficient. Tests can be green while the specification is underspecified; GUI checkers can miss unsafe intermediate states; simulators can hide physical risk. The harness should route feedback by type: compiler errors to syntax repair, test failures to behavioral diagnosis, security findings to policy escalation, ambiguous review to arbitration.

The practical version of this for V is evidence-carrying actions. Each accepted action records not just success/failure but which checks ran, what they verified, and what remains unknown. A trace entry that says "tests passed, no coverage on branch X, no browser regression run" is far more useful than one that says "ok." (source: 2605.code-as-harness.pdf, §5.2.2)

The "progressive disclosure" pattern

You don't have to choose between summaries and raw. AHE's design is a clean middle: a layered, drill-down corpus.

• Per-task root-cause report. A debugger agent walks the trajectory, identifies the inferred root cause, and writes a structured summary with a pass/fail status.
• Benchmark-level overview (~10K tokens). Aggregates the per-task reports.
• Raw and lightly-processed traces still live behind the overview, addressable by path.

The evolver reads the overview by default and walks back to raw logs only when it needs to verify a claim. Best of both worlds: cheap to read, never lossy.

What "having V" looks like in practice

A useful test: can a script that's never seen your harness compute a per-task pass rate from your trace files?

If yes, you have V. If the answer involves "well, you'd parse the log lines and look for the string [done]", you don't.

The good news is that V is mostly a serialization discipline. Take the things you already know at each turn (tool, args, result, ok, latency) and write them as one structured object instead of three log lines. Decide on a schema, version it, document it. That's V.

Quick check

What's the practical test for whether your harness has a real V?