Engineering Rigor: DeepSeek Harness's Verification Gate (Part 14)

Series: Inside the DeepSeek Harness — Part 14 of 16

Thirteen posts of elegant primitives, real invariants, and dated bug fixes raise an obvious question: what actually stops ~150 packages from drifting apart? This post is the answer — the real verification pipeline, the philosophy behind it stated in the contributors’ own words, and a pointed reminder from Part 12’s PM-0001 that even a rigorous gate has a specific, nameable blind spot.

The real tiers, and what each one actually proves

The coverage gate’s own stated philosophy is worth quoting directly, because it inverts the usual instinct: “An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on.” The gate isn’t there to be satisfied by writing more tests — sometimes the correct response to a coverage failure is deleting the line the gate flagged, not testing around it.

doc-sync’s verify-type-equiv step is the sharpest tool in the whole pipeline: it extracts a documented type declaration straight from source using the real TypeScript parser, and byte-diffs it against whatever’s pasted into the corresponding doc page. Every code snippet in every post of this series that says “real code, not paraphrase” is implicitly leaning on the same idea — this repo has built a mechanical version of exactly that promise, and enforces it on every doc, every commit.

“We are DeepSeek — do not ration real-API tests”

That’s a real, verbatim line from the testing philosophy, and it’s a genuinely distinctive piece of engineering culture worth sitting with: “A no-key test proves plumbing; only a with-key run proves the agent works against a real model.” Most organizations ration real-model test calls because inference is expensive relative to their budget. DeepSeek, as the model’s own creator, is explicitly leaning on the fact that inference is cheap for them to justify a testing posture — liberal real-model smoke tests — that most teams genuinely cannot afford to copy. It’s a rare case of a company’s testing philosophy being shaped by its own economics, stated out loud rather than left implicit.

A few more real, quotable rules from the same source:

“Verify the world, not the self-report.” — an e2e assertion has to re-run the command or re-read the file externally; a keyword probe on the agent’s own output text would let a confused or cheating agent pass anyway.

“Mock only the expensive or non-deterministic boundary (LLM adapter, network, clock); keep everything downstream real.”

“A guard only guards if the regression actually fails it.” — the required workflow: introduce the regression, watch the test actually go red, then revert. Not aspirational. A stated process requirement.

What 100% coverage didn’t catch — and why that’s not a contradiction

Part 12’s PM-0001 is worth revisiting through this lens specifically. The ACP server had 178 green tests and full line coverage, and still shipped a bug where export default silently dropped its inject array, crashing the server at load time. That’s not a failure of the coverage gate’s logic — it’s a precise illustration of what “100% line coverage” actually promises and what it doesn’t. Every test in that suite mounted the plugin by hand, supplying inject manually. That path can never exercise unwrapExports, the real Loader function that only runs when a plugin is mounted the way production actually mounts it. The lines were covered. The path production actually takes wasn’t.

This is exactly the kind of thing “real API tests, verify the world, match the real Loader path” tries to close — and it’s a genuinely fair, non-defensive thing for a codebase’s own documentation to have said about itself, since the fix (test through the real Loader, not a hand-mounted stand-in) is now presumably part of how ACP gets tested going forward.

AGENTS.md: the rules a contributor — human or AI — is actually held to

A handful of lines from the repository’s own contributor guide, verbatim, because they read as a genuinely coherent design philosophy rather than a style-guide grab-bag:

“Registrations are effects: every contribution goes through ctx.effect() / ctx.on(); a registry’s register() returns the disposer.” — the Part 1 primitive, restated as a hard rule for anyone adding code.

“Explicit > implicit at package boundaries: defaulting is an explicit resolve(request): Spec step in the owning implementation, never a hidden ?? default inside run().” — Part 5’s shell seam, generalized into a repo-wide law.

“No hardcoded tunables in plugins: deployment-varying choices are validated Config fields changeable from cordis.yml; a DEFAULT_* constant or test hook is not configurability.” — a sharp, specific rejection of a very common shortcut.

“Trust TypeScript at typed same-process boundaries. Do not add runtime validation… for values the static interface requires; validate at parser/config, queued, model/tool JSON, durable/file, worker, process, and wire boundaries.” — a precise rule about where defensive coding earns its cost, and where it’s just noise.

“Misconfiguration fails loud at load when self-contained, otherwise at the earliest resolvable point; never silently skip a missing referent.” — the exact instinct that would have caught PM-0002’s !!js bug from Part 12, stated as a general principle rather than a one-off fix.

One structural rule ties directly back to this series’ own primary sources: any non-trivial change requires an Agent Note — a short design-rationale document — in the same pull request, and once archived, a note is frozen and never edited again. That’s the actual origin of the .agents/notes/ directory this series has cited repeatedly: it’s not incidental documentation, it’s a mandatory, append-only paper trail baked into how changes get made at all.

Coming up in this series

(All 16 parts are live today — no daily drip.)

Part 16 shipped after this series’ initial 15 posts, once Agent Presets and Code Mode landed — it reads best right after Part 4, which is why its row sits there instead of at the end.

Next: the finale — fourteen posts of primitives, seams, invariants, and real incidents, synthesized into a direct comparison against April’s Agent Harness series, and a short list of what’s genuinely worth stealing.

References

• docs/testing.md, deepseek-ai/deepseek-harness — verification tiers and testing philosophy quoted above (all direct quotes in this post are sourced from this file and AGENTS.md, not docs/development.md)
• AGENTS.md — the contributor guide, quoted extensively above
• docs/postmortem/0001-acp-default-export-drops-inject.md — the coverage-gate blind spot discussed above (also covered in Part 12)
• .agents/notes/proposed/testing/2026-06-11-deterministic-and-stress-testing.md and 2026-06-11-mutation-testing.md — proposed future testing rigor
• Part 1 through Part 13 of this series