AI coding agents can take on increasingly large implementation tasks. A common workflow is to write a detailed specification, give the agent access to the repository, and ask it to implement a feature end to end.
But larger implementation scope may create unnecessary work for the model. Even with a detailed specification, an agent implementing several interacting components simultaneously has to reason about their interfaces, tests, dependencies, and failures throughout the run.
An alternative is to keep the overall system design unchanged while reducing the implementation scope at each step: implement one component, verify it, and continue only after that component reaches a known-good state.
A small controlled experiment was conducted to compare these two approaches. In this case, the structured workflow reduced estimated token-based cost by approximately 21%, used 25% fewer non-cached input tokens, and completed in roughly half the agent execution time. Both implementations passed the same 57-test acceptance suite.
Experiment setup
The experiment used a retrieval component from an existing AI application. The implementation consisted of two main layers:
Query
|
v
VectorRetriever
|
| ranked chunks
v
DocumentRetrievalAdapter
|
| ranked unique documents
v
Retrieval evaluationVectorRetriever embeds the query and retrieves ranked chunks from Qdrant. It handles filtering, embedding-dimension validation, batching, and conversion of Qdrant results into the application's retrieval types.
DocumentRetrievalAdapter operates one level above the retriever. Because multiple chunks can belong to the same source document, the adapter converts the chunk ranking into a ranking of unique documents. If the initial retrieval does not contain enough unique documents, it progressively retrieves additional chunks.
The repository already contained the interfaces, shared types, evaluation code, and tests. The implementations of these two components were removed before the experiment, giving both runs the same starting point.
The specification, repository, Python environment, model configuration, and acceptance tests were also held constant. The primary variable was how the implementation work was presented to Codex.
Workflow A: Implement the complete specification
The first run used a conventional one-shot workflow.
Codex received the complete specification and was asked to implement both retrieval components, inspect the repository as needed, run the tests, and correct any failures.
Conceptually:
Complete SPEC
|
v
Implement complete
retrieval feature
|
v
Run tests / debug
|
v
DoneThe resulting implementation passed all 57 acceptance tests.
The run also involved several correction cycles. Some were environment-related rather than implementation failures—for example, resolving the Python/test runner and dependency environment—so these were treated separately when interpreting the result.
Workflow B: Test-gated implementation
The second run started from the same repository snapshot and used the same complete specification. The difference was that implementation was divided into two independently verifiable milestones.
The first milestone covered only VectorRetriever. Codex implemented the component and stopped after its relevant test suite passed.
Result: 20 tests passed.
A Git checkpoint was then created before beginning DocumentRetrievalAdapter. The agent could use the completed retriever but was instructed not to redesign it unless an integration issue required a change.
Result: 14 tests passed without modifying the first component.
Finally, the complete retrieval acceptance suite was executed.
Complete SPEC
|
v
VectorRetriever
|
20 tests
|
v
checkpoint
|
v
DocumentRetrievalAdapter
|
14 tests
|
v
checkpoint
|
v
Integration tests
|
57 passedNo implementation changes were required during the final integration step.
Results
| Metric | One-shot | Test-gated | Change |
|---|---|---|---|
| Estimated token-based cost | $0.326 | $0.256 | -21.3% |
| Input tokens | 832,953 | 694,156 | -16.7% |
| Cached input tokens | 784,640 | 658,048 | -16.1% |
| Non-cached input tokens | 48,313 | 36,108 | -25.3% |
| Output tokens | 7,199 | 5,244 | -27.2% |
| Reasoning tokens | 266 | 219 | -17.7% |
| Total tokens | 840,152 | 699,400 | -16.8% |
| Agent execution time | ~12m 34s | ~6m 13s | -50.5% |
| Code churn | 190 lines | 297 lines | +56.3% |
| Acceptance tests | 57/57 | 57/57 | — |
The cost figures are token-based estimates rather than observed ChatGPT subscription charges. They were calculated from cached input, non-cached input, and output usage using a common reference rate card. The figures therefore provide a consistent basis for comparing the two runs, but should not be interpreted as the actual amount billed for either Codex session.
The reduction was also visible across several usage metrics rather than appearing in cost alone. Total input decreased by 16.7%, the more expensive non-cached input decreased by 25.3%, and output decreased by 27.2%.
Where did the reduction come from?
One possible explanation is that smaller implementation steps simply cause the agent to generate less code.
The experiment does not support that explanation.
The test-gated implementation produced 297 lines of code churn compared with 190 lines in the one-shot implementation, an increase of approximately 56%.
Comparison of the implementations showed that much of the additional code came from more explicit validation and defensive behavior. The incremental implementation included additional checks around query inputs, filters, embedding dimensions, resource lifecycle, and concurrency.
The reduction in token usage therefore cannot be explained simply by the agent performing less engineering work.
A more interesting difference appeared at the integration boundary.
During the final integration step, the incremental workflow used 89,569 input tokens, of which 88,704 were cached. Only 865 input tokens were non-cached, and no implementation changes were required.
This suggests a possible mechanism behind the observed reduction.
In a one-shot implementation, an error discovered late in the process can originate from either component, their interface, or assumptions introduced earlier in the run. Resolving the problem may require the agent to inspect and reason across a relatively large implementation surface.
With test-gated implementation, the first component is already in a known-good state when work begins on the second. Once both component-level gates pass, integration begins from two independently verified boundaries.
The model still has access to the global architecture, but the active implementation problem is smaller.
Separating design scope from implementation scope
The experiment highlights an important distinction:
The scope used for system design does not need to be the scope used for implementation.
Codex still received the complete system specification. It knew that the retriever would eventually feed a document-level adapter and understood the interface between the two components.
What changed was the unit of implementation.
Instead of:
Understand system -> implement system -> test systemthe structured workflow became:
Understand system
↓
implement A -> verify A
↓
implement B -> verify B
↓
verify A + BThis is a relatively small change to the AI-assisted development process. It does not require a different model, agent framework, or repository architecture.
A practical pattern is therefore to define the overall architecture and interfaces first, while giving the coding agent one independently testable implementation boundary at a time.
Limitations
This experiment has several important limitations.
First, it contains only one run of each workflow. Model behavior is stochastic, so the approximately 21% difference observed here should not be interpreted as an expected 20% saving across AI coding tasks.
Second, the experiment involved one retrieval project with two relatively well-defined components. Results may differ when components are tightly coupled or cannot be meaningfully tested in isolation.
Finally, both implementations passed the same 57 public acceptance tests, but a separate hidden retrieval-quality evaluation was not completed because usable credentials were unavailable in the A/B environment. The result therefore demonstrates equivalence against the frozen acceptance suite, not identical behavior across every retrieval workload.
A stronger experiment would repeat both workflows multiple times and compare median cost, execution time, correction cycles, and implementation quality.
What's next
The next step is to repeat the experiment across additional engineering tasks and multiple runs per workflow. Two questions are particularly interesting: whether the reduction persists as the number of modules increases, and at what point the overhead of additional agent turns begins to offset the benefit of smaller implementation scopes.
For this retrieval task, keeping the design global while making implementation incremental reduced estimated token-based cost by approximately 21% and cut agent execution time roughly in half, while preserving the same acceptance-test result.
The broader hypothesis is straightforward:
Give the coding agent the full architecture—but not necessarily the entire architecture to implement at once.
Try the Test-Gated Coding Prompt
The practical pattern from this experiment is simple:
Give the agent the full architecture, but only one implementation boundary at a time.
A reusable prompt template is available for applying this workflow to multi-component coding tasks with Codex and other coding agents. It is a practical pattern motivated by this experiment, not a guarantee of lower token cost.
The companion repository preserves the shared specification, workflow and provenance documentation, implementation snapshots, and implementation diff and findings. The exact original prompts and gate instructions were not preserved; this reusable template is not a reconstruction of them.
The initial result is promising enough to justify testing that hypothesis further.