Creating effective, production-ready applications with Large Language Models (LLMs) remains a major stumbling block for teams seeking real ROI on AI. The failure usually boils down to a few missteps: misunderstanding how LLMs function, applying them without structured architecture, and treating them as autonomous black boxes rather than bounded, probabilistic components within a deterministic system.
This is the common cause of the poor reputation AI is picking up: hallucination, inconsistency, and a general unreliability that people then blame on the technology itself.
The model you choose is not the defining question. Models are commodities; your architecture should be model-agnostic by design. The real product is the harness: the scaffolding, context pipeline, and execution loops working together like cogs in a machine to deliver repeatable, reliable outcomes across automated workflows, document pipelines, and bespoke internal tools.
This is not a fringe view any more. In early 2026 OpenAI published an account of a small team that shipped roughly a million lines of production code over five months without a human writing a single line. Their conclusion was that the engineers’ job had shifted from writing code to designing environments, specifying intent, and building feedback loops so agents could do reliable work (OpenAI, Harness engineering: leveraging Codex in an agent-first world). The model was not the story. The harness was.
To build software that works, you first must rethink how you view the model. Approach it like an engineer would approach anything.
What is a harness?
In practice, a harness is the middleware layer that bridges the model and your application domain. It manages state, injects relevant context, executes tool calls, retrieves external data, and enforces schemas. Whether you are looking at GitHub Copilot or an enterprise data-processing agent, the harness dictates how context enters the model and what happens to the output.
This discipline is the foundation of Context Engineering and Harness Engineering. The industry has settled on a simple shorthand for it: Agent = Model + Harness. If you are not the model, you are the harness.
LLMs are stateless, probabilistic text-prediction engines. The harness houses all the deterministic components that make an application reliable and seemingly intelligent, including memory persistence, output verification, rate limiting, and side-effect execution.
Anthropic’s engineering team makes the same point from the other direction: even a frontier coding model, left to loop across multiple context windows with only a high-level prompt, will fall short of building a production-quality application without a harness that tracks progress and leaves artefacts for the next session (Anthropic, Effective harnesses for long-running agents).
See this GitHub repo for a simple harness written in Python, as a demonstration of how modern AI IDEs and agents actually operate.
Shift your understanding about AI
Prompting alone only defines the loose boundaries for a model’s generation. Generic out-of-the-box harnesses (like default IDE assistants or raw chat UIs) only take you so far. Real utility comes from building project-specific harnesses tailored to your exact domain constraints.
A strong starting point is using structured methodologies like GitHub’s Spec-Kit or schema-first design. However, taking it a step further, when building out applications you should ask: “Where does probabilistic generation actually belong in this system, and where must deterministic code handle the rest?”
Birgitta Böckeler at Thoughtworks frames this usefully as a split between guides, which steer the agent before it acts, and sensors, which observe the result and let it self-correct, with each being either deterministic (linters, tests, validators) or inferential (LLM-as-judge). The deterministic ones are cheap and fast enough to run on every change (Böckeler, Harness engineering for coding agent users).
That is exactly the layer most teams skip. When you make this shift, you quickly realise that smaller, cheaper, or even local models can deliver exceptional results when wrapped in a disciplined execution loop. There are endless possibilities with this kind of systems thinking; you do not need the latest and greatest frontier models.
Effective harness engineering
Building production-grade harnesses requires moving away from massive prompt blobs and toward disciplined engineering patterns. These are the patterns we landed on during the development of our internal Apex methodology:
- Personas and policies as config. Decouple agent instructions, system prompts, and tool definitions into version-controlled configuration files rather than hardcoded runtime strings.
- Orchestrated specialist routing. Use a central entry point to classify and dispatch tasks to scoped, specialised worker agents rather than relying on a single monolithic prompt to do everything.
- Deterministic verification and self-correction. Give the model execution feedback (unit test runners, linter output, JSON schema validators) so it can self-check and correct errors before returning a final result.
- Human-in-the-loop safeguards. Gate destructive actions and high-stakes decision points behind explicit human review.
This is also why Apex is built the way it is. As Riaan Swart, who leads the Apex framework and harness at First Digital, puts it: “The point is not to write code faster for its own sake, but to keep the quality bar high while the delivery cycle gets shorter.”
Apex threads AI through every stage of delivery rather than bolting an assistant onto one of them, and it deliberately stays model-agnostic, running on cloud or private token factories depending on what the work and the client’s data sovereignty needs allow.
When you invest in the harness, you stop getting inconsistent low-grade output and start generating grounded, verifiable results. You can finally trust your AI-powered application because you engineered the system that governs it.
