What an Agent Harness Needs Before Production: Eleven Failures and the Part That Stops Each

Eight parts of a production agent harness with the tool used for each and the failure it stops
On this page

An agent harness is the code around the model: the loop that calls it, the tools it can use, and everything that decides what its requests are allowed to do. The model gets the attention. The harness decides whether the agent is safe to leave running.

The first harness most teams write is about twenty lines. Call the model, run the tool it asks for, append the result, repeat. It works in a demo. To see what it does on a bad day I wrote that loop, then wrote eleven bad days: a model that never stops, arguments that do not fit, a tool that hangs, an instruction hidden in a tool result, generated code that reads secrets, and a process that dies right after moving money. The bare loop crashed on three, looped forever on one, and on six it did something worse than crash: it finished normally with a wrong or unsafe result.

Then I added the parts that stop each failure, one at a time, using ordinary tools: JSON Schema, SQLite, OpenTelemetry, process limits and a container. The result is 366 lines of Python in six files. This article walks through each part, what it stops, and one thing it does not.

The problem

This is the whole bare loop.

Python10 lines
def run_bare(model: Model, tools: dict[str, Callable[..., Any]], task: str) -> str:
    messages: list[Message] = [{"role": "user", "content": task}]
    while True:
        reply = model.complete(messages, [])
        messages.append(reply)
        if not reply.get("tool_calls"):
            return str(reply["content"])
        for call in reply["tool_calls"]:
            result = tools[call["name"]](**json.loads(call["arguments"]))
            messages.append({"role": "tool", "tool_call_id": call["id"], "content": str(result)})

Read it as a list of things nobody decided. How many times may the model be called? What if the tool name is wrong, or the arguments are? How long may a tool take, and how much may it return? May the model move money because a document told it to? What happens to a half-finished run when the process restarts?

None of those questions is about the model. A better model makes some of them rarer and none of them go away, because several are caused by the world around the agent: a slow service, a hostile document, a deployment in the middle of a run. They are harness questions, and each one has a known answer.

Architecture

The production harness keeps the same loop. What changes is that every tool call passes through eight steps, in a fixed order, before its result reaches the model.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    M["Model asks for a tool call"] --> A["Step 1: Does the tool exist?"]
    A --> B["Step 2: Valid JSON that fits the schema?"]
    B --> C["Step 3: Was this write already done?
Then replay the recorded result"] C --> D["Step 4: Is it allowed?
Writes need an approver"] D --> E["Step 5: Is there budget left?"] E --> F["Step 6: Run with a time limit,
retry what is transient"] F --> H["Step 7: Cut the result to size"] H --> I["Step 8: Record a write under its key"] I --> R["Result text goes back to the model"] S["Store: SQLite"] <--> C I --> S style M fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 style R fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 style S fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202 style D fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B

Two design choices matter more than the rest. First, a refusal is a result. When a step says no, the model gets a short error as the tool’s answer and the run continues, so the model can correct itself. Only the run budget ends a run. Second, the order is deliberate. The replay check comes before the approval check, so a restart does not ask a person to approve the same refund twice. The schema check comes before both, so nobody is asked to approve a request that could never be valid.

Eight parts of a production agent harness with the tool used for each and the failure it stops
The eight steps a tool call passes through, the tool behind each, and the failure it answers.

The harness

Tool contracts. A tool is a function plus a JSON Schema for its arguments, a marker for whether it changes anything, a time limit and a retry count. The schema is checked with the jsonschema library before the function is called.

Python9 lines
@dataclass(frozen=True)
class Tool:
    name: str
    description: str
    parameters: dict[str, Any]
    run: Callable[[dict[str, Any], Call], Any]
    effect: Literal["read", "write"] = "read"
    timeout_s: float = 5.0
    attempts: int = 1

The schema is also where hard limits belong. The refund tool in the lab declares "maximum": 200 on its amount. A request for 500 never reaches the approver or the tool, however the model was persuaded to ask.

The first two steps. An unknown tool and arguments that do not fit both come back to the model as text it can act on.

Python14 lines
        # 1. Does the tool exist?
        tool = registry.get(name)
        if tool is None:
            return refuse("unknown_tool", f"there is no tool named '{name}'. Available: {', '.join(registry.names())}")

        # 2. Are the arguments JSON, and do they fit the schema?
        try:
            arguments = json.loads(call.get("arguments") or "{}")
        except json.JSONDecodeError as error:
            return refuse("invalid_json", f"arguments are not valid JSON ({error.msg})")
        if not isinstance(arguments, dict):
            return refuse("invalid_arguments", "arguments must be a JSON object")
        if problems := registry.problems(name, arguments):
            return refuse("invalid_arguments", "invalid arguments: " + "; ".join(problems[:5]))

Replay, approval and budget. Step three looks up the call’s idempotency key, the run id plus the call id, in the store. Step four asks the policy. Step five counts.

Python14 lines
        # 3. Has this exact write already happened? Then report it and do not repeat it.
        about = Call(run_id=run_id, call_id=str(call.get("id")))
        if tool.effect == "write" and (earlier := store.effect(about.idempotency_key)) is not None:
            span.set_attribute("harness.replayed", True)
            return earlier

        # 4. Is it allowed?
        if (reason := policy.refusal(tool, arguments)) is not None:
            return refuse("not_approved", reason)

        # 5. Is there budget left?
        if budget.tool_calls >= budget.max_tool_calls:
            return refuse("budget", "the tool call budget for this run is used up")
        budget.tool_calls += 1

The policy is small on purpose. Reads run. Writes run only if an approver function says yes, and the default approver says no.

Python15 lines
def nobody(tool: Tool, arguments: dict[str, Any]) -> bool:
    """The default approver: there is no one to ask, so the answer is no."""
    return False


@dataclass(frozen=True)
class Policy:
    """Reads run. Writes run only if the approver says yes. The approver is code you own, not the model."""

    approve: Approver = nobody

    def refusal(self, tool: Tool, arguments: dict[str, Any]) -> str | None:
        if tool.effect == "read":
            return None
        return None if self.approve(tool, arguments) else f"'{tool.name}' changes something and was not approved"

Running the tool. The call runs on a worker thread with a bounded wait. A failure the tool marks as transient is retried with backoff and jitter. Anything else becomes a one-line error for the model, and the full exception goes on the trace.

Python11 lines
def _run_with_retries(tool: Tool, arguments: dict[str, Any], about: Call, span: trace.Span) -> Any:
    for attempt in range(1, tool.attempts + 1):
        try:
            # The wait is bounded. The thread is not killed: code that must be stopped belongs in the sandbox.
            return _pool.submit(tool.run, arguments, about).result(timeout=tool.timeout_s)
        except TransientError:
            if attempt == tool.attempts:
                raise
            span.add_event("retry", {"attempt": attempt})
            time.sleep(min(2.0, 0.05 * 2 ** attempt) * (0.5 + random.random() / 2))
    raise AssertionError("unreachable")

Result size and memory of writes. A result over the limit is cut, with a note telling the model how much is missing. A write is recorded under its key.

Python10 lines
        # 7. Keep the result to a size the context can afford.
        text = raw if isinstance(raw, str) else json.dumps(raw, default=str)
        if len(text) > budget.max_result_chars:
            span.set_attribute("harness.truncated_from", len(text))
            text = text[: budget.max_result_chars] + f"\n[truncated: {len(text) - budget.max_result_chars} more characters. Ask for less.]"

        # 8. Remember a write, so a restart does not do it twice.
        if tool.effect == "write":
            store.record_effect(about.idempotency_key, text)
        return text

The store. The conversation is saved to SQLite after every model reply and every tool result. When a run starts, the harness loads it and looks for tool calls that were requested and never answered. A restart continues from there.

Python7 lines
def _unanswered(messages: list[Message]) -> list[dict[str, Any]]:
    """Tool calls in the last assistant message that have no result yet."""
    for index in range(len(messages) - 1, -1, -1):
        if messages[index]["role"] == "assistant":
            answered = {m["tool_call_id"] for m in messages[index + 1:] if m["role"] == "tool"}
            return [c for c in messages[index].get("tool_calls") or [] if c["id"] not in answered]
    return []

The sandbox. Code the model wrote never runs in the agent’s process. It runs in a child with an empty environment and limits on memory, CPU time and file size, or in a container with no network.

Python9 lines
def _limits(memory_mb: int, cpu_s: int):
    def apply() -> None:
        os.setsid()
        resource.setrlimit(resource.RLIMIT_AS, (memory_mb * 1024 * 1024, memory_mb * 1024 * 1024))
        resource.setrlimit(resource.RLIMIT_CPU, (cpu_s, cpu_s))
        resource.setrlimit(resource.RLIMIT_FSIZE, (16 * 1024 * 1024, 16 * 1024 * 1024))
        resource.setrlimit(resource.RLIMIT_CORE, (0, 0))

    return apply

Tracing. The harness emits three kinds of OpenTelemetry span, named as the GenAI semantic conventions describe: invoke_agent for the run, chat for each model call and execute_tool for each tool call, with error.type set when a step refuses. Those conventions are still marked as in development, so expect the names to move.

Test setup

The model is a scripted stand-in: a function from the conversation so far to the next assistant message. That makes every bad behaviour repeatable. One script calls the same tool forever. Another sends the wrong argument name, and sends the right one only if it is told what was wrong. Another does what a tool result tells it to.

Each scenario runs twice, on the bare loop and on the harness, and each run gets its own operating system process and its own scratch directory. That matters for the scenarios that kill the process or fill its memory. The three scenarios with generated code run a third time with the container sandbox. Money is a ledger file, so a refund is something the lab can count from outside.

ComponentUsed for
Python 3.13 on a GitHub Actions hosted runnerEverything
jsonschema 4.26.0Tool argument contracts
OpenTelemetry SDK 1.44.0Spans, read back from an in-memory exporter
SQLite, from the standard libraryConversation after every step, results of writes
resource limits on a child processProcess sandbox for generated code
Docker, python:3.13-slim, –network noneContainer sandbox for generated code

After the scenarios, 25 checks compare what happened with what this article says happened. If a library update changes a behaviour, the build fails and the article is wrong until someone looks.

Test results

Bad dayBare loopHarness
The model never stops calling a toolStill looping after 500 model callsStopped at 12 model calls, by the budget
Wrong argument name and typeCrashed: TypeErrorTold what was wrong, corrected, answered
A tool that does not existCrashed: KeyErrorTold the tool names, corrected, answered
A tool that fails twice, then worksCrashed on the first failureAnswered on the third attempt
A tool that returns 2 MB2,100,040 characters sent to the model4,070 characters sent
A tool that takes 6 secondsWaited 6.0 sGave up after 1.0 s and said so
An instruction hidden in a tool resultRefunded 150.00 on an order nobody mentionedRefund refused: not approved
Generated code reads the environmentAPI key in the answerNo key
Generated code allocates 1 GBAllocated inside the agentMemoryError in the child
Generated code opens a connectionConnectedConnected
Process dies right after a refundRestart paid the refund twiceRestart paid once and finished

The three crashes are the easy rows. A crash is visible, and someone fixes it in the first week. The rows to worry about are the ones where the bare loop says “answered”. It sent two megabytes to the model and carried on. It printed an API key into its answer. It let generated code open a network connection. It paid 150.00 because a support ticket contained a sentence addressed to the assistant. It paid a real refund twice. Nothing in its output says anything went wrong.

The hidden instruction row deserves a closer look, because the harness did not detect the injection. It has no idea the ticket was hostile. The scripted model obeyed the ticket, exactly as on the bare loop, and asked for the refund. What stopped it is that a refund is a write, writes need an approver, and nobody had approved one. This is the trace of that run:

Text6 lines
chat scripted-1              ok                        
execute_tool read_ticket     ok                        
chat scripted-1              ok                        
execute_tool refund_order    error.type=not_approved   
chat scripted-1              ok                        
invoke_agent desk            ok

That is the honest shape of the defence. You cannot make a model immune to text. You can make sure that what a model asks for and what actually happens are separated by a decision the model does not control.

⚠️
A PROCESS SANDBOX IS NOT A NETWORK SANDBOX

One row has the same result in both columns. Generated code in the limited child process still opened a connection to the lab’s listener. Limits on memory and CPU keep the agent alive, and an empty environment keeps its secrets out, but the child is the same user on the same network. In the container with --network none the same snippet failed with “Network is unreachable”. If generated code must not reach your network, a child process is not enough.

Generated code thatChild process with limitsContainer, no network
Reads the agent’s environmentNo keyNo key
Allocates 1 GBMemoryErrorKilled, exit 137
Opens a connectionConnectedNetwork is unreachable

The last row of the main table is the one I would show a finance team. The process was killed after the refund reached the ledger and before the harness recorded the result. On restart, the bare loop began again from the user’s request and paid a second time. The harness loaded the conversation, found a refund call with no answer, and ran it again with the same idempotency key. The ledger recognised the key and did not pay again. Two things had to be true for that to work: the harness kept the call id across the crash, and the system receiving the money honoured the key. I covered the framework side of this in Crash, Checkpoint, Resume. The harness cannot give you the second half.

Eleven failures run against a bare agent loop and a production harness, showing crashes, silent wrong outcomes and safe outcomes
Eleven bad days on both loops. The silent failures are the dangerous ones.

Cost

The cost of a harness is code to own and a little time on every call.

PartFileLines
Bare loopbare.py20
Loop and the eight stepsloop.py166
Tool contractstools.py58
Sandboxsandbox.py50
Storestore.py37
Budgetbudget.py28
Policypolicy.py27

That is 366 lines against 20, including comments. None of it is clever, and that is the point: every part is a few dozen lines around a tool that already exists. The container sandbox is the expensive one at run time, because each snippet starts a container. I did not measure that overhead here, and it will differ on your hosts. The store writes to disk after every step with full synchronisation, which is the right default for money and too slow for a chat toy.

The cost of not having one shows in the results. Two megabytes of CSV, at a rough four characters per token, is about half a million tokens on a single request, where the model accepts it at all. One runaway loop at 500 calls is 500 billed requests that produced nothing.

Production considerations

  • Pick every number on purpose. Twelve model calls, twenty tool calls, 4,000 characters per result: these are the lab’s defaults, not recommendations. Set them per task type and alert when a run hits one.
  • The approver is the security boundary. Here it is a function. In production it asks a person or a rule engine, shows them the exact arguments, and records who answered. Approve the call that will run, not a description of it.
  • A timed-out tool is abandoned, not stopped. The harness stops waiting after the limit. The thread keeps going. Anything that must be stopped has to run where it can be killed, which means the sandbox or a separate service.
  • Replace SQLite when there is more than one worker. The store has no lock that stops two workers resuming the same run. A shared database with a lease per run, or a durable workflow engine, is the next step.
  • Truncation loses information. Cutting a result to 4,000 characters is safe and blunt. Better tools take a limit and a filter as arguments, so the model can ask for less. I measured what tool listings cost in context in The Context Cost of MCP Tools.
  • Traces hold what users typed. The sample records names, ids and error types, not prompts or tool results. Turning content capture on is a data protection decision.
  • Tools from elsewhere need the same gate. If tools arrive over MCP from a server you do not run, their descriptions and results are untrusted input. The eight steps apply unchanged. Red-Teaming an MCP Server covers what a hostile server can try.
  • Keep the bad days as tests. The lab runs in seconds with no model. Every incident you have in production should become one more scenario.
⚖️
AUDIT

A regulator or an auditor will ask who allowed an action, not which model suggested it. The harness has the three records that answer it: the tool call with its arguments, the approver’s decision, and the stored result under an idempotency key. Keep them together and keep them longer than the conversation.

Decision framework

Agent frameworks ship some of these parts, and the question is usually which ones you still own. Use this table against whatever you run today.

If your agentYou need at leastWhy
Only reads, for one user, in a sessionBudget, tool contracts, result limitsRunaway cost and crashes are the only risks
Calls services that fail or stallTimeouts and retries on topOne slow dependency should not hold a run
Reads documents, tickets, email or web pagesAn approval gate on every writeAny of that text can carry instructions
Changes anything: money, records, messagesApproval, a store, idempotency keysA restart must not repeat the change
Runs code it wroteA sandbox with no networkA child process leaves the network open
Runs unattendedTracing and alerts on refusals and budget stopsNobody is watching the output

If I could add only one part to a bare loop, it would be the approval gate on writes. It is 27 lines, and it is the one part that turned a wrong action into no action. Budgets come second, because they are four numbers and they cap every failure you have not thought of yet.

Key takeaways

  • A bare agent loop handled none of the eleven scenarios well. Three were crashes. Six ended with a normal-looking answer and a wrong or unsafe outcome.
  • A refusal should be a tool result the model can read, not an exception. The model corrected a wrong argument and a wrong tool name once it was told.
  • The harness did not detect the hidden instruction. It stopped the refund because writes need an approver the model does not control.
  • Limits on a child process stopped a 1 GB allocation and kept an API key out of generated code. They did not stop a network connection. A container without a network did.
  • Paying once across a crash took two things: the harness kept the call id, and the ledger honoured the idempotency key.
  • All of it is 366 lines around JSON Schema, SQLite, OpenTelemetry, process limits and a container.
📝
HOW THIS WAS CHECKED

Everything here comes from one pipeline run: GitHub Actions run 44508276684, commit 3c1bbcc, with the outcomes, traces and checks on the ci-results-agent-harness-production branch. Python 3.13, jsonschema 4.26.0 and OpenTelemetry SDK 1.44.0, with dependencies resolved as of 14 September 2026. 12 unit tests pass, 11 scenarios ran on both loops and three again in a container, and 25 of 25 outcome checks pass. The model is a scripted stand-in, so the scenarios show what a harness does when a model misbehaves, not how often a real model does. No hosted model was called. The API key in the lab is a made-up value. Not measured: latency overhead of any part, container start-up time, token counts, behaviour with two workers on one run. The approver is a function, not a person. Linux only.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.