Who on-calls an AI? This is the unresolved crisis of agent ownership

At 2:07 a.m., an AI agent starts failing.

The application is still running, but requests are timing out because a downstream API is returning errors. The agent has already retried twice, using more tokens than usual, and its response time has roughly doubled.

The first question should not be:

It should be:

That question is easy to overlook when building AI agents. Teams often pour effort into prompts, retrieval, tools, and model selection, but production ownership determines whether an agent becomes a dependable service or another source of operational risk.

“Teams often pour effort into prompts, retrieval, tools, and model selection, but production ownership determines whether an agent becomes a dependable service or another source of operational risk.”

A production agent needs an owner, boundaries, telemetry, failure policies, escalation paths, and a clear distinction between what the system may automate and what requires human intervention.

A production agent should not have a direct path from reasoning to action. Each tool invocation should pass through explicit controls, remain observable, and have a defined path to human intervention.

Here’s a practical way to build those controls.

An agent should have the same operational identity as any other production service.

Start by defining its ownership metadata.

This may look simple, but it changes the conversation.

The agent is no longer an experiment running somewhere in a Python process. It is a service with an accountable team and an escalation path.

“The agent is no longer an experiment running somewhere in a Python process. It is a service with an accountable team and an escalation path.”

You can later attach the same metadata to logs, traces, dashboards, alerts, and deployment records.

Ownership also means being clear about what an agent is allowed to do.

An agent that can only read telemetry carries far less risk than one that can restart production services, change infrastructure, or deploy code.

That’s why I prefer an explicit tool allowlist, which limits the agent to the actions you’ve deliberately granted it.

The guiding principle is deny by default.

Adding a new capability should be an intentional engineering decision, not something that happens because a tool became available to the agent.

For destructive operations, authorisation should be even stricter. A production restart or deployment should require authenticated human approval, not the agent’s own judgment.

An agent that can reason indefinitely is difficult to operate.

Every production workflow should have limits around execution time, model calls, tool calls, and ideally token consumption.

Execution limits should be configurable rather than buried in application code. By passing these limits through the constructor, engineers can assign different budgets to different agents without changing global settings. A diagnostic agent might need only three steps, while a more complex research workflow may require considerably more. When no limits are supplied, the class falls back to environment variables. What matters is that every workflow has a defined ceiling, and that the same budget is enforced throughout its execution.

An owner cannot operate what they cannot see.

At minimum, record:

A simple wrapper provides a useful boundary.

The request ID becomes especially valuable when an incident involves several services.

An engineer should be able to move from an alert to the agent request and then to the underlying model and tool calls.

In production, supplement this logging layer with distributed tracing and metrics for a more complete view of system behaviour and performance.

A failed model call does not necessarily mean the entire service should fail.

Likewise, a successful model call does not mean the agent’s recommendation should execute automatically.

These are different decisions.

This distinction matters during a real incident.

The agent might go down, but the engineering team should still be able to respond using the underlying workflows and runbooks. That is what graceful degradation looks like in practice.

One of the most useful patterns for enterprise agents is separating recommendation from execution.

For example:

The agent might recommend:

But it shouldn’t push that change to production just because the model generated the recommendation.

Here’s what a simple approval boundary looks like:

A real system needs a more sophisticated implementation, but the architectural boundary is what matters.

“The agent can own reasoning without owning authorisation.”

The agent can own reasoning without owning authorisation.

A production agent needs a health endpoint just like any other service.

With FastAPI:

Liveness and readiness serve different purposes. The liveness endpoint confirms that the application process is responsive, while readiness checks whether the service can handle requests based on the availability of its required dependencies.

In this example, the readiness endpoint checks a configured model-provider health URL and returns HTTP 503 when the check fails. The check is deliberately lightweight and time-limited to avoid introducing unnecessary delays or costs. In production, teams should adapt it to their provider’s supported health-check mechanism and other critical dependencies, such as a vector database or configuration service.

Keeping these checks separate lets orchestration platforms such as Kubernetes distinguish between an application that needs restarting and one that can’t serve traffic temporarily.

One of the worst failure modes is letting an agent keep running after its safety mechanisms go down.

For example:

Notice that different failures produce different outcomes. An authorisation failure should block the requested operation, while an exhausted execution budget should trigger escalation rather than another retry. A temporary model failure, meanwhile, should follow the system’s defined failure-handling policy.

Another important detail: the execution budget must persist throughout the agent’s workflow. Creating a new budget for every invocation would reset its counters, allowing repeated calls to bypass the intended limits. Instead, create the budget once and pass it into each successive invocation, as shown below.

With the same budget shared across invocations, each counted step and tool call contributes to the workflow’s overall limit. Once that limit is reached, the agent stops rather than continuing with a fresh budget. These controls only work if they remain effective throughout execution, especially when an agent encounters repeated failures.

This is where operational ownership becomes real.

Suppose the agent starts failing at 2 a.m.

The on-call engineer should be able to answer five questions quickly:

The last question matters most.

Every production AI system should have a clear degradation path.

For an incident-response agent, that might mean:

The AI system should speed up operations, not become something the organisation can’t operate without.

The answer should never simply be:

Production ownership needs to be explicit.

Someone owns:

That doesn’t mean one person has to handle everything. It means everyone knows their role well enough that, when an incident hits, the team isn’t wasting time figuring out who should take charge.

The easiest time to operate an AI agent is during business hours when everything is working.

The real test comes at 2 a.m.

A production-ready agent should have an owner who can be paged, an execution budget that prevents runaway behaviour, observability that explains what happened, authorisation boundaries that constrain what it can do, and a fallback that allows engineers to continue without it.

The goal isn’t to take humans out of the loop.

“A production-ready agent should have an owner who can be paged, an execution budget, observability that explains what happened, authorisation boundaries, and a fallback that allows engineers to continue without it.”

It’s to make it clear where the agent’s responsibility ends and where a human needs to step in.

Because when something goes wrong at 2 a.m., the question isn’t whether the model was smart enough.

It’s whether the system was designed well enough that the person on call knows what happened, what the agent did, and what to do next.