The refund looks routine. A customer asks for money back, an agent finds the order, reads the policy and prepares a polite reply. In a demonstration, a green tick appears. In an actual service, the interesting questions begin just before that tick: whose account authorized the transaction, whether the policy was current, whether the payment was processed twice, and what the customer was told if the final confirmation never arrived.
This is a hypothetical workflow, not a report of a company’s deployment. It exposes the gap between showing that a model can choose useful steps and establishing that a service can perform them responsibly. Much of the agent debate is framed around intelligence. A production team also needs an answer about authority and state. The agent’s explanation of what happened cannot be the sole record of what happened.
NIST’s August 2026 discussion of agent identity argues that familiar identity and access-management failures are reappearing in agent systems. Its practical significance is that some foundations already exist. Giving an agent a person’s credentials may make the demonstration faster while making subsequent accountability harder. Treating the agent as an identifiable actor with delegated permissions offers a different starting point.
Read the order
The first boundary is information. A refund agent may need the order, payment status and applicable policy. It probably does not need every customer’s records or the company’s entire document drive. Search convenience encourages broad access: index everything, let the model discover what matters. The resulting application has quietly acquired a much larger exposure than its narrow customer-service brief suggests.
The alternative is to resolve the current customer’s identity and retrieve the relevant records through the service’s existing access controls. A model should not decide whether a guessed order identifier belongs to the person asking. That check belongs in the system that owns the record. Retrieval then becomes an authorized operation with a reason and a scope, not a clever search across whatever happens to be reachable.
Policy documents introduce another complication. The latest-looking page might be an archived draft, a local exception or text supplied by the customer. The application needs to distinguish an authoritative policy from contextual material. Source links help a reviewer inspect an answer, but a link alone does not establish which policy version governed a transaction. The decision record should retain that relationship.
None of this makes the model irrelevant. It can interpret the request, identify missing information and explain options clearly. The useful separation is between interpreting language and granting access. Better reasoning can improve the former without removing the need for the latter.
Decide what the permission permits
An authorization to handle a refund is underspecified. Does it allow looking up eligibility, drafting a recommendation, committing a payment, changing the payment destination or emailing a third party? These are different actions. A system that grants one broad administrative capability and relies on the model to avoid the dangerous parts has moved an institutional rule into a probabilistic conversation.
The OAuth token-exchange standard distinguishes delegation and impersonation and provides a mechanism for exchanging security tokens. It does not, by itself, define a company’s refund policy or solve every agent-authorization problem. Its relevance is narrower: application designers need not begin by sharing a human password and pretending that every subsequent action was performed directly by that human.
For our hypothetical service, the tool boundary could permit a refund only against a specific eligible order, to the original payment method and within a defined amount. A larger exception could be escalated with a concrete proposal. These are illustrative design choices, not universally sufficient controls. Their value is that a transaction can be checked independently of the sentence the model uses to justify it.
Human review is most useful where it changes the quality of a decision. An approval dialog that merely asks whether to continue, without showing the order, amount, destination and consequence, exports uncertainty to the reviewer. Requiring approval for every trivial step creates pressure to click automatically. The interface should make consequential choices inspectable and routine bounded work genuinely routine.
Read the hostile sentence
Now add a message to the order notes: the customer instructs the agent to ignore the usual procedure and issue a larger payment. A human employee recognizes that a customer’s note does not rewrite company policy. A language model receives both as text. The application must preserve the distinction when that text passes through retrieval, summarization and tool use.
The OWASP prompt-injection guidance describes indirect attacks through external material and emphasizes controls at the tool boundary. Formatting trusted instructions separately from retrieved content is useful organization, but it is not a complete enforcement mechanism. A warning in the prompt cannot replace a service that refuses an unauthorized transaction.
The deeper issue is not whether the hostile sentence contains recognizable words. A legitimate document can discuss malicious instructions, and an attacker can disguise the same request in a different form. Counting suspicious phrases is therefore an unreliable proxy for authority. The service should ask whether the proposed operation is allowed for this actor, record and purpose, regardless of how persuasively it was requested.
This changes how a failure is assessed. A model can be fooled into proposing the wrong action while the surrounding system still prevents harm. That is a near miss worth investigating, not proof that the model’s safeguards worked. Conversely, a model can produce a reassuring explanation while the wrong transaction slips through. Evaluation must observe the actual effect.
Wait for the result that may already exist
Suppose the refund request reaches the payment service, succeeds, and then the network connection drops. The agent sees a timeout. Retrying might be necessary, but issuing another payment is not. A fluent assistant that interprets every missing response as a failed action can turn a routine service interruption into a financial incident.
This is an old distributed-systems problem in a newly conversational interface. The execution layer needs transaction identifiers, a way to query the authoritative state and an operation-specific retry policy. The model can help decide what to tell the customer, but it should not manufacture certainty because silence is uncomfortable. “The request is pending verification” can be a correct outcome.
Google’s SRE discussion of overload explains why clients and backends need to handle resource restrictions deliberately. Applied to agents, the lesson is to bound the work a task may generate. A single user request can become repeated searches, retries and tool calls. A limit on incoming chat messages is not necessarily a limit on the load imposed downstream.
Operational budgets should therefore follow the task. Define which errors can be retried, how often, with what delay and when the workflow must stop. Record the difference between an exhausted budget and a completed request. A service that avoids infinite work but silently drops the customer has solved only half the problem.
Grade the outcome, then inspect the route
A test that asks another model whether the final reply sounds helpful misses much of this workflow. The reply may be excellent while the refund is wrong. A test that checks only the payment may miss an unauthorized lookup or an email containing somebody else’s information. Outcome and process need different evidence.
Anthropic’s agent-evaluation guide discusses combining code-based, model-based and human graders. The useful distinction is between mechanically checkable state and judgments that need interpretation. A transaction amount can be checked directly. Whether an explanation is clear may require a rubric and human calibration. Neither should be allowed to conceal failure in the other.
For the illustrative refund workflow, a meaningful trial would vary policy versions, ambiguous customer requests, missing records, tool failures and adversarial notes. It would check that the right order was accessed, the permitted amount was processed at most once, and the customer received an accurate account of the outcome. It would also count correct refusals and appropriate escalations. Finishing every case is not the objective if some cases should not be completed automatically.
The test set needs separation from day-to-day development examples. Otherwise, a team can improve performance on familiar cases while learning little about unfamiliar ones. The evaluation should disclose its environment and scope: simulated payments are valuable for repeatability but do not prove that the live integration, credentials and monitoring are correctly configured.
There is an economic question inside the quality question. Count completed work after review and remediation, not only responses generated. A workflow that saves drafting time but requires an expert to reconstruct every transaction may move labor rather than remove it. At the same time, a narrowly scoped agent that handles common cases reliably can create real value even if it escalates unusual requests. Coverage and correctness should be reported separately.
One particularly revealing test changes the environment halfway through a task. A customer updates an address, an operator changes a policy, or another process resolves the ticket. The agent’s earlier observation is no longer the current state. A system that treats its own conversation history as authoritative can make a perfectly coherent decision about an obsolete situation. The execution layer needs a way to detect relevant changes before committing an action.
The same distinction applies to stopping work. Canceling a chat response does not necessarily cancel an operation already submitted to another service. The product should say what cancellation means, whether queued work can be withdrawn and whether an action has crossed a point that requires a compensating operation. A visible stop button is a useful control only when its relationship to downstream work is understood.
These cases expose why a transcript alone is a weak operating record. It describes what the agent saw and said at particular moments. The authoritative service state describes what persisted. Linking the two with stable task and transaction identifiers allows an investigation to distinguish stale information, an incorrect proposal, a denied operation and an execution failure. Without that distinction, every incident can be misdiagnosed as the model simply getting confused.
Monday morning is the release test
The strongest argument against this emphasis on controls is speed. Small teams can spend months building infrastructure around a capability that users do not want. That is a legitimate risk. The answer is to reduce the initial consequence and scope: draft rather than execute, use a test environment, limit the audience, and learn whether the task is valuable before expanding autonomy.
What does not follow is that a successful prototype can inherit production authority by default. Moving from suggestion to action changes the product. It creates obligations to identify actors, enforce permissions, observe state and recover when dependencies fail. Those costs belong in the decision to deploy, not in a later cleanup budget.
An effective first release may therefore look less impressive than its demo. It may handle fewer cases, expose a clear pending state and stop when policy is uncertain. That restraint is not a failure of ambition. It makes the operating envelope visible enough to improve with evidence. The next expansion can target a measured limitation rather than an optimistic assumption.
Before the hypothetical refund agent goes live, the decisive exercise is simple: ask an operator to reconstruct a difficult case from the service’s records without trusting the agent’s retrospective story. Can they establish what was requested, what was authorized, what actually happened and what remains unresolved? If they cannot, the green tick arrived too early.
Sources & context
This article draws on the sources linked in the text. Analysis and illustrative scenarios are identified separately from reported findings.
- August 2026 discussion of agent identity
- OAuth token-exchange standard
- OWASP prompt-injection guidance
- SRE discussion of overload
- agent-evaluation guide
