AI agents and legacy systems: useful, bounded, accountable

An agent changes the risk model

Traditional integration executes a path that engineers designed in advance. An agent selects actions at runtime from a goal, available tools, retrieved context, and previous results. That flexibility is valuable for investigation, support, reconciliation, and operational assistance. It also means the exact path is not fully known before execution.

The model should never be the final authorization layer. Its output may be wrong, ambiguous, manipulated by untrusted content, or inappropriate for the current business state. The architecture must assume that a plausible-looking plan can still be unsafe.

Start with a narrow business outcome such as explaining a failed order, preparing a reconciliation report, or drafting a service request. “Operate the legacy system” is not a bounded use case.

A reference architecture for controlled automation

User or scheduled task
  -> authenticated agent session
  -> planner and policy-aware orchestrator
  -> tool gateway
       -> authorization and approval service
       -> schema and business-rule validation
       -> rate, cost, and time limits
       -> idempotency and audit ledger
  -> anti-corruption adapters
  -> legacy APIs, queues, files, and applications

All stages -> traces, evaluations, alerts, and incident controls

The agent sees purpose-built tools, not raw infrastructure. A tool such as prepare_refund expresses business intent more safely than generic SQL, shell, or browser access. The gateway authenticates every invocation independently and enforces policy outside the model. Adapters translate the approved operation into legacy protocols without exposing their quirks to the reasoning layer.

Separate planning from execution. The planning stage can gather evidence and produce a structured proposal. The execution stage accepts only a validated command with a defined schema, known actor, explicit scope, current approval, and idempotency key.

Build trust boundaries around every input and tool

Treat retrieved content as untrusted data

Emails, tickets, documents, web pages, database notes, and tool responses can contain instructions intended to redirect the agent. Label provenance, isolate content from system instructions, limit what each retrieval source can influence, and never let retrieved text silently expand permissions.

Grant the smallest useful capability

Create task-specific tools with narrow schemas. Separate read, prepare, approve, and execute capabilities. Issue short-lived credentials for one user, tenant, environment, purpose, and time window. Do not pass broad user tokens through intermediaries or allow an agent to discover production credentials.

Validate outputs as hostile input

Use strict structured outputs, allowlists, type and range checks, canonical identifiers, state preconditions, and server-side business rules. Never concatenate model output into SQL, shell commands, URLs, file paths, or legacy terminal input.

Constrain the run

Set limits for turns, tool calls, elapsed time, tokens, cost, retries, records, and transaction value. Prevent loops by recording completed actions and rejecting duplicate or conflicting work. A kill switch must revoke tool access without waiting for the agent to cooperate.

Make approval a trusted transaction

Human review should match consequence, not merely exist. Read-only searches may run automatically. Drafts can require lightweight review. Financial, customer-visible, privileged, destructive, or irreversible actions need explicit approval by an authorized person.

The approval interface must be generated from trusted structured data, not free-form model text. Show the exact action, target, before-and-after values, evidence, policy result, side effects, expiration, and a clear reason. Bind approval cryptographically or transactionally to that immutable action so the agent cannot change parameters afterward.

• Two-person control: use it for high-value or high-impact actions.
• Separation of duties: the requester, approver, and production executor should not silently collapse into one identity.
• Fresh authorization: recheck permissions and current system state at execution time.
• Expiration: approvals should become invalid when context or business state may have changed.

Do not ask a user to approve hundreds of opaque actions. Approval fatigue turns a control into ceremony. Aggregate only when every included action is visible, bounded, and governed by the same policy.

Test software deterministically and behavior statistically

Keep normal engineering tests. Unit tests verify policy, validation, adapters, idempotency, and authorization. Contract and integration tests verify tool schemas and legacy connections. End-to-end tests prove approval binding, execution, audit, rollback, and recovery.

Add evaluations for the behavior that conventional assertions cannot fully capture:

1. Task success: did the agent reach the correct outcome with supported evidence?
2. Tool selection: did it choose only necessary tools and valid parameters?
3. Policy compliance: did it stop, refuse, or request approval at the right boundary?
4. Adversarial resilience: did direct and indirect prompt injection fail to expand its authority?
5. Recovery: did timeouts, stale state, duplicate messages, and partial failures remain safe?
6. Efficiency: were latency, tokens, calls, and cost within budgets?

Version prompts, models, tools, policies, retrieval indexes, and evaluation datasets. Run a representative regression suite before any of them changes. Use repeated trials where model variance matters, grade critical cases with deterministic checks, and have qualified humans review sampled traces and disputed results.

Production feedback should become new evaluation cases, especially near misses. A high average score must not hide a critical failure category. Set separate release gates for unauthorized actions, data exposure, incorrect approvals, and other intolerable outcomes.

Operate the whole decision chain

Record the user, session, model and version, trusted instructions, retrieved-source identifiers, proposed plan, tool calls, policy decisions, approvals, legacy results, and final outcome. Protect sensitive content, define retention, and make the audit trail tamper-evident where the risk requires it.

Dashboards should cover business completion, refusal and escalation rates, tool errors, policy denials, approval latency, repeated calls, token and cost budgets, legacy latency, and evaluation drift. Trace identifiers must cross the agent, gateway, adapter, and legacy transaction.

Prepare incident controls before production: pause new runs, revoke credentials, disable individual tools, quarantine pending work, cancel where possible, reconcile completed actions, notify owners, and replay safely after repair. Keep a non-agent path for critical operations.

A maturity model that earns autonomy

1. Explain: read approved data and answer with citations. No side effects.
2. Draft: prepare a plan, message, report, or structured command for human review.
3. Execute with approval: perform a narrowly scoped, reversible action after trusted confirmation.
4. Execute within policy: automate low-consequence actions inside strict value, volume, and time limits, with sampling and immediate revocation.
5. Coordinate bounded workflows: handle several steps only after each tool, transition, and failure mode has earned evidence.

Advance one use case at a time. More autonomy is justified by measured reliability, effective containment, stable business rules, successful adversarial testing, and demonstrated recovery. It is not justified by a more capable model alone.

Production readiness checklist

• The business outcome, prohibited actions, and risk owner are explicit.
• The agent has task-specific tools instead of raw database, shell, or broad browser access.
• Every tool call is authenticated, authorized, validated, bounded, idempotent, and audited.
• Untrusted content cannot change policy, permissions, or approval requirements.
• High-consequence approvals show trusted parameters and bind to an immutable action.
• Secrets are short-lived, scoped, and unavailable to the model context.
• Unit, contract, integration, end-to-end, evaluation, and adversarial suites gate releases.
• Critical failure categories have individual thresholds, not only average scores.
• Limits exist for time, cost, turns, calls, records, retries, and transaction value.
• Operators can disable tools, revoke access, reconcile actions, and use a safe fallback path.

The goal is not an agent that can do everything. It is an architecture in which the agent can be genuinely helpful while the surrounding system remains responsible for authority, correctness, and recovery.