Evaluating Multi-Agent Systems With Amazon Bedrock AgentCore: Helpfulness, Business Accuracy and Explainability

AWS shows a three-layer way to test agents on AgentCore. Built-in evaluators set a baseline, custom evaluators check business rules, and separate explainability evaluators check whether agents can say why.

Evaluating Multi-Agent Systems With Amazon Bedrock AgentCore: Helpfulness, Business Accuracy and Explainability

A fluent answer is not the same as a correct decision. When an orchestrator agent delegates to sub-agents that call tools, query databases and weigh business constraints, quality depends on whether it picked the right tools, followed the right workflow and respected the rules, not just on how the final response reads. AWS has published a reference implementation that tests exactly that, using Amazon Bedrock AgentCore Evaluations on a multi-agent supply-chain assistant [1]. Its most useful idea is treating explainability as a separate thing to measure.

Why it matters

Most teams evaluate agents the way they evaluate chatbots: is the answer helpful and relevant? AWS argues that this is not enough for agentic systems, where correctness depends on tool selection, workflow execution and business constraints [1:1]. Separating accuracy from explainability also gives sharper feedback. An agent can be right but unable to explain itself, or articulate but wrong, and each needs a different fix [1:2]. For teams that need sign-off from risk or compliance before agents touch real decisions, that distinction is what makes an evaluation report actionable.

The setup

The example is a fictional retailer, AnyCompany Retail, that wants help with inventory allocation, distribution and routing. It is built with the Strands Agents SDK: an orchestrator plus four sub-agents for optimization, distribution, routing and analytics. Each runs on AgentCore Runtime with AgentCore Memory and Observability turned on, and the sub-agents call MCP tools backed by mock APIs [1:3].

AgentCore Runtime is the hosting layer. It works with LangGraph, Strands, CrewAI or custom agents, and each user session runs in its own microVM with isolated CPU, memory and filesystem, which is terminated and sanitized when the session ends. Sessions can last up to 8 hours on microVMs or 14 days on EC2-based Instances [2].

How the three layers work

Layer 1: built-in evaluators, no setup. Every agent is scored on Helpfulness, plus one built-in evaluator aimed at its most likely failure: Tool Selection Accuracy for the orchestrator, Response Relevance for optimization and distribution, Instruction Following for routing and Faithfulness for analytics [1:4].

Layer 2: custom evaluators for business accuracy. Each agent gets checks that encode domain rules [1:5]:

  • Orchestrator: plan coherence (are sub-agent outputs combined without contradiction?) and tool trajectory (did it route to the right sub-agent?).
  • Optimization: constraint satisfaction (budget, inventory between demand and twice demand, warehouse capacity) and KPI attainment.
  • Distribution: groundedness in current inventory and demand data, and whether risk of stockouts or overstock improves.
  • Routing: route feasibility (delivery window, cost, carrier capacity, region) and service level.
  • Analytics: SQL correctness, data grounding in the database results and no unsupported claims.

Layer 3: explainability, measured on its own. Six cross-cutting evaluators ask whether the agent explained why it made the recommendation, cited the data or tool output it used, named the constraints that shaped the answer, explained trade-offs between cost, service level and inventory risk, said why each sub-agent or tool was called, and stated its assumptions when data was incomplete [1:6].

Running it

AgentCore Evaluations has two modes. On-demand is for development benchmarking, regression tests and CI/CD gates. Online reuses the same evaluators on a sample of production traces, for example 1 to 10%, and streams scores to CloudWatch dashboards and alarms [1:7].

In the walkthrough, a test client sends 20 sample queries, five per sub-agent, as multi-turn sessions. You then register the custom evaluators and run them against a session ID; the API returns immediately and writes results to S3. Traces take 3 to 5 minutes to reach CloudWatch, so wait before evaluating [1:8].

AWS also notes that evaluations judge quality after the fact. Amazon Bedrock Guardrails, with content filters, denied topics and grounding checks, is the control that enforces safety while the agent runs [1:9].

What data and AI engineers should do

  • Score each agent on its own failure mode, not just the final answer. An orchestrator that routes to the wrong sub-agent can still produce a pleasant response.
  • Write business rules as evaluators. If a recommendation must fit a budget or a delivery window, check it in code or a rubric, every run.
  • Keep explainability separate from accuracy. High accuracy with low explainability means better prompting for reasoning; the reverse means fixing decision logic.
  • Use the same evaluators in CI and in production, gating releases on-demand and sampling live traffic online.

Limitations and open questions

This is a demo on a fictional company, with mock APIs and 20 test queries. The post shows how to run the evaluators but publishes no scores, so it says nothing about how reliable the LLM-based judges are, how often they disagree with human reviewers, or what thresholds make a good release gate. Teams will need a larger, labeled test set of their own before trusting these scores to block a deployment.

Key takeaways

  • AWS’s reference design evaluates agents in three layers: built-in quality, custom business rules and explainability. [1:10]
  • Explainability is scored separately, so “right but unexplained” and “explained but wrong” show up as different failures. [1:11]
  • The same evaluators run on-demand for CI/CD and online on a 1 to 10% sample of production traces. [1:12]
  • AgentCore Runtime isolates each session in its own microVM, sanitized when the session ends. [2:1]

Sources


  1. AWS Machine Learning Blog, “Evaluating multi-agent systems for explainability and helpfulness with Amazon Bedrock AgentCore”, 2026-10-05. https://aws.amazon.com/blogs/machine-learning/evaluating-multi-agent-systems-for-explainability-and-helpfulness-with-amazon-bedrock-agentcore/ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎

  2. AWS Documentation, “Host agent or tools with Amazon Bedrock AgentCore Runtime”. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agents-tools-runtime.html ↩︎ ↩︎

Discussion

Join the discussion: create a free account or sign in to comment.

No comments yet.

Keep reading

OpenAI Dots: A Persistent Cloud Agent for 4,000+ Apps
AI Engineering

OpenAI Dots: A Persistent Cloud Agent for 4,000+ Apps

Dots gives Pro and Business Premium users a dedicated cloud computer powered by GPT-6 Astra, but its high default cost makes the cheaper GPT-6.1 Sol a critical consideration for engineers.

Oct 26, 2026 · 4 min read
GPT-6.1 Sol on Amazon Bedrock: Near-Astra performance at a fraction of the cost
AI Engineering

GPT-6.1 Sol on Amazon Bedrock: Near-Astra performance at a fraction of the cost

OpenAI's GPT-6.1 Sol brings near-Astra performance to coding, computer use and professional work at one-fifth of Astra's standard token prices. Here's what Amazon Bedrock adds, from pricing and a 1M-token context to endpoints and code.

Oct 06, 2026 · 13 min read
Grok 4.7 on Amazon Bedrock: Routing, reasoning effort, and cost with OpenAI-compatible APIs
AI Engineering

Grok 4.7 on Amazon Bedrock: Routing, reasoning effort, and cost with OpenAI-compatible APIs

xAI's Grok 4.7 is now on Amazon Bedrock with a 500K-token context window. Here's how reasoning effort, Geo vs Global routing and service tiers change cost and latency, with OpenAI-compatible code you can reuse.

Oct 05, 2026 · 9 min read