ProvenanceGuard: Checking That MCP Agents Cite the Right Source, Not Just the Right Fact

Multiverse Computing's verification layer keeps track of which MCP tool supports each claim. In a medical test it caught 138 of 139 expert-flagged claims and all 50 deliberate source swaps.

ProvenanceGuard: Checking That MCP Agents Cite the Right Source, Not Just the Right Fact

An MCP agent can call a search tool, read a patient or account record, query a database and combine the results into a single answer. This ability to work across multiple tools makes AI agents considerably more useful than systems that rely on a single knowledge source. An agent can gather information from different places, compare relevant details and construct an answer that would otherwise require a person to consult several systems manually.

But this workflow introduces a quieter failure mode than conventional hallucination: an agent can produce a statement that is factually correct while attributing it to the wrong source [1].

Consider an agent responding to a customer who wants to know whether a subscription includes a 30-day refund window. The agent checks an account record and a separate policy document. Its final answer says, “According to the account record, this plan includes a 30-day refund window.” The statement may be accurate, but the account record might contain only subscription details, while the refund condition actually comes from the policy document.

The problem is not necessarily the factual content. It is the relationship between the claim and its supporting evidence. Multiverse Computing’s ProvenanceGuard is designed to address this issue by tracking which MCP tool output supports each claim and checking whether that source matches the one identified by the answer. The team calls this failure mode cross-source conflation [1:1].

Why it matters

The reliability of an AI agent depends on more than its ability to retrieve relevant information or generate a coherent response. It also depends on whether the information can be traced back to the evidence that supports it.

This distinction is easy to overlook in retrieval-augmented generation (RAG) and multi-tool agent architectures. An agent may collect several documents, tool responses or database records and place them into a common context. Once the information has been assembled, a verification system can examine the answer against the combined evidence.

Most answer-checking approaches, including RAGAS faithfulness, MiniCheck and AlignScore, evaluate whether claims are supported by the available evidence after it has been pooled together. They do not establish which individual tool output supports a claim or whether that output matches the source identified in the answer [1:2].

Consider an agent with access to three sources:

  • A customer account record containing subscription details.
  • A policy document describing refund eligibility.
  • A support interaction containing information about an earlier customer complaint.

The agent retrieves all three and generates an answer about refund eligibility. A conventional pooled-evidence checker might find the 30-day refund condition in the policy document and conclude that the statement is supported. From a factual verification perspective, that conclusion is reasonable.

However, if the answer explicitly says that the condition comes from the account record, the attribution is incorrect. The account record may establish which subscription the customer has, but it does not establish the refund condition. The answer is therefore factually supported by the available evidence but incorrectly attributed to a particular source.

This creates two separate verification requirements:

  1. The factual claim: Does the available evidence support the statement?
  2. The source attribution: Does the particular source named or implied in the answer support that statement?

A verification process that checks only the first requirement can miss the second. A source can be relevant to the overall question without supporting every detail in the generated response.

The problem becomes more consequential in regulated environments. Consider a medical agent that accesses patient records, research articles and other information sources. A statement about a patient’s medical history may be present in the patient record, while a statement about a research finding may be present in a published article. Both can contribute to the same response, but they are not interchangeable.

If information from one patient’s history is presented as though it were a finding from medical literature, the answer becomes misleading even when the underlying detail is true. Similarly, attributing a research finding to an individual patient record changes the meaning of the evidence.

For data and AI engineering teams, this introduces a more demanding definition of grounding. A statement should not automatically be considered grounded merely because a matching fact appears somewhere in the agent’s context. Where the answer identifies a source, that source should be able to support the statement attributed to it [1:3].

This has practical implications for agent evaluation and release gates. A system that performs well on conventional factual-grounding checks may still produce misleading source attributions. If provenance matters to the application, it needs to be tested independently. Source-aware verification does not replace factual verification; it adds the requirement that the evidence must come from the source the answer claims supports it.

How it works

ProvenanceGuard operates after the agent has generated its response. Rather than changing the underlying agent or requiring additional model training, it examines the captured MCP execution trace, which contains the tool outputs together with their source IDs.

These identifiers allow the verification layer to retain the identity of each source instead of treating all retrieved information as one undifferentiated evidence collection. Without that association, distinguishing a correctly attributed statement from one that merely matches information somewhere in the combined context becomes difficult.

ProvenanceGuard maintains source identity through five steps [1:4].

1. Break the answer into specific claims.

The first step is to decompose the generated answer into individual claims that can be evaluated independently. An AI-generated response may contain several sentences, and each sentence may communicate more than one piece of information. Evaluating the entire response as a single block can make it difficult to identify precisely which part is supported and which part has an attribution problem.

For example, consider the statement:

“The customer’s account is on the premium plan, and according to the account record, this subscription includes a 30-day refund window.”

This contains at least two distinct assertions: the customer’s subscription type and the refund condition, including its claimed source. Breaking the response into individual claims allows the verification system to evaluate these statements separately. It can determine whether the subscription information is supported by the account record and whether the refund condition is actually supported by the source named in the answer.

This claim-level approach also accounts for the fact that a response does not have to be entirely correct or entirely incorrect. Some claims may be properly grounded, while others may be unsupported or attributed incorrectly.

2. Find the most relevant source for each claim.

Once the answer has been divided into individual claims, ProvenanceGuard identifies the most relevant source for each one. It uses the information retained in the MCP trace, considering individual tool outputs and their associated source identities rather than comparing the claim against a single pooled collection of evidence.

In a multi-tool environment, different sources may contain different parts of the information required to answer a question. One tool might return an account record, another a policy document and another relevant research material. The source-routing step attempts to establish which output is most relevant to each claim.

For example, an account record may be relevant when determining which subscription a customer has, while a policy document may be more relevant when determining the conditions attached to that subscription. The system must evaluate these claims individually rather than assuming that the source most relevant to the overall question supports every statement in the answer.

3. Check whether that source supports the claim.

After identifying the most relevant source, ProvenanceGuard evaluates whether the source actually supports the claim. This is the factual support check, but it is performed with source identity preserved.

Suppose an agent has retrieved two documents: one containing subscription information and another containing the refund policy. If the refund condition is supported by the policy document but not by the account record, the system should not treat the account record as valid evidence for that claim.

A source may discuss the same subject without supporting every detail in the generated response. The verification process therefore needs to establish a meaningful relationship between the individual claim and the contents of the selected source.

4. Compare that source with the one the answer names or implies.

This is the central provenance check. Even when the system identifies a source that genuinely supports the claim, it must compare that source with the attribution made by the generated answer.

The answer may explicitly identify a source using language such as “according to the patient record” or “the research article reports.” It may also imply a source through the way the statement is framed. ProvenanceGuard checks whether the source supporting the claim matches the source named or implied by the answer.

If the two do not match, the claim has an attribution problem even if the information itself appears somewhere in the evidence. A claim can be supported by the correct source, unsupported by the identified source, or supported by another source but incorrectly attributed.

The last category is particularly important because it represents the cross-source conflation problem that ProvenanceGuard is intended to detect.

5. Return a per-claim source verdict and an overall allow or block decision.

The final step converts the individual verification results into an output that can be used by the surrounding agent workflow.

Instead of returning only a single judgement about the entire answer, ProvenanceGuard produces a verdict for each claim. This identifies where a problem occurred and which source relationship needs attention. It also generates an overall allow or block decision.

For engineering teams, this creates a potential release-gate mechanism. An answer that satisfies the required provenance checks can proceed, while an answer containing claims that fail the configured verification policy can be blocked. The per-claim results also distinguish a response containing one problematic attribution from one in which several claims have failed verification.

The tested local architecture

In the evaluated configuration, the entire verification process ran locally using three model components:

  • MiniLM handled routing claims to relevant sources.
  • DeBERTa, configured for natural-language inference (NLI), performed support checks.
  • A local language model handled the decomposition of generated answers into individual claims.

These components divide the verification workload into distinct responsibilities. Claim decomposition converts a complete answer into smaller units that can be checked. Source routing determines which tool output is most relevant to each unit, while natural-language inference evaluates whether the selected source supports the claim.

The implementation also applies a stricter requirement to literal values. Numbers, dates and identifiers must appear in the source to pass verification [1:5].

This is useful because a claim can appear semantically plausible while containing a specific value that is not supported by the evidence. A statement might correctly describe a policy but give a different date or identifier from the source. A general semantic similarity check may not be sufficient to establish that such a detail is correct, so requiring the actual value to appear in the source introduces a more direct check.

The team makes an important distinction about the models used in the evaluation: MiniLM, DeBERTa and the local language model represent the tested configuration, not mandatory components of the approach. Other model combinations could be used, but they would need their own calibration [1:6]. The architectural requirement is to retain source identity and evaluate support and attribution separately.

What happens when an answer is blocked?

ProvenanceGuard includes the possibility of sending blocked answers through a RARR-style repair step. The repaired response is then checked again [1:7].

The purpose is to address problematic claims rather than simply allow the original response to pass. However, a repair process should not be treated as an automatic guarantee of correctness. A rewritten answer still needs to satisfy the same verification requirements; otherwise, the system risks replacing one attribution error with another.

The repeated verification step ensures that the final decision remains tied to the provenance checks. The repair mechanism can attempt to produce a safer response, but the rewritten output must still pass verification.

What happened

The team evaluated ProvenanceGuard using 281 real traces from a medical agent that interacted with patient records, research articles and other tools [1:8].

The evaluation covered several aspects of the verification problem: expert-reviewed claims, comparisons with existing checkers, similar-source discrimination, deliberate attribution changes and the repair workflow. Together, these tests provide a more detailed view of how the system handles source attribution than a single aggregate accuracy figure.

Expert-reviewed claims

The first evaluation examined claims extracted from 40 held-out answers. These answers contained 361 claims in total, of which experts identified 139 that should not have passed verification.

ProvenanceGuard caught 138 of those claims and allowed one to pass. However, the system also held 67 claims that experts considered supported [1:9].

The second figure is important when interpreting the results. A verification layer that blocks questionable statements but also rejects many legitimate claims can create operational difficulties. Excessive blocking can interrupt otherwise useful agent workflows and introduce additional review requirements for engineering or operational teams.

The reported behaviour reflects a deliberately cautious configuration. ProvenanceGuard prioritised avoiding unsupported or incorrectly attributed claims, even though that meant holding some claims that experts considered supported.

The team additionally reported that it identified the correct source approximately 86% of the time where one was identifiable [1:10]. This provides another dimension to the evaluation: detecting whether a claim should be blocked and identifying the precise source supporting it are related but distinct tasks. A system may recognise a provenance problem without necessarily identifying the exact correct source every time.

Against other checkers

The evaluation compared ProvenanceGuard against four other systems, making five systems in total.

On the paper’s measure of blocking the right claims while avoiding unnecessary blocks, ProvenanceGuard achieved the highest score among the evaluated systems. The team also highlighted that it was the only system in the comparison to record which source supported each claim [1:11].

This distinction goes beyond the comparison score. A conventional answer checker may provide a judgement about whether a response is supported by available evidence, but it does not necessarily produce the source-level information required to establish whether the answer’s attribution is correct.

ProvenanceGuard adds that information to the verification result. Instead of knowing only that a response has failed a grounding check, an engineering team can inspect the relationship between the claim and the source that supposedly supports it.

Similar sources

One of the more revealing tests involved a harder scenario in which several sources were similar. This is a challenging situation because multiple tool outputs may contain related or overlapping information.

For example, a subscription record and a policy document might discuss the same subscription. Both are relevant to the customer’s question, but only one may contain the specific condition being described. Broad semantic similarity alone cannot establish which source actually supports the claim.

In this harder evaluation, ProvenanceGuard achieved an F1 score of 0.846 for identifying claims that should be blocked. However, it correctly identified the exact source for only 50.3% of claims [1:12].

The difference between these results is significant. The system demonstrated a strong ability to identify problematic claims according to the reported blocking measure, but its ability to select the exact source was considerably weaker when the evidence contained several similar sources.

This suggests that source-level discrimination remains a difficult part of the problem. A system may recognise that a response contains a claim that should not pass while still struggling to distinguish between multiple plausible supporting sources.

For teams evaluating similar systems, this is why blocking performance and exact-source identification should be assessed independently.

Deliberate source swaps

The deliberate source-swap test provides another perspective. In this experiment, the team changed the source named by the answer in 50 cases while leaving the underlying evidence intact.

ProvenanceGuard caught all 50 cases [1:13].

This test isolates the attribution problem. The factual evidence remained available and the content supporting the claim did not change; only the source attribution was altered. A verifier that checks the answer against pooled evidence could potentially miss this change because the claim would still be supported somewhere in the available evidence.

The result demonstrates that ProvenanceGuard’s source-aware checks detected every deliberate swap in the evaluated set. It also suggests a useful testing pattern for engineering teams: verification systems should be tested against responses in which the fact remains correct but its source is deliberately changed, not just against fabricated facts or unsupported claims.

The result is encouraging, although it should be interpreted within the scope of the reported experiment rather than as a guarantee that every possible attribution error will be detected.

Repair and speed

The evaluation also examined the repair workflow and the additional processing time introduced by verification.

Across the full run, the repair loop resolved all 173 blocked answers. However, 144 of those cases ended with safe fallback text rather than a rewritten answer [1:14].

A repair workflow does not necessarily have to produce a polished replacement for every blocked response. If the system cannot establish a sufficiently reliable answer after the repair attempt, a safe fallback can prevent the original questionable response from reaching the user. That may be preferable to allowing an uncertain rewrite to pass without adequate verification.

The reported results suggest that the repair process handled all 173 blocked answers, but most relied on fallback text rather than rewritten responses. For teams building production agents, this creates a practical trade-off: stricter verification may improve reliability while increasing the number of cases in which the system declines to provide a detailed answer.

The additional processing overhead was approximately half a second per answer [1:15]. This provides a reference point for teams considering the impact of adding source-aware verification to an existing agent workflow, although actual performance would depend on the models, hardware and configuration used.

Early integration

The approach has also moved beyond the evaluation described by the team. NVIDIA’s NVFlow has already merged an optional grounding check for its finance agent based on this approach [1:16].

This demonstrates an integration of the underlying verification idea into another agent-oriented workflow. The optional nature of the check also reflects an important deployment consideration: verification can be incorporated as a distinct layer without necessarily requiring every application to follow the same enforcement strategy.

What data and AI engineers should do

ProvenanceGuard offers an architectural lesson for data engineers, ML engineers and teams building agentic applications: source attribution should be treated as part of the data and evidence pipeline, not merely as a formatting requirement in the final response.

Log tool outputs with source IDs.

Source-aware verification is only possible when the agent’s execution trace preserves the identity of the information returned by each tool. If tool outputs are combined into one context without retaining their origins, a downstream verification layer has less information available to distinguish between different sources.

For example, an agent may retrieve an account record, a policy document and a research article. If the system stores only the combined text, determining which individual output supports a particular claim becomes difficult. Retaining source IDs allows the verification process to map claims back to the corresponding tool outputs.

This should be considered when designing trace capture and logging. Source identity is not simply metadata for debugging; it can become an important part of the evidence required to validate an agent’s response.

Define “grounded” for your release gate.

Engineering teams should explicitly decide what they mean when they classify an answer as grounded. There are two different standards:

  • The claim is supported by any available tool output.
  • The claim is supported by the specific source named or implied in the answer.

These standards are not equivalent. A system that accepts the first may allow a response to pass even when its source attribution is incorrect. A system that requires the second introduces a stricter condition.

The appropriate policy depends on the application, but the distinction should be deliberate. If an agent operates in an environment where the origin of information matters, source attribution should be included in the verification criteria rather than left to the generation model.

Check literal values strictly.

Numbers, dates and identifiers deserve particular attention. These details can appear in otherwise plausible statements, and their correctness should not be inferred simply because the surrounding explanation makes sense.

The evaluated ProvenanceGuard configuration requires literal values to appear in the supporting source [1:17]. Engineering teams can use this as a model for designing stricter checks around information that needs exact correspondence with the underlying evidence. A date appearing in an answer, for example, should be checked against the source rather than accepted because it appears consistent with the broader narrative. The same principle applies to identifiers and numerical details.

Decide what happens to blocked answers.

A verification layer needs an explicit failure-handling policy. Teams should decide whether a failed response should be repaired, sent for review or replaced with a safe fallback.

The ProvenanceGuard evaluation illustrates why this matters. Although the repair loop resolved all 173 blocked answers, 144 ended in fallback text rather than a rewrite [1:18]. A fallback is not necessarily a failure of the verification system. In applications where incorrect attribution can have serious consequences, declining to provide an uncertain answer may be preferable to producing a replacement that has not been adequately checked.

However, teams should track how frequently fallback behaviour occurs. A high fallback rate can affect the usefulness of the agent even when the verification layer successfully prevents questionable responses from passing. Monitoring blocked claims, repaired answers and fallback responses helps engineers understand the operational consequences of their chosen verification policy.

Limitations and open questions

The reported results are promising, but ProvenanceGuard should not be interpreted as a universal solution to every grounding or attribution problem.

The evaluation was conducted in one medical domain and was based on the authors’ own testing. It demonstrates how the approach performed in that particular environment, but does not establish that the same performance will automatically transfer to every type of MCP agent or enterprise workflow. Different applications may expose different source structures, levels of evidence overlap and attribution requirements. Model configurations and verification thresholds may therefore need calibration for each environment.

The conservative behaviour reported in the evaluation is another consideration. ProvenanceGuard caught 138 of the 139 claims that experts identified as claims that should not pass, but it also held 67 claims that experts considered supported [1:19]. For applications where preventing misleading statements is more important than maximising the number of responses delivered, this cautious behaviour may be appropriate. Where blocking legitimate responses creates significant operational friction, teams will need to examine the balance between verification strictness and unnecessary rejection.

The similar-source evaluation also exposes an important technical limitation. Although the system achieved an F1 score of 0.846 on identifying claims to block in the harder test, its exact-source accuracy was only 50.3% [1:20]. This indicates that distinguishing between several similar sources remains challenging. The ability to detect a potentially problematic claim does not necessarily mean the system can reliably determine which of several closely related sources actually supports it.

There is also an unresolved comparison involving the cost and effectiveness of a simpler LLM-based approach.

In discussion on the original post, the authors acknowledged that they had not compared ProvenanceGuard against a cheaper LLM judge prompted to perform the same checks. They argued that a second LLM could introduce its own source confusion, but the trade-off between accuracy, latency and cost remains unmeasured [1:21].

This is an important open question for engineering teams. A dedicated verification pipeline introduces additional components and processing requirements. A simpler LLM judge might be easier to integrate into an existing workflow, but its ability to preserve and evaluate source identity would need to be tested.

The available evaluation does not establish whether ProvenanceGuard is consistently more accurate, more efficient or more cost-effective than such an alternative. A meaningful comparison would need to examine more than whether both approaches identify unsupported claims. It should also consider whether they detect deliberate source swaps, distinguish similar sources, avoid unnecessary blocks and maintain acceptable processing overhead.

Teams should therefore evaluate source-aware verification against the actual failure modes of their applications. A system that performs well on conventional factual-grounding benchmarks may still be insufficient when precise attribution matters. Equally, a system that performs well in one medical evaluation may need additional testing before it can be trusted in another domain.

ProvenanceGuard makes the case for treating source identity as an independent part of agent reliability, while its limitations show that accurate source discrimination remains an active engineering challenge.

Key takeaways

  • Cross-source conflation occurs when an agent produces a true claim but attributes it to the wrong source. Conventional pooled-evidence checkers can miss this because they evaluate whether information appears somewhere in the combined evidence rather than whether the named source supports it. [1:22]
  • ProvenanceGuard preserves source identity through claim decomposition, source routing, support verification and attribution checking. It operates on captured MCP traces after the agent generates its answer, without requiring retraining. [1:23]
  • In the medical-agent evaluation, ProvenanceGuard caught 138 of 139 expert-flagged claims, while also holding 67 claims that experts considered supported. It identified the correct source approximately 86% of the time where one was identifiable. [1:24]
  • The system caught all 50 deliberate source swaps, demonstrating the value of testing attribution errors separately from conventional factual-grounding failures. [1:25]
  • Distinguishing between similar sources remains difficult. In the harder evaluation, the system achieved 0.846 F1 on identifying claims to block but only 50.3% exact-source accuracy. [1:26]
  • For data and AI engineers, retaining tool-output source IDs, defining explicit grounding requirements, checking literal values and designing safe fallback behaviour are important steps towards more reliable multi-tool agents. [1:27]

Sources


  1. Hugging Face Blog, “Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents”, 2026-09-29. https://huggingface.co/blog/MultiverseComputingCAI/getting-the-source-right-not-just-the-fact-source ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎

Discussion

Join the discussion: create a free account or sign in to comment.

No comments yet.

Keep reading

TorchTPU: PyTorch Runs Natively on Google TPUs, and Hugging Face Models Come Along
AI Engineering

TorchTPU: PyTorch Runs Natively on Google TPUs, and Hugging Face Models Come Along

Google and Meta built TorchTPU, a native PyTorch backend for TPUs with eager execution and torch.compile. Hugging Face Transformers, Diffusers and Kernels now treat "tpu" as just another device.

Nov 15, 2026 · 5 min read
Helion and NVIDIA CompileIQ: Autotuning GPU Kernels Down to the Compiler
AI Engineering

Helion and NVIDIA CompileIQ: Autotuning GPU Kernels Down to the Compiler

Kernel autotuners stop at tile sizes and warp counts. Helion can now search ptxas compiler controls in the same loop, using CompileIQ's Advanced Control Files, for 2 to 7% more speed on kernels already tuned.

Nov 19, 2026 · 4 min read
TorchTitan Update: New Frontier Models, Faster MoE Training and TitanRL for Reinforcement Learning
AI Engineering

TorchTitan Update: New Frontier Models, Faster MoE Training and TitanRL for Reinforcement Learning

PyTorch's TorchTitan adds Qwen3.5, Kimi K3 and DeepSeek-V4, an RL stack whose trainer and generator can produce bitwise-identical log-probabilities, and FlexShard, which cut Muon optimizer time by up to 79%.

Nov 18, 2026 · 4 min read