Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents

Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments, where they operate over untrusted web content and execute actions with direct consequences. This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour. Existing security benchmarks adopt an \textit{attack-centric} perspective, focusing on the technical feasibility of injections while overlooking the nuanced distribution of resulting harms. In practice, however, prompt-injection risk is victim-dependent: a single exploit can produce asymmetric consequences for different stakeholders, and the same attack pattern may exhibit substantially different effectiveness depending on whom it targets. To capture these properties, we introduce \textbf{\sysname}, a \textit{stakeholder-centric} benchmark to systematically categorize and attribute harm in real-world web agent systems. It distinguishes between affected entities (e.g., user, seller, platform), decomposes the attacks into concrete objectives, and evaluates each case with complementary outcome- and process-level metrics. Our results reveal substantial and heterogeneous vulnerabilities: not a single attack objective is reliably resisted by current agents, and failures distribute across qualitatively distinct modes ranging from \emph{stealthy parasitism} (attack succeeds without disrupting the user's delegated task) to \emph{misaligned disruption} (task disrupted without attack success) and \emph{compounded failure} (both adversarial objective and task integrity simultaneously violated). These patterns are missed by conventional evaluation, highlighting the need for stakeholder-aware assessment of LLM-based agents in real-world deployments. Benchmark is available at https://github.com/StakeBench/SBC.

Paper

Similar papers

Agents reading web pages, emails and repository files are hijacked by prompt injections hidden in untrusted content, yet 2026 benchmarks show deployed guards over-block benign text, making them unusable inline. A naive single prompt asking a base model whether text is an injection fails because without the agent's declared task it cannot separate legitimate imperative content, such as a README's install instructions, from instructions subverting the agent, returning a bare score with no offending span or operating point. Evaluate a task-conditioned LLM judge seeing the declared task and permitted capabilities alongside the untrusted output, using constrained structured decoding to emit is_injection, targeted_capability, quoted span and confidence, retrieving injection patterns for few-shot grounding, and applying a threshold calibrated on a benign corpus with self-consistency voting. Measure ROC-AUC, recall at sub-one-percent false-positive rate, span-localization precision and Brier calibration on AgentDojo-style labeled attack traces plus held-out benign corpora, beating a naive single-prompt base-model classifier and a Llama-Guard-style guardrail. Deliver as the prototype a minimal runnable Python MCP server (stdio) exposing the priced AI tool screen_tool_output that invokes a language or ML model to produce its output, with a typed input/output schema, an x402-style pay-per-call metering stub that records a per-call price in USDT and emits a settlement receipt, and one smoke test that exercises the tool end to end.Dogukan Ali Gundogan2026 · Zenodo (CERN European Organization for Nuclear Research) In Library
Agents reading web pages, emails and repository files are hijacked by prompt injections hidden in untrusted content, yet 2026 benchmarks show deployed guards over-block benign text, making them unusable inline. A naive single prompt asking a base model whether text is an injection fails because without the agent's declared task it cannot separate legitimate imperative content, such as a README's install instructions, from instructions subverting the agent, returning a bare score with no offending span or operating point. Evaluate a task-conditioned LLM judge seeing the declared task and permitted capabilities alongside the untrusted output, using constrained structured decoding to emit is_injection, targeted_capability, quoted span and confidence, retrieving injection patterns for few-shot grounding, and applying a threshold calibrated on a benign corpus with self-consistency voting. Measure ROC-AUC, recall at sub-one-percent false-positive rate, span-localization precision and Brier calibration on AgentDojo-style labeled attack traces plus held-out benign corpora, beating a naive single-prompt base-model classifier and a Llama-Guard-style guardrail. Deliver as the prototype a minimal runnable Python MCP server (stdio) exposing the priced AI tool screen_tool_output that invokes a language or ML model to produce its output, with a typed input/output schema, an x402-style pay-per-call metering stub that records a per-call price in USDT and emits a settlement receipt, and one smoke test that exercises the tool end to end.Dogukan Ali Gundogan2026 · Zenodo (CERN European Organization for Nuclear Research) In Library
© 2026 NYSGPT2525 LLC