Benchmarking Regulatory Reasoning

01Introduction

The first paper in this series, What a Regulatory Design Agent Must Do, argued that regulation-driven development is not a document-search problem. It is the ongoing translation of interconnected requirements into engineering decisions, and an agent that supports it has to do more than find and quote the right clause.

That argument raises an obvious question: how would you know whether an agent actually does this reliably? General-purpose language models can produce fluent, well-structured answers to regulatory questions. Fluency is not the same as precision. An answer can cite the right standard and still apply the wrong rule, miss a conditional exception, or treat a requirement the source deliberately leaves open as if it were settled.

This paper describes how we measure regulatory reasoning. We introduce an internal benchmark of 110 engineering questions across 11 regulatory and technical documents, explain how answers are scored, and report how Eureka, our regulatory assistant at Frontier Instruments, performs relative to general-purpose agents given access to the same documents. We close with what the results do and do not establish.

02Background: what is being measured

Part 1 set out five linked capabilities that a regulatory design agent must have, summarized in Figure 1. The agent must understand the engineering context, identify the applicable requirements even when the engineer does not name them, interpret and reconcile those requirements, translate them into design decisions, and produce an answer that is defensible because its sources and qualifications are visible.

These stages reinforce one another. An omitted provision can invalidate a correct calculation. An accurate quote can still lead to a wrong decision if its conditions are misunderstood. A technically plausible answer is not decision-grade unless its rationale is clear.

A benchmark for this work therefore has to test the chain, not the links in isolation. Three design consequences follow. The questions must be posed the way engineers pose them, in product language, without naming the governing standard, so that identifying what applies is part of the task. The answers must be scored on the specific obligations, values, and qualifications a correct response contains, not on overall plausibility. And the set must include questions whose premises are misleading or whose answers are deliberately absent from the source, so that staying grounded is rewarded over sounding complete.

Part 1 worked one such question in full: the acceptance limits for a variable-dose injection pen, where the correct answer depends on the dialling resolution and on a transition between absolute and relative limits that a uniform percentage rule misses. The benchmark below is built from 110 questions of that kind.

03Benchmark design

To test whether agents can perform this work reliably, we built an internal benchmark of 110 engineering questions across 11 regulatory and technical documents used in medical device development. The set covers topics such as risk management, medical device software lifecycle processes, and combination products.

Each document contributes ten questions: five medium-complexity and five hard-complexity cases. Medium questions typically combine two or three related requirements from a single document. Hard questions require synthesis across distant clauses or annexes, conditional rules, table-derived values, or recognition that the source deliberately leaves an issue undefined.

The questions are written in everyday engineering language, without naming the relevant standard or clause. This means that an agent must first identify what applies before it can answer.

04How we score answers

Each question has four to seven binary rubric items, for a total of 737 evaluation criteria across the benchmark. An item might test whether the answer gives the correct small-dose tolerance; another checks that it does not apply a uniform percentage where an absolute criterion is required.

We use GPT-5 as the rubric judge. For each question, the fraction of satisfied items gives its score, and the overall benchmark score is the average across the 110 questions. The benchmark deliberately includes misleading premises and requests for details absent from the reference material, testing whether agents stay grounded rather than simply produce plausible-sounding answers.

05How Eureka is built

Eureka’s regulatory document assistant has a two-tier architecture. An orchestrator identifies the relevant standards and delegates analysis to a specialized reader subagent. The reader uses a domain-specific document tool to search an individual standard and return pertinent provisions and findings.

Separating document selection from detailed extraction lets us improve both sides of the workflow: what information gets surfaced and how the agent uses it to formulate an answer. It also helps distinguish a missed source from a reasoning failure.

A hard example: when every risk is green

A benchmark question asks what remains before product release when every line of a device team’s safety matrix is green and the design review is signed.

The critical issue is that acceptable individual residual risks do not automatically establish the acceptability of overall residual risk. The benchmark’s reference answer requires a separate overall evaluation in relation to the benefits of intended use, disclosure of significant residual risks where applicable, and a pre-release review by someone with the authority designated in the risk management plan. The outcome must be captured in the risk management report.

An agent that says only “the risks are controlled, so release is acceptable” misses the obligations that make the answer operationally useful.

06Results: accuracy and cost

We ran the same 110 questions through Eureka and compared its results with general-purpose agents. Codex and Claude configurations were given the same 11 documents as Markdown files, read-only access, and instructions to consult and cite those files before answering. Figures 2 and 3 and Table 1 report the results of these benchmark runs; they are not an independent third-party evaluation.

Cost notes: Eureka and Codex costs were computed from recorded tokens at list prices, without caching discounts; Claude costs came from SDK-reported billing. Reported total run costs were $25.47, $91.55, $18.66, $103.68, and $32.78, respectively, in the table’s order. Cost methods are not fully identical across providers.

Three findings

Eureka high leads on accuracy. It scores 96.2% overall, including 95.8% on hard questions. Its reported rubric error rate is 3.8%, versus 5.7% for Claude Fable, 8.3% for Claude Opus, and 10.3% for Codex. The hardest cases are the ones most likely to require chained interpretation rather than isolated retrieval.

Specialization improves efficiency in this comparison. At $0.232 per question, Eureka high costs approximately 72% less than Claude Fable, 75% less than Claude Opus, and 22% less than Codex. Eureka medium reaches 94.2% overall at $0.170 per question.

The hard subset is especially revealing. Eureka high scores 95.8% on hard questions, compared with 93.8% for Claude Fable, 89.7% for Claude Opus, and 88.1% for Codex. These are the questions for which missing a distant clause or a conditional exception can change the engineering recommendation.

What the results do and do not establish

The reported results support a narrower conclusion than a claim of general regulatory reliability. This evaluation involves 11 documents, 110 questions, a specific judge, and particular agent configurations. It does not by itself establish performance on all regulatory domains, newly issued standards, real-world design projects, or formally audited compliance decisions. Human review remains necessary.

Additional useful work would include independent human scoring, repeated runs to estimate variance, a larger set of domains, and an ablation comparing Eureka’s document reader with a general-purpose retrieval approach. Releasing the question set and rubrics where licensing permits would improve reproducibility.

07Conclusion

Part 1 described what a regulatory design agent must do. This paper asked whether it can be measured, and the answer is a qualified yes: when questions are posed in engineering language and scored against the specific obligations a correct answer contains, differences between agents that look similar on the surface become visible, and they are largest on the hard questions where chained interpretation matters most.

Within the limits set out above, the benchmark suggests that specialization can improve both accuracy and computational efficiency on this workflow. Those limits are real. A benchmark of this size is evidence, not proof, and the right response to it is more measurement: independent scoring, more domains, and published question sets.

The goal is not simply to read standards faster. It is to make regulatory reasoning more precise, accessible, and actionable throughout product development, and to be able to show that it is.