The Case for Vendor-Agnostic AI Evaluation Infrastructure in Government and Enterprise

Enterprise spending on generative AI reached $37 billion in 2025, more than tripling the $11.5 billion recorded in the prior year.[27] Across the federal government, agencies are integrating large language models into workflows ranging from intelligence analysis to acquisition support, driven by executive directives and competitive pressure from near-peer adversaries. Yet this expansion rests on an evaluation ecosystem that is, by the assessment of multiple independent bodies, fundamentally broken.

The European Union's Joint Research Centre, in a meta-review of approximately 110 studies presented at AIES 2025, identified "a series of systemic flaws in current benchmarking practices, such as misaligned incentives, construct validity issues, unknown unknowns, and problems with the gaming of benchmark results."[8] The Stanford AI Index Report 2025 documented benchmark saturation so severe that scores on foundational benchmarks like GPQA rose by 48.9 percentage points in a single year, rendering them effectively uninformative as discriminators of model quality.[25] And in April 2025, Meta's submission of an undisclosed experimental variant of Llama 4 Maverick to the LM Arena leaderboard—achieving a misleading first-place ranking—demonstrated that even the evaluation platforms widely regarded as independent are vulnerable to strategic manipulation.[33,35]

Against this backdrop, the Defense Innovation Unit and the Office of the Director of National Intelligence issued the MYSTIC DEPOT solicitation in March 2026, seeking "an evaluation harness and government-specific benchmarks that together enable rigorous, reproducible, vendor-agnostic assessment of any AI system against government-defined criteria."[11,12] This solicitation signals an institutional acknowledgment that current evaluation approaches are insufficient for the stakes involved. When an 80% failure rate characterizes government AI initiatives and only six percent of the federal acquisition workforce has received even basic AI training,[18] the gap between what is being procured and what is being measured is no longer a technical nuisance—it is a systemic risk.

This paper argues that the solution is not incremental improvement to existing benchmarks, but rather the construction of vendor-agnostic evaluation infrastructure capable of operating across classification levels, combining traditional capability benchmarks with behavioral profiling, and supporting continuous monitoring throughout the model lifecycle. We examine the evidence for this position, survey the regulatory and institutional landscape, and propose an architectural framework suitable for government and enterprise deployment.

The ecosystem for evaluating large language models has grown rapidly but unevenly. A taxonomy of the principal evaluation mechanisms reveals both their contributions and their structural limitations.

The Chatbot Arena (now LM Arena, formerly LMSYS) has emerged as the de facto standard for conversational AI ranking. Its methodology—crowdsourced pairwise blind comparisons producing Bradley-Terry model rankings from over two million human votes—represents a genuine advance in ecological validity.[30] The HuggingFace Open LLM Leaderboard, which launched its V2 iteration in October 2024 with updated benchmarks including MMLU-Pro and GPQA, provides automated evaluation for open-weight models.[31] LiveBench, a spotlight paper at ICLR 2025, addresses contamination by releasing new questions monthly derived from recent information sources and scoring them against objective ground truth without relying on LLM judges.[1]

Each platform addresses specific deficiencies of its predecessors, yet none provides the comprehensive, standardized, and reproducible evaluation that procurement decisions demand. Leaderboard rankings produce ordinal comparisons that obscure the multi-dimensional nature of model capability, while automated benchmarks optimize for measurability at the expense of ecological validity.

The standard benchmark battery has undergone significant turnover as models have saturated earlier instruments. MMLU, long the workhorse of broad knowledge assessment, has been largely supplanted by MMLU-Pro, which employs ten answer choices and expert review to resist guessing strategies.[25] SWE-bench, measuring real-world software engineering capability, saw solve rates rise from 4.4% to 71.7% between 2023 and 2024. GPQA, designed for graduate-level scientific reasoning, experienced a 48.9 percentage point score increase in a single year. As the Stanford AI Index observed: "Every year, we look at how these algorithms are performing across benchmarks, and every year it seems like they're beating those benchmarks."[25]

This trajectory of rapid saturation means that benchmark scores become uninformative precisely when adoption decisions are being made. A model scoring 92% on MMLU in 2025 cannot be meaningfully compared to a model scoring 88% in the same period when the benchmark's discriminative power has been exhausted.

Perhaps the most consequential source of evaluation data for procurement officers remains the vendor technical report. These publications are, by construction, exercises in selective disclosure. No standardized reporting format exists; each vendor highlights benchmarks where its models excel while omitting unfavorable results. Self-reported numbers often derive from optimized configurations—temperature settings, system prompts, few-shot examples—that are unavailable to end users. Independent evaluations conducted by organizations such as Epoch AI and Scale AI frequently produce scores that diverge materially from vendor claims.[25] The absence of mandatory, standardized disclosure creates an information asymmetry that distorts procurement decisions across both government and enterprise.

Data contamination—the overlap between training corpora and test sets—is the most technically pernicious failure mode of the current evaluation paradigm. Research presented at ACL 2025 introduced AntiLeakBench, an automated framework for constructing evaluation samples with knowledge explicitly absent from training sets, responding to the growing evidence that contamination inflates scores by approximately ten points on affected datasets.[2] Analysis of software engineering benchmarks revealed extreme leakage ratios: 100% for QuixBugs and 55.7% for BigCloneBench, though average rates across 83 benchmarks were lower at 4.8% for Python and 2.8% for Java.[3]

An ICML 2025 study examined whether the sheer scale of modern training datasets provides natural protection through "forgetting" of contaminated samples, concluding that while large datasets offer some mitigation, this mechanism is unreliable as a systematic defense.[10] Contamination detection itself remains immature: the first dedicated workshop was held only in 2024, and no algorithmic detection method has achieved broad acceptance. The implication is that any evaluation regime relying primarily on static benchmarks with known test sets operates on a foundation that is structurally compromised.

Goodhart's Law—"when a measure becomes a target, it ceases to be a good measure"—has found perhaps its most vivid contemporary illustration in AI evaluation. The SWE-bench experience is instructive: because the initial test set was limited to Python repositories, developers trained models exclusively on Python code, producing impressive benchmark scores that masked complete inability to handle other programming languages. The benchmark became a target rather than a measure.[8]

More broadly, the EU JRC's AIES 2025 meta-review documented a culture of "SOTA-chasing" in which benchmark scores are valued more highly than genuine capability assessment. Teams fine-tune on test sets, exploit unlimited submission policies, or selectively report results—practices that often operate within the norms of the field rather than violating them.[36] For procurement officers who must make consequential deployment decisions, this means that published benchmark scores are, at best, noisy signals contaminated by optimization pressure and, at worst, deliberately misleading.

The reproducibility crisis in AI research mirrors and amplifies the broader scientific reproducibility problem first highlighted in Science in 2018.[9] Current estimates indicate that nearly 70% of AI researchers report difficulty reproducing published results, with the rate increasing when AI is integrated into the research pipeline itself.[9] Primary barriers include inadequate documentation of hyperparameters and training conditions, unavailable code and data, poor adherence to reporting standards, and the inherent sensitivity of machine learning training to initial conditions. Princeton's ML Reproducibility Challenge, held in August 2025, represented one institutional response, but the field remains far from established norms of reproducible practice.[7]

For government and enterprise consumers, non-reproducibility translates directly to risk: a model that achieved published benchmark scores under laboratory conditions may perform differently when deployed in production environments, and the evaluating organization has no reliable mechanism to verify the claimed performance independently.

The April 2025 Llama 4 Maverick controversy merits detailed examination as it crystallizes the structural vulnerabilities of the current evaluation ecosystem in a single, well-documented episode.

In early April 2025, Meta submitted what it described as "Llama 4 Maverick" to the LM Arena (Chatbot Arena) leaderboard, where it achieved a first-place ranking with an Elo rating of 1,417, surpassing GPT-4o, Gemini 2.0 Pro, and all other listed models. Within days, independent analysis revealed that the model submitted to the Arena was not the same model publicly released under the Llama 4 Maverick designation. Meta had submitted an "experimental chat version" specifically optimized for conversational characteristics—notably excessive verbosity and stylistic flourishes—that exploited the Arena's known bias toward longer, more engaging responses.[33,34]

When the publicly available, unmodified version of Llama 4 Maverick was independently tested on the same platform, it ranked below GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—a material discrepancy from the ranking claimed in Meta's public communications. Subsequent reporting revealed that Meta had tested 27 private model variants before the public release, a practice that allows vendors to optimize for evaluation platforms while publishing only the most favorable results.[35]

The LM Arena administrators stated publicly that "Meta's interpretation of our policy did not match what we expect from model providers."[37] Meta Vice President Ahmad Al-Dahle denied that the company had trained on test sets but did not address the fundamental issue: that the model evaluated and the model released were substantively different artifacts.[33]

This incident is significant not because it represents an anomaly, but because it illustrates the structural incentives at work. The Arena platform, despite its methodological sophistication, had no mechanism to verify that the model submitted for evaluation was identical to the model subsequently released. Vendors face strong incentives to maximize leaderboard placement, and the absence of cryptographic model identity verification, mandatory disclosure of evaluation variants, or independent model sampling means that the evaluation infrastructure cannot enforce the assumption on which its credibility depends: that the model being evaluated is the model being deployed.

For government procurement, where model selection decisions may affect intelligence analysis, logistics planning, or operational safety, this vulnerability is not an academic concern. It is a supply chain integrity problem.

The MYSTIC DEPOT Commercial Solutions Opening, issued jointly by the Defense Innovation Unit and the Office of the Director of National Intelligence in March 2026, represents the most explicit institutional articulation of requirements for vendor-agnostic AI evaluation infrastructure to date. The solicitation defines two lines of effort: first, an evaluation harness comprising a model interface, execution engine, measurement and scoring system, and output reporting capability; and second, government-specific benchmarks addressing requirements gathering, task decomposition, scoring criteria, baseline establishment, validation, and resistance to gaming.[11,12]

Critically, the solicitation requires deployment across unclassified, classified cloud, and air-gapped environments, and mandates capabilities for stress testing under Denied, Degraded, Intermittent, or Limited (DDIL) conditions, automated red-teaming, and human-machine teaming assessment across human-only, AI-only, and collaborative scenarios. These requirements go substantially beyond what any existing public evaluation platform provides.

The Federation of American Scientists has published specific recommendations for government AI evaluation infrastructure, proposing that the Chief Digital and Artificial Intelligence Office lead a formalized AI Benchmarking Initiative, with a $10 million expansion of testing and evaluation budgets and the creation of a centralized benchmarking repository.[13] The FAS framework calls for standardized defense-specific benchmarks, formalized pre-deployment evaluation integrated into acquisition platforms such as Tradewinds, theater-specific operational benchmarks contextualized to commands such as INDOPACOM and EUCOM, and human-in-the-loop evaluation measuring operator trust alongside model accuracy.

The NIST AI Risk Management Framework (AI RMF 1.0) provides a complementary structure, centering on Testing, Evaluation, Verification, and Validation (TEVV) as a core component to be performed regularly rather than as a one-time checkpoint. The framework requires fairness indicators, robustness measures under stress and adversarial inputs, continuous monitoring with drift detection, privacy leakage testing, and stress testing under extreme and degraded conditions.[15]

Additionally, Intelligence Community Directive 203 establishes analytic standards—objectivity, independence from political consideration, timeliness, multi-source integration, and rigorous tradecraft—that map directly to behavioral evaluation requirements for AI systems supporting intelligence analysis.[14] An AI system deployed in an analytic workflow must be evaluated not only for accuracy but for calibrated uncertainty expression, source acknowledgment, and resistance to confirmation bias: behavioral characteristics that no existing capability benchmark measures.

The federal AI procurement environment is characterized by an acute capability deficit. Only approximately 12,000 of the 200,000 federal acquisition professionals had registered for GSA AI training as of late 2024, and an estimated 80% of government AI initiatives fail to achieve their stated objectives.[18] The Office of Management and Budget has issued requirements mandating that contracts bar vendors from using non-public government data to train publicly available AI systems and delineate intellectual property ownership rights, while the Federal Acquisition Regulation is undergoing modernization to incorporate AI governance clauses.[19] Yet without evaluation infrastructure capable of independently verifying vendor claims, these contractual safeguards amount to unverifiable compliance theater.

The technical foundation for vendor-agnostic evaluation already exists in the form of API abstraction layers that decouple application logic from provider-specific interfaces. The question is not whether such abstraction is feasible, but how to architect it for the specific demands of evaluation infrastructure across classification levels and operational contexts.

Three architectural patterns for model abstraction are currently deployed at scale. OpenRouter provides a managed cloud gateway offering access to over 500 models from multiple providers through a unified API, enabling model switching without code changes.[28] LiteLLM offers an open-source Python routing layer that translates a single standard API call into provider-specific formats, with self-hosted deployment that keeps data within organizational boundaries.[29] Custom AI gateways, built on frameworks such as Ray Serve or KServe, provide full infrastructure control at the cost of development investment.

For government evaluation infrastructure, cloud-based gateways are unsuitable for classified environments. MYSTIC DEPOT explicitly requires deployment across unclassified, classified cloud, and air-gapped environments, mandating architectures that can operate without external network connectivity.[11]

The Holistic Evaluation of Language Models (HELM) framework, developed by Stanford's Center for Research on Foundation Models, provides the most mature precedent for vendor-agnostic evaluation architecture. HELM evaluates models across seven dimensions—accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency—through a unified model interface that abstracts across providers including OpenAI, Anthropic, and Google.[26]

HELM's design principles—holistic multi-dimensional assessment, standardized evaluation protocols ensuring reproducibility, full transparency through inspectable prompts and responses, and extensibility to incorporate new benchmarks—map directly to the requirements of government evaluation infrastructure. IBM's HELM Enterprise extension, which adds domain-specific evaluation datasets for finance, legal, climate, and cybersecurity, demonstrates both the extensibility of the pattern and the insufficiency of generic benchmarks for domain-specific procurement.[32]

Drawing on HELM's architectural precedent and MYSTIC DEPOT's operational requirements, we propose a vendor-agnostic evaluation architecture comprising four layers. The model interface layer provides a standardized, pluggable abstraction over diverse model types, deployable across classification levels. The execution engine orchestrates evaluation workflows, managing prompt sequencing, parameter control, and response collection across heterogeneous configurations. The scoring and measurement system evaluates model outputs against both capability benchmarks and behavioral rubrics, supporting automated, LLM-judged, and human evaluation modalities. The reporting and monitoring layer produces auditable evaluation records and supports continuous drift detection for deployed models.

This architecture must be self-hosted and air-gap capable, support evaluation of both API-accessible and locally deployed models, maintain cryptographic provenance of evaluation artifacts, and integrate with existing acquisition and compliance workflows. It must also support living benchmarks—evaluation instruments that are refreshed regularly to prevent contamination—following the model pioneered by LiveBench.[1]

Capability benchmarks measure what a model can do: its accuracy on knowledge retrieval, its solve rate on coding problems, its performance on mathematical reasoning. Behavioral evaluation measures how a model does it: its tendencies, consistencies, risk profile, and alignment characteristics. We argue that both dimensions are essential for responsible procurement and that the current evaluation landscape systematically neglects the latter.

Research published in Nature Machine Intelligence in 2025 applied comprehensive psychometric methodology to personality assessment of large language models, testing 18 models and finding that personality measurements in outputs of large, instruction-tuned models are "reliable and valid."[5] This finding has significant implications for evaluation: if models exhibit stable, measurable behavioral characteristics, then those characteristics are evaluable and should inform deployment decisions. The SpecEval framework, developed concurrently, tests whether models consistently adhere to their own published behavioral specifications, finding that it remains "unclear how consistently models adhere to the specifications" and that model outputs often "superficially mimic human participants, but fail to display coherent behaviour tied to their own internal states."[4]

For government and enterprise deployment contexts, we propose evaluation across sixteen behavioral dimensions: instruction adherence, refusal patterns, uncertainty expression, response consistency, verbosity characteristics, reasoning style, tone and formality, hallucination patterns, cultural and political bias, adversarial robustness, context window utilization, multi-turn coherence, latency characteristics, tool use behavior, error recovery, and citation and sourcing tendencies.

The procurement significance of behavioral profiling is substantial. Two models with identical MMLU scores may exhibit radically different behavioral profiles: one may express uncertainty calibrately and refuse requests outside its competence, while another may confabulate confidently and comply with adversarial prompts. Capability benchmarks cannot distinguish between these models; behavioral profiling can. For intelligence analysis applications governed by ICD 203 standards—requiring objectivity, source diversity, and calibrated uncertainty—behavioral evaluation is not supplementary but primary.[14]

MYSTIC DEPOT explicitly requires subject matter expert interfaces for evaluating workload, usability, and performance across human-only, AI-only, and collaborative scenarios.[11] The FAS recommendations similarly call for measurement of operator trust and confidence alongside model accuracy.[13] Behavioral evaluation provides the foundation for these assessments: operator trust is a function not of model capability per se, but of behavioral characteristics such as uncertainty expression, consistency, and error acknowledgment that shape the human experience of working with the system.

The European Union's AI Act, which began phased implementation with prohibitions on unacceptable-risk AI in February 2025 and General Purpose AI model obligations in August 2025, creates binding evaluation requirements for any organization deploying AI in European markets.[21,22] GPAI model providers must maintain technical documentation making development, training, and evaluation traceable; implement state-of-the-art evaluation protocols; conduct adversarial testing to identify and mitigate systemic risks; and publish transparency reports describing capabilities, limitations, and risks. Models classified as posing systemic risk face additional requirements for model evaluation using standardized protocols and tools "reflecting the state of the art," documented adversarial testing, and robust incident response.[24]

Full high-risk AI system obligations take effect in August 2026, requiring conformity assessments that evaluate quality management, risk management, accuracy, and robustness before market placement.[23] Enforcement carries penalties of up to EUR 35 million or 7% of global annual turnover—a scale of regulatory exposure that demands institutionalized evaluation capability rather than ad hoc testing.

The NIST AI Risk Management Framework and the EU AI Act share substantive alignment across multiple dimensions: risk-based evaluation approaches, continuous rather than point-in-time monitoring, adversarial testing requirements, fairness and bias evaluation, transparency and documentation standards, and post-deployment monitoring for behavioral drift.[15] This convergence has practical implications for organizations operating in both jurisdictions. A vendor-agnostic evaluation infrastructure designed to satisfy NIST AI RMF requirements can, with appropriate extension, serve EU AI Act compliance needs as well, avoiding the costly duplication of maintaining separate compliance evaluation pipelines for each regulatory regime.

The competitive dynamics between US and Chinese AI development further complicate the evaluation landscape. The Stanford AI Index documented a dramatic narrowing of the performance gap on major benchmarks: the MMLU gap decreased from 17.5 to 0.3 percentage points, and the HumanEval gap from 31.6 to 3.7 percentage points between the end of 2023 and end of 2024.[25] As model capabilities converge, the incentive to game evaluations intensifies, making independent evaluation infrastructure correspondingly more critical.

The case for vendor-agnostic evaluation extends beyond technical rigor to strategic risk management. Foundation models are updated on monthly or weekly cycles; vendor lock-in has been characterized as a "silent productivity killer" that constrains organizational agility and complicates integration with existing enterprise systems.[39] Azure confirmed retirement dates for older GPT-4 variants in mid-2025, and OpenAI experienced a global API disruption in June 2025—events that underscore the operational risk of single-vendor dependence.

Organizations that build evaluation infrastructure coupled to a specific vendor's API, benchmarks, or reporting format inherit this dependency in their assessment capability. When the vendor updates, retires, or reprices a model, the organization's evaluation data becomes non-comparable, and the investment in evaluation infrastructure is partially stranded. Vendor-agnostic architecture ensures that evaluation capability is an organizational asset independent of any provider relationship.

Model performance is not static. Vendors routinely update deployed models without version changes, modify safety guardrails that affect behavioral characteristics, alter pricing structures, and retire models with limited notice. Organizations therefore require infrastructure to continuously re-evaluate models against their operational criteria—not solely at procurement time, but throughout the model lifecycle. This is precisely the "continuous monitoring" requirement articulated in both the MYSTIC DEPOT solicitation and the FAS recommendations.[11,13]

The enterprise data supports this urgency: nearly half of companies abandoned AI projects in 2025, approximately 65% of enterprises cannot advance AI pilots to production, and the gap between capability claims and operational reality continues to widen.[27] While these failures have multiple causes, the inability to independently and continuously verify model performance against operational requirements is a contributing factor that evaluation infrastructure directly addresses.

The Securities and Exchange Commission has penalized companies for false or misleading statements about AI capabilities, and the Federal Trade Commission has warned against unsubstantiated AI performance claims. Vendor claims such as "99.9% accuracy in production" are factual assertions that can be tested, verified, and litigated. Yet without standardized evaluation infrastructure, procurement organizations lack the capability to conduct such verification, creating an environment in which inflated claims persist unchecked. Vendor-agnostic evaluation infrastructure transforms AI procurement from a trust-based regime to a verify-based regime—a transition that benefits both buyers and legitimate vendors whose products perform as advertised.

Several challenges remain unresolved. Behavioral evaluation, while theoretically grounded, lacks the standardized instruments available for capability assessment; the sixteen dimensions proposed in this paper require validation and community refinement. The cost of maintaining living benchmarks that resist contamination is nontrivial and requires sustained investment. Air-gapped deployment of evaluation infrastructure introduces latency and operational complexity that may limit the frequency of evaluation cycles in classified environments. And the relationship between behavioral evaluation scores and operational outcomes—whether a model with higher adversarial robustness scores actually produces better results in contested information environments—remains an empirical question requiring longitudinal study.

The evidence presented in this paper supports three principal conclusions. First, the current AI evaluation landscape is structurally inadequate for the stakes involved in government and enterprise deployment. Benchmark contamination, selective vendor reporting, evaluation gaming, and non-reproducibility are not edge cases but systemic features of an ecosystem organized around misaligned incentives. The Llama 4 Maverick incident is illustrative rather than exceptional.

Second, the institutional demand for vendor-agnostic evaluation infrastructure is no longer hypothetical. The MYSTIC DEPOT solicitation, the FAS continuous benchmarking recommendations, the NIST AI RMF's TEVV requirements, and the EU AI Act's evaluation mandates collectively describe a convergent institutional consensus that independent, reproducible, and continuous evaluation is a prerequisite for responsible AI deployment. These are not aspirational positions; they are operational requirements with associated funding, regulatory enforcement, and organizational accountability.

Third, capability benchmarks alone are insufficient. Behavioral evaluation—measuring how models express uncertainty, maintain consistency, resist adversarial manipulation, and adhere to organizational norms—is essential for deployment contexts where the consequences of model failure extend beyond inaccuracy to questions of safety, trust, and institutional credibility. Two models indistinguishable on MMLU may diverge dramatically on the behavioral dimensions that determine operational fitness.

We propose that the development of open, vendor-agnostic evaluation infrastructure—combining capability benchmarks with behavioral profiling, supporting deployment across classification levels, and enabling continuous monitoring throughout the model lifecycle—should be treated as institutional infrastructure rather than a research exercise. The $37 billion now flowing annually into enterprise generative AI, and the national security implications of AI deployment across the Department of Defense and Intelligence Community, demand evaluation infrastructure commensurate with the investment and the risk. The alternative—continued reliance on vendor self-reporting, compromised benchmarks, and evaluation platforms vulnerable to strategic manipulation—is a posture that neither government nor enterprise can responsibly sustain.