The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

2609.09853 ↗ · 2026-09-09 · cs.AI

Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary

12 pages

The finding

Headline claim

A fully generated, seeded fictional enterprise estate spanning 66 product simulators with computed exact answer keys can measure LLM agent accuracy over enterprise systems of record, and current models score only 42.4% to 76.8% on its simulator track.

The test

Deterministic generation of 23 fictional companies (plus 83 scenario combinations) across ten industries and six size tiers from a single seed; 33 simulator questions answered three times each by nine models via MCP tools, with paired statistical tests; realism validated by a scorecard and adversarial detector.

Effect size & scope

Model accuracy ranged from 42.4% to 76.8%; only three of 36 pairwise model differences remained supported after multiple-comparison correction, so most models are statistically indistinguishable. Realism scores rose from 61.8 to 97.0 across generator revisions, with synthetic-flagged records falling from 55.2% to zero. Results hold only in this simulated, read-only estate; no real production environments were tested.

Limitations

The authors concede the realism scorecard is benchmarked against Eon's own operational data and published statistics rather than paired real companies, and the internal-database generator is an internal tool not released to clients. The text is truncated before the full limitations section, but obvious unstated gaps include: only 33 questions and 12 composed questions, vendor simulator fidelity versus real product behavior, questions derived from the same templates that generate the data (template-specific skill may not generalize), and no evidence that scores on this benchmark predict real-world enterprise task performance.

Counterpoint

A vendor benchmark built by a company selling enterprise agents, with its own realism scorecard defining realism against its own data, is lab PR dressed as science until independently reproduced. Small question counts, template-generated questions, and read-only grading mean high scores may reflect template familiarity rather than genuine agentic competence; three statistically supported pairwise differences out of 36 suggests the comparison itself is underpowered.

Economy implication

Little or nothing yet for AI economics directly; the indirect signal is that enterprise agent reliability is being instrumented more seriously, and if this style of exact-ground-truth evaluation spreads, it could harden enterprise procurement criteria for agent vendors within 12 to 24 months.

Who this matters for

Technical Capability: Gives agent builders the first multi-system enterprise estate with computed ground truth, exposing that even top models miss roughly a quarter of cross-system retrieval and abstention questions.

Market & Valuation: Benchmark quality is becoming a differentiator in enterprise agent tooling; Eon's benchmark is also marketing for Eon, so investors should treat reported model rankings as unaudited vendor claims.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We present the Era by Eon Benchmark for evaluating LLM agents that use enterprise tools. The benchmark is built around a complete fictional company. It includes product simulators, company-specific internal databases, benchmark questions, and computed answer keys. Industry, company size, business model, application portfolio, and a seed define each company. One seeded entity graph supplies shared company data to simulators of Salesforce, Zendesk, Slack, Gong, and other products. A questionconditioned generator creates the schemas and records for internal databases. It takes shared entities, keys, and values from the same graph before generating database-specific facts. Both mechanisms therefore describe one consistent enterprise estate. Every expected answer is computed from the final records, so grading is exact. Design and answer-key checks validate the internal databases. A realism scorecard and adversarial detector validate the entity graph. Across 23 generated companies, the mean realism score rose from 61.8 to 97.0, with zero records flagged as synthetic. In the reported simulator-track comparison, nine models answered the same 33 questions three times each. Accuracy estimates ranged from 42.4% to 76.8%, and three of 36 pairwise differences remained supported after correction.