Models struggle with the long-horizon workflows across the tools and apps that professionals use daily. This is the biggest obstacle to AI models reshaping knowledge work and creating trillions of dollars of productivity gains.
The issue is that benchmarks often test capabilities in isolation where there are clean, well-specified problems with clear inputs. But real professional work is messy: you have to find the right information across hundreds of files, maintain context through long task horizons, coordinate multiple tools, and handle difficult requirements. Closing this sim-to-real gap is the biggest unlock for improving AI capabilities. That's why Mercor created APEX-Agents.
模型在专业人员日常使用的工具和应用中执行长周期工作流时表现不佳。这是 AI 模型重塑知识工作、创造数万亿美元生产力提升的最大障碍。
问题在于，现有基准测试往往孤立地测试能力——都是清晰的、定义良好的问题。但真实专业工作是混乱的：你必须在数百个文件中找到正确信息，保持长任务周期的上下文，协调多种工具，处理复杂需求。弥合这种 Sim-to-Real 鸿沟是提升 AI 能力的最大突破口，这就是 Mercor 创建 APEX-Agents 的原因。
Mercor's AI Productivity Index for Agents (APEX-Agents) comprises a benchmark and dev sets, built by domain experts executing realistic, long-horizon, cross-application tasks. The benchmark measures how well you can automate everything you can do in Google Workspace and other enterprise applications across Law, Management Consulting, Finance and Banking, Data Science, Human Resources, and Medicine.
Using Mercor's Studio, they've created realistic work environments – complete Google Workspace clones populated by domain experts acting out real project scenarios. Hundreds of files, email threads, and actual tools in fully developed and realistic worlds.
Mercor 的 AI 生产力指数（APEX-Agents）包含基准测试和开发集，由领域专家执行真实的长周期跨应用任务构建。该基准衡量模型在法律、管理咨询、金融银行、数据科学、人力资源和医学等领域中自动化 Google Workspace 及其他企业应用任务的能力。
通过 Mercor 的 Studio 平台，创建了逼真的工作环境——完整的 Google Workspace 克隆，由领域专家扮演真实项目场景。每个环境包含数百个文件、邮件线程和实际工具。
APEX-Agents is built on three components that create a comprehensive "gym" for evaluating AI agents on real-world long-horizon professional tasks: (1) Worlds, (2) Apps, and (3) Tasks and verifiers.
APEX-Agents 基于三大组件构建了一个综合性的"训练场"，用于评估 AI Agent 在真实长周期专业任务上的表现：(1) Worlds（世界环境）、(2) Apps（应用工具）、(3) Tasks & Verifiers（任务与验证器）。
Worlds are high-fidelity digital workspaces that simulate a real-world professional environment. Each world serves as the context-rich "data room" where the agents complete tasks.
How Worlds are Built:
World 是高保真数字工作空间，模拟真实专业工作环境。每个 World 作为内容丰富的"数据室"，供 Agent 在其中完成任务。
构建方式：
| Characteristic | Why It Matters | 特征 | 重要性 |
|---|---|---|---|
| Authentic artifact trails | Experts generate artifacts the way professionals actually do — with revisions, email threads, and realistic messiness | 真实文件痕迹 | 专家以专业人员的真实方式生成文件——含修订、邮件线程和现实中的混乱 |
| Hundreds of interconnected files | 100-600+ files per scenario requiring information retrieval across a data room | 数百个关联文件 | 每场景 100-600+ 个文件，需在数据室中进行信息检索 |
| Cross-application complexity | Tasks require coordinating docs, spreadsheets, slides, emails, and domain-specific tools | 跨应用复杂性 | 任务需要协调文档、表格、幻灯片、邮件和领域专用工具 |
| Domain-specific realism | Each World is built by practitioners in that domain — M&A due diligence looks different from medical diagnosis | 领域专业真实性 | 每个 World 由该领域从业者构建——并购尽调与医学诊断截然不同 |
APEX-Agents contains realistic applications that agents interact with — primarily containerized apps with MCP servers spanning productivity tools and domain-specific software.
APEX-Agents 包含 Agent 可交互的真实应用——主要是容器化应用加 MCP 服务器，涵盖生产力工具和领域专用软件。
| Domain | App Group | Application | 领域 | 应用类别 | 具体应用 |
|---|---|---|---|---|---|
| Finance | Financial data | FMP, EDGAR | 金融 | 金融数据 | FMP, EDGAR |
| Finance | Tax / Accounting | Quickbooks, Xero, TaxJar, ADP | 金融 | 税务/会计 | Quickbooks, Xero, TaxJar, ADP |
| Consulting | Productivity | Slide Tools (think-cell) | 咨询 | 生产力工具 | 演示工具 (think-cell) |
| Law | Patent & Trademark | USPTO, EDGAR | 法律 | 专利商标 | USPTO, EDGAR |
| Medicine | Medical Records | OpenEMR | 医学 | 医疗记录 | OpenEMR |
| Data Science | Data Viz / Code | Looker/Tableau, SQL, Code Exec | 数据科学 | 数据可视化/代码 | Looker/Tableau, SQL, 代码执行 |
| HR | HCM / Recruiting | Workday, SAP, BambooHR | HR | 人力资源管理 | Workday, SAP, BambooHR |
| All domains | Productivity | Sheets, Docs, Slides, Calendar, Chat, Email, PDF, Code | 通用 | 生产力工具 | 表格、文档、演示、日历、聊天、邮件、PDF、代码 |
Each task is a real request that a professional would create. The agent's trajectory — files accessed, tools used, pathway taken — is traced step-by-step for granular understanding.
每个任务都是专业人员会提出的真实请求。Agent 的执行轨迹——访问的文件、使用的工具、采取的路径——被逐步跟踪记录，提供细粒度理解。
Scenario: NorthStar Inc. transferred consumer data to an outside entity. The legal team needs to determine notification obligations for affected Colorado residents.
What SOTA Models Do: The model locates the Colorado statute and concludes NorthStar is required to notify residents.
Why This Is Wrong: Models miss the exception that actually governs this case — NorthStar qualifies as a "covered entity" under a specific statutory exception that exempts them from notification. Any practicing privacy attorney would identify this immediately.
场景：NorthStar Inc. 将消费者数据转移给外部实体。法律团队需确定对受影响的科罗拉多州居民的通知义务。
SOTA 模型的表现：模型找到科罗拉多州法规，得出 NorthStar 需要通知居民的结论。
为何错误：模型遗漏了实际适用于此案的例外条款——NorthStar 属于特定法定例外下的"受保护实体"，豁免通知义务。任何执业隐私律师都能立即识别这一点。
| Domain | Tasks | Price/Task | Total | 领域 | 任务数 | 单价 | 总价 |
|---|---|---|---|---|---|---|---|
| Finance | 830 | $1,800 | $1,494,000 | 金融 | 830 | $1,800 | $1,494,000 |
| Consulting | 590 | $1,800 | $1,062,000 | 咨询 | 590 | $1,800 | $1,062,000 |
| Law | 530 | $2,000 | $1,060,000 | 法律 | 530 | $2,000 | $1,060,000 |
| Medicine | 250 | $2,000 | $500,000 | 医学 | 250 | $2,000 | $500,000 |
| Education | 100 | $1,600 | $160,000 | 教育 | 100 | $1,600 | $160,000 |
| Data Science | 200 | $1,600 | $320,000 | 数据科学 | 200 | $1,600 | $320,000 |
| Accounting | 40 | $1,800 | $72,000 | 会计 | 40 | $1,800 | $72,000 |
| HR | 80 | $1,400 | $112,000 | HR | 80 | $1,400 | $112,000 |
| Total | 2,620 | — | $4,780,000 | 合计 | 2,620 | — | $4,780,000 |
APEX-Agents evaluates frontier agents on n=480 tasks, split evenly across Investment Banking, Corporate Law, and Management Consulting.
APEX-Agents 在 480 个任务上评估前沿 Agent，均分于投行、公司法和管理咨询三个领域。
| Domain | Avg. Unique Tools | Avg. Total Calls | Expected Time (h) | SOTA Pass@1 | SOTA Accuracy | 领域 | 平均独立工具数 | 平均总调用数 | 预计时间 (h) | SOTA Pass@1 | SOTA 准确率 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Finance | 8.91 | 31.2 | 1.56 | 6.7% | 40% | 金融 | 8.91 | 31.2 | 1.56 | 6.7% | 40% |
| Consulting | 8.60 | 30.1 | 2.13 | 30% | 53% | 咨询 | 8.60 | 30.1 | 2.13 | 30% | 53% |
| Law | 7.25 | 27.2 | 4.46 | 15% | 33% | 法律 | 7.25 | 27.2 | 4.46 | 15% | 33% |
| Medicine | 10.07 | 32.4 | - | 1.4% | 38% | 医学 | 10.07 | 32.4 | - | 1.4% | 38% |
| Education | 7.71 | 24.6 | 1.42 | 8.6% | 19% | 教育 | 7.71 | 24.6 | 1.42 | 8.6% | 19% |
| Data Science | 6.98 | 31.4 | 2.67 | 20% | 21% | 数据科学 | 6.98 | 31.4 | 2.67 | 20% | 21% |
| Accounting | 8.69 | 31.1 | 1.60 | 4.2% | 8.7% | 会计 | 8.69 | 31.1 | 1.60 | 4.2% | 8.7% |
| HR | 10.45 | 45.6 | - | 18.2% | 32% | HR | 10.45 | 45.6 | - | 18.2% | 32% |
| Total | 8.44 | 30.0 | 2.45 | 15% | 39% | 合计 | 8.44 | 30.0 | 2.45 | 15% | 39% |
Of 480 tasks, 192 (40%) are unsolvable by any model even with 8 attempts. All agents pass at least 5 unique tasks no other agent can pass. GPT-5.2 passes the most unique tasks (n=17), concentrated in management consulting.
在 480 个任务中，192 个（40%）所有模型即使 8 次尝试也无法通过。所有 Agent 至少通过 5 个其他 Agent 无法通过的独特任务。GPT-5.2 通过最多独特任务（17 个），集中在管理咨询领域。
| Agent | Pass@8 | Unique Passes | Shared Passes | Missed (others pass) | 独特通过 | 共同通过 | 未通过（他人通过） |
|---|---|---|---|---|---|---|---|
| claude-opus-4-5 | 163 | 7 | 156 | 125 | 7 | 156 | 125 |
| gemini-3-flash | 176 | 6 | 170 | 112 | 6 | 170 | 112 |
| gemini-3-pro | 179 | 6 | 173 | 109 | 6 | 173 | 109 |
| GPT-5 | 149 | 5 | 144 | 139 | 5 | 144 | 139 |
| GPT-5.2 | 192 | 17 | 175 | 96 | 17 | 175 | 96 |
| grok-4 | 158 | 7 | 151 | 130 | 7 | 151 | 130 |
Overall: GPT-5.2 uses todo_write (planning) 21.6pp more and toolbelt_inspect_tool (tool discovery) 17.6pp more. Gemini 3 Pro reads documents more directly (word_read_document_content +17.4pp).
Management Consulting (GPT-5.2 wins): GPT-5.2 plans more (+39.4pp todo_write) and reads images/PDFs more extensively. Gemini focuses on specific apps (Word, PPT, Excel) which may be used inappropriately without proper search first.
Investment Banking (GPT-5.2 wins): GPT-5.2 inspects tools 43.8pp more and plans 26.7pp more. Better at searching before executing.
Corporate Law (Gemini wins): Gemini uses toolbelt functions (+65.2pp), file listing (+64.0pp), and document reading (+57.2pp) far more — demonstrating the same exploration-first approach that makes GPT-5.2 successful in other domains.
总体：GPT-5.2 更多使用 todo_write（规划）+21.6pp 和 toolbelt_inspect_tool（工具发现）+17.6pp。Gemini 3 Pro 更直接地阅读文档（word_read_document_content +17.4pp）。
管理咨询（GPT-5.2 胜出）：GPT-5.2 规划更多（+39.4pp todo_write），更广泛地读取图片/PDF。Gemini 更多使用特定应用（Word、PPT、Excel），在未充分搜索时可能导致不当使用。
投资银行（GPT-5.2 胜出）：GPT-5.2 的工具检查高出 43.8pp，规划高出 26.7pp。更擅长在执行前进行搜索。
公司法（Gemini 胜出）：Gemini 的 toolbelt 函数使用率高出 +65.2pp，文件列举高出 +64.0pp，文档阅读高出 +57.2pp——展现了与 GPT-5.2 在其他领域成功时相同的"先探索再执行"策略。
| Tool | GPT-5.2 (%) | Gemini 3 Pro (%) | Δ absolute |
|---|---|---|---|
todo_write | 50.4% | 28.9% | GPT-5.2 +21.6pp |
toolbelt_inspect_tool | 29.0% | 11.4% | GPT-5.2 +17.6pp |
word_read_document_content | 23.2% | 40.6% | Gemini +17.4pp |
toolbelt_list_tools | 87.7% | 100.0% | Gemini +12.3pp |
toolbelt_add_tool | 87.3% | 99.4% | Gemini +12.1pp |
filesystem_list_files | 86.7% | 98.6% | Gemini +11.9pp |
pdfs_read_pdf_pages | 29.8% | 41.1% | Gemini +11.3pp |
pdfs_search_pdf | 29.3% | 19.4% | GPT-5.2 +10.0pp |
filesystem_read_image_file | 10.5% | 1.2% | GPT-5.2 +9.2pp |
pdfs_read_page_as_image | 11.7% | 4.4% | GPT-5.2 +7.3pp |
Mercor partnered with Applied Compute to post-train an open-source model using one of their expert-labeled dev sets, resulting in substantial performance gains on the APEX-Agents benchmark.
Mercor 与 Applied Compute 合作，使用专家标注的开发集对开源模型进行后训练，在 APEX-Agents 基准上取得了显著的性能提升。
Base Model: GLM 4.6 (355B parameter MoE, 32B active) — baseline: 3.8% Pass@1, 12.1% mean score.
Training Data: 874 tasks across 50 unique worlds, split across domains. None overlap with the benchmark.
Method: Single-epoch RL with Applied Compute's proprietary long-horizon RL stack. No SFT warmup, no filtering, no task/rubric modifications. Two held-out worlds per domain for validation.
Key Findings:
基础模型：GLM 4.6（355B 参数 MoE，32B 激活参数）——基线：Pass@1 3.8%，Mean Score 12.1%。
训练数据：874 个任务，覆盖 50 个独立 World，跨领域分布。与基准测试无重叠。
方法：使用 Applied Compute 专有的长周期 RL 栈进行单轮 RL 训练。无 SFT 预热、无过滤、无任务/评分标准修改。每个领域保留两个 World 用于验证。
关键发现：
Baseline model: Produced a professional-looking but factually flawed memo stating the corporation is in "substantial compliance." Used 18 tool calls, worked linearly. Took documents at face value without cross-checking.
Post-trained model: Correctly identified non-compliance. Found that one shareholder was overpaid in 2022 and 2024, citing 26 USC §1366(a)(1). Spent 9 steps reading specific tax code sections, ran Python scripts to verify distribution math, and performed self-correction when it realized it hadn't completed its checklist.
基线模型：生成了看起来专业但事实有误的备忘录，声称公司"基本合规"。使用 18 次工具调用，线性工作。未交叉验证即接受文档表面信息。
后训练模型：正确识别出不合规。发现一名股东在 2022 和 2024 年被多付，引用 26 USC §1366(a)(1)。花费 9 步阅读具体税法条文，运行 Python 脚本验证分配数学，并在发现未完成检查清单时进行自我纠正。
Mercor's network of 2M+ professionals across 50+ domains. Every expert meets: 5+ years experience, credential verification, top-tier institutions, active practitioners preferred.
Mercor 拥有覆盖 50+ 领域的 200 万+ 专业人员网络。每位专家需满足：5+ 年经验、资质认证、顶级机构背景，优先选择活跃从业者。
| Archetype | Expert Pool | Rubric Experts | Sample Profile | 专家类型 | 专家池 | 评分专家 | 示例背景 |
|---|---|---|---|---|---|---|---|
| Doctors | 5k+ | 1k+ | Stanford residency, VCU School of Medicine MD | 医生 | 5k+ | 1k+ | 斯坦福住院医师，弗吉尼亚联邦大学医学博士 |
| High Finance | 5k+ | 500+ | JP Morgan Healthcare IB → PE, Harvard MBA | 高端金融 | 5k+ | 500+ | 摩根大通医疗IB → PE，哈佛 MBA |
| STEM PhDs | 20k+ | 500+ | UC Berkeley Microbiology PhD, Industry Staff Scientist | STEM 博士 | 20k+ | 500+ | 加州伯克利微生物学博士，行业资深科学家 |
| MBB Consultants | 2k+ | 500+ | UChicago Booth MBA, BCG Partner | MBB 咨询师 | 2k+ | 500+ | 芝加哥大学布斯 MBA，BCG 合伙人 |
| Lawyers | 5k+ | 1k+ | UVA JD, Cadwalader Associate | 律师 | 5k+ | 1k+ | 弗吉尼亚大学法学博士，Cadwalader 律师 |
| Olympiad Competitors | 500+ | 250+ | 2022 IMO Gold Medal (Brazil) | 奥赛选手 | 500+ | 250+ | 2022 IMO 金牌（巴西） |
Task: Evaluate whether Pret should expand into mid-sized US cities with a discount loyalty program. Blends ROI analysis, market landscape assessment, and population data across multiple files.
Model Failure: Overestimates loyalty discount cost due to incorrect mathematical analyses and missed prompt details — multi-level calculation and reasoning failures.
任务：评估 Pret 是否应以折扣忠诚度计划进入美国中型城市。融合 ROI 分析、市场格局评估和人口数据，需整合多个文件。
模型失败原因：由于错误的数学分析和遗漏提示细节，高估了忠诚度折扣成本——多层计算和推理失败。
Task: Update DCF assumptions for an optimistic operating case and recalculate equity value/share price for MTZ. Requires parsing Excel, extracting FMP data.
Model Failure: Ignores existing DCF model, attempts to build its own in Python. Returns a negative share price — impossible — showing calculation and conceptual failures.
任务：按照更乐观的运营假设更新 DCF 模型，重新计算 MTZ 的股权价值和每股价格。需解析 Excel、提取 FMP 数据。
模型失败原因：忽略现有 DCF 模型，试图用 Python 自建。返回负股价（不可能），暴露计算和概念推理失败。
Task: Diagnose a postmenopausal patient with urinary symptoms. Must synthesize symptoms, labs, and menopausal status to identify Genitourinary Syndrome of Menopause (GSM).
Model Failure: Shifts focus when family cancer history is introduced, overweighting malignancy risk despite absence of red-flag symptoms. Highlights reasoning robustness challenges in dynamic prompts.
任务：诊断一位有泌尿症状的绝经后患者。需综合症状、实验室结果和绝经状态，识别更年期泌尿生殖综合征（GSM）。
模型失败原因：当引入家族癌症史后注意力偏移，在缺乏危险信号症状时过度关注恶性肿瘤风险。暴露了动态提示下的推理鲁棒性挑战。
Task: Assess the impact of lowering the Loyalty Hoarder points threshold. Requires priority-based, mutually exclusive segment assignment.
Model Failure: Applies threshold as standalone filter, ignoring priority hierarchy — overstating segment size with materially inaccurate results.
任务：评估降低忠诚度囤积者积分阈值的影响。需要基于优先级的互斥分群分配。
模型失败原因：将阈值作为独立筛选条件，忽略优先级层次——导致分群规模被严重高估。
Task: Evaluate Senior SWE candidates using documented interview policies, calculate weighted scores, and create offer profiles in Greenhouse ATS.
Model Failure: Advances an unqualified candidate by ignoring explicit edge-case rules. Creates 5 profiles instead of 4, incorrectly flags required fields as missing.
任务：根据文档化的面试评估政策评估高级软件工程师候选人，计算加权分数，在 Greenhouse ATS 中创建 offer 档案。
模型失败原因：忽略明确的边界情况规则，推进了一名不合格候选人。创建 5 份档案而非 4 份，错误标记必填字段为缺失。
All APEX-Agents data is delivered through Mercor Studio with the following structure:
所有 APEX-Agents 数据通过 Mercor Studio 交付，目录结构如下：
A gym for agent inference and evaluation. Supports hosted evaluation platform (run evals, compare models, view side-by-side) and local deployment via Docker containers.
Agent 推理与评估的训练场。支持托管评估平台（运行评测、对比模型、并排查看）和通过 Docker 容器的本地部署。
| Inference | Evaluation | 推理 | 评估 | ||
|---|---|---|---|---|---|
| Purpose | Run agent against a task | Grade agent's response | 目的 | 在任务上运行 Agent | 对 Agent 响应评分 |
| Input | World apps/data, prompt, LLM | Inference outputs, verifiers, judge LLM | 输入 | World 应用/数据、提示词、LLM | 推理输出、验证器、评审 LLM |
| Output | Response, trajectory, modified data | Pass/fail grades + reasoning | 输出 | 响应、轨迹、修改后数据 | 通过/失败评分 + 推理过程 |
Mercor's QA employs a comprehensive four-layer verification system:
Mercor 的质量保证采用全面的四层验证体系：
| Layer | Type | Function | Coverage | 层级 | 类型 | 功能 | 覆盖率 |
|---|---|---|---|---|---|---|---|
| 1 | Deterministic Checks | Automated validation (structure, formatting, completeness) | 100% | 1 | 确定性检查 | 自动验证（结构、格式、完整性） | 100% |
| 2 | Human Review | Expert evaluation (pass/fail against rubric) | 100% | 2 | 人工评审 | 专家评估（按评分标准通过/不通过） | 100% |
| 3 | LLM Audit | Calibration verification of human review accuracy | 100% | 3 | LLM 审计 | 验证人工评审准确性的校准检查 | 100% |
| 4 | Human Audit | Discrepancy resolution & pipeline validation | Sample | 4 | 人工审计 | 差异解决与流水线验证 | 抽样 |
17,000+ experts certified through a 5-hour scaffolded curriculum. Final assessment scores back-tested against on-project KPI outcomes. Only experts scoring 85%+ are eligible to be staffed. Every criterion must be: Falsifiable, Unambiguous, Self-contained, Non-stacked, Timeless — with universal expert agreement.
17,000+ 名专家通过 5 小时结构化课程认证。期末评估分数经项目 KPI 结果反向验证。仅 85%+ 得分的专家才有资格参与项目。每条标准必须满足：可证伪、无歧义、自包含、不堆叠、时效无关——并达成专家普遍共识。