Real-SWE評測：企業真實程式碼，AI最高只答對38.8% | Real-SWE Review: AI Tops Out at 38.8% on Real Code

9月 13, 2026

By Kit 小克 | AI Tool Observer | 2026-09-14

🇹🇼 Real-SWE評測：企業真實程式碼，AI最高只答對38.8%

如果你懷疑過「AI編碼工具在公開跑分上很猛，一丟進公司真實 codebase 就掉鏈子」，現在有數據可以佐證了。新發布的 Real-SWE 評測不用 SWE-bench 那種公開 GitHub 題庫，而是直接向真實企業租用私有生產環境程式碼當考題，結果最強的 Claude Fable 5.1 也只解出 38.8% 的任務——這才是 AI 編碼能力的真實水位。

Real-SWE 跟 SWE-bench 差在哪

Real-SWE 由新創 Specific Labs 打造，題目來自真實公司的私有 codebase，例如一個 20 萬用戶規模的活動平台、一個處理超過 10 萬份銀行對帳單的消費金融產品。跟公開題庫比，Real-SWE 的任務平均要改 11 個檔案（公開題庫多半只要 6 個），題目描述平均長達 1,742 字元，內容涉及帳務計稅、跨服務資料庫遷移、API 配額計量這類「改錯會出人命」的業務邏輯，不是單純修個 bug。

Real-SWE 跑分結果（Pass@1，8 次跑分平均）

Fable 5.1（Anthropic）：38.8%，排名第一
GPT-6 Astra（OpenAI）：33.8%
Gemini 3.8 Flash（Google）：31.2%
GLM 5.3：28.8%；Grok 4.6 與 Muse Spark 1.3 並列 23.8%
Kimi K3：18.8%；GPT-5.6 Sol 敬陪末座，16.2%

比分數更重要的是失敗模式

Real-SWE 同時記錄每個模型「怎麼死的」。Fable 5.1 有 36.7% 的失敗案例是漏看需求；GPT-5.6 Sol 有 43.3% 的失敗來自「沒查證就亂猜」。更值得注意的是，71.4% 在 10 分鐘內就結束的任務全部失敗——代表模型急著交卷、沒把 codebase 的前後文讀懂就動手改，這跟工程師常說的「AI 改 code 改得很快但改錯地方」完全吻合。花費也不是越貴越準，每次任務花 $2.50 到 $6.96 美元不等，貴不代表解得出來。

對開發者的實際意義

Real-SWE 的分數看起來低，但這才誠實——沒有一個模型能穩定解決企業級的複雜任務，任務難度落差極大（0% 到 67.2% 都有）。如果你一直靠公開跑分挑選 AI 編碼工具，現在該多一個心眼：Real-SWE 提醒我們「跑分冠軍」不等於「丟進你家 codebase 也一樣好用」。目前 Fable 5.1、GPT-6 Astra 相對穩，但差距不到 5 個百分點，實際用起來還是得看你的 codebase 複雜度跟技術棧而定。

好不好用，試了才知道。

🇺🇸 Real-SWE Review: AI Tops Out at 38.8% on Real Code

If you've ever suspected that AI coding tools look great on public benchmarks but stumble the moment you point them at your company's real codebase, there's now data to back that up. The new Real-SWE benchmark skips public GitHub repos like SWE-bench entirely and instead licenses private, production codebases from real companies. The top scorer, Claude Fable 5.1, only resolved 38.8% of tasks — and that's the honest baseline for enterprise AI coding today.

How Real-SWE Differs from SWE-bench

Built by startup Specific Labs, Real-SWE sources tasks from actual private codebases — think a 200K-user events platform or a consumer fintech product processing 100K+ bank statements. Compared to public benchmarks, Real-SWE tasks require touching a median of 11 files (versus 6 elsewhere), average 1,742-character instructions, and cover business-critical logic like tax calculations, cross-service database migrations, and API quota metering — not toy bug fixes.

Real-SWE Resolution Rates (Pass@1, avg. of 8 runs)

Fable 5.1 (Anthropic): 38.8% — #1
GPT-6 Astra (OpenAI): 33.8%
Gemini 3.8 Flash (Google): 31.2%
GLM 5.3: 28.8%; Grok 4.6 and Muse Spark 1.3 tied at 23.8%
Kimi K3: 18.8%; GPT-5.6 Sol last at 16.2%

The Failure Patterns Matter More Than the Score

Real-SWE also logs how each model fails. 36.7% of Fable 5.1's failures came from missing requirements; 43.3% of GPT-5.6 Sol's failures were unverified assumptions. More telling: 71.4% of rollouts that finished in under 10 minutes failed — models rushing to submit without actually understanding the surrounding codebase, which matches what engineers have long complained about: AI edits code fast, just not in the right place. Cost didn't track with accuracy either — rollouts ranged $2.50 to $6.96, and paying more didn't guarantee a solve.

What This Means If You're Picking a Coding Agent

The low scores are actually the useful part — no model reliably handles enterprise-grade complexity, and task difficulty swings wildly (0% to 67.2% resolution). If you've been choosing AI coding tools off public leaderboards alone, Real-SWE is a reminder that a benchmark champion isn't automatically good at your codebase. Fable 5.1 and GPT-6 Astra currently lead, but the gap is under 5 points — what actually works still depends on your own codebase and stack.

好不好用，試了才知道 (You won't know if it's good until you've actually tried it).

Sources / 資料來源

延伸閱讀 / Related Articles

AI 工具觀察站 — 每日精選 AI Agent 與工具趨勢
AI Tool Observer — Daily curated AI Agent & tool trends

搜尋此網誌

AI小貼士