GPT-6.1 Sol評測:COBOL遷移飆97.9%,測試黑箱藏陷阱 | GPT-6.1 Sol Review: 97.9% COBOL Score, Hidden Test Catch
By Kit 小克 | AI Tool Observer | 2026-10-22
🇹🇼 GPT-6.1 Sol評測:COBOL遷移飆97.9%,測試黑箱藏陷阱
OpenAI本週發布GPT-6.1 Sol,宣稱在COBOL遷移測試上準確率從前代GPT-5.6 Sol的5.4%,一口氣衝到86.6%(低推理量)甚至97.9%(拉高推理量),同時API成本只要前代的61%。COBOL撐著全球四成以上的網路銀行系統、八成的實體刷卡交易、九成五的ATM提款,這數字乍看像是遺留系統轉型的福音——但細看測試方法,事情沒那麼簡單。
數字很猛:5.4%飆到97.9%是怎麼回事
這個COBOL遷移評測會把老COBOL程式碼轉成Python、Java、Kotlin、C++或Rust,前代模型幾乎全軍覆沒,新模型卻能衝到近滿分。OpenAI把它歸功於推理量拉高加上架構優化,同時強調低推理量模式下成本直接砍四成。
但測試題目是OpenAI自家的黑箱
開發者社群馬上戳破一個關鍵:OpenAI同時公開的另一個基準MultiLangBench放在GitHub上可以自己重跑驗證,但COBOL遷移評測用的是OpenAI自己設計、沒公開的隱藏測試,外部完全無法獨立複現。換句話說,97.9%這個分數,目前只有OpenAI自己說了算。
真正的地雷:四捨五入誤差
有工程師指出COBOL的定點數運算(fixed-point arithmetic)跟Python、Java常見的float運算,四捨五入規則不一樣——老系統小數點第三位直接捨去,新程式碼卻拿去做四捨五入,兩邊算出來的金額對不上。如果驗證只檢查「數字差不多」,這種誤差完全抓不到,放到銀行或ATM系統裡就是真金白銀的帳務問題。
該不該直接拿來用在生產系統?
- 先用自己的平行驗證(parity harness)比對新舊程式輸出,不要只信OpenAI公布的分數
- 拿真實生產資料測試,不要只測開心路徑(happy path)
- 遷移完的程式碼照樣要人工審查,GPT-6.1 Sol目前還不是全自動方案
GPT-6.1 Sol在COBOL遷移上的進步是真的,但「97.9%準確率」跟「可以直接上線」中間還隔著一段距離。對銀行、保險這類系統,省下來的人力成本可能都不夠補一次對帳出錯的損失。
好不好用,試了才知道。
🇺🇸 GPT-6.1 Sol Review: 97.9% COBOL Score, Hidden Test Catch
GPT-6.1 Sol, OpenAI's latest model, claims a dramatic leap on COBOL migration: accuracy jumped from 5.4% (GPT-5.6 Sol) to 86.6% at low reasoning effort, and 97.9% at high reasoning effort — while costing about 61% less per API call. COBOL still runs over 40% of online banking, 80% of in-person card transactions, and 95% of ATM withdrawals worldwide, so a real jump here matters. But the benchmark details tell a more cautious story.
The Numbers: 5.4% to 97.9%
The COBOL migration eval translates legacy COBOL into Python, Java, Kotlin, C++, or Rust. Earlier models mostly failed; GPT-6.1 Sol clears nearly the whole test set at higher reasoning effort, while the low-effort mode cuts API cost by roughly 61% compared to GPT-5.6 Sol.
The Catch: It's OpenAI's Own Hidden Test Suite
Developers quickly flagged a problem: OpenAI also published MultiLangBench, a public benchmark anyone can rerun on GitHub — but the COBOL migration score comes from OpenAI-designed hidden tests that nobody outside the company can independently verify. The 97.9% figure, for now, is OpenAI grading its own homework.
The Real Risk: Rounding Errors
Engineers reviewing the claim point to a subtler issue: COBOL's fixed-point arithmetic rounds differently than the float operations common in Python or Java. Old code might truncate a third decimal place; migrated code might round it instead. Any test that only checks "close enough" output will miss this — and in a bank or ATM system, that's real money going missing.
Should You Trust It in Production?
- Build your own parity harness to compare old vs. new output byte-for-byte, rather than trusting OpenAI's score alone
- Test against real production data, not just happy-path cases
- Keep human review in the loop — GPT-6.1 Sol is not a fully automated migration solution yet
The COBOL migration improvement is real, but there's still a gap between "97.9% accuracy" and "safe to ship." For banks and insurers, the labor savings may not cover the cost of a single bad reconciliation.
You won't know until you try it.
Sources / 資料來源
- OpenAI says GPT-6.1 Sol is dramatically better at migrating legacy code (BleepingComputer)
- OpenAI says GPT-6 is much better with legacy programming languages such as COBOL and Fortran (The Decoder)
- OpenAI's COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden (DEV Community)
延伸閱讀 / Related Articles
- GPT-6 Pro平行代理評測:多工同時跑,兩百美元方案卻縮水 | GPT-6 Pro Parallel Agents Review: Multitask, Quota Shrinks
- Mercury 3評測:擴散式語言模型,8倍速度打臉GPT-6 | Mercury 3 Review: Diffusion LLM Beats GPT-6 at 8x Speed
- OpenAI募資評測:搶在Anthropic上市前卡位1.4兆美元 | OpenAI Funding Review: Racing Anthropic's $1.4T IPO
AI 工具觀察站 — 每日精選 AI Agent 與工具趨勢
AI Tool Observer — Daily curated AI Agent & tool trends
留言
張貼留言