Google TurboQuant 實測解析：KV Cache 壓縮到 3-bit，記憶體砍 6 倍還不掉精度 | Google TurboQuant Explained: 3-bit KV Cache Compression Cuts Memory 6x With Zero Accuracy Loss

By Kit 小克 | AI Tool Observer | 2026-04-12

🇹🇼 Google TurboQuant 實測解析：KV Cache 壓縮到 3-bit，記憶體砍 6 倍還不掉精度

你有沒有遇過這種情況？想跑一個長上下文的 LLM，結果 GPU 記憶體直接爆掉？KV Cache 就是那個吃掉你大半 VRAM 的元兇。Google Research 在 ICLR 2026 發表的 TurboQuant，號稱能把 KV Cache 壓到 3-bit，記憶體用量砍 6 倍，而且精度不掉。這麼猛的東西，到底是怎麼做到的？

什麼是 KV Cache？為什麼它是 LLM 推理的瓶頸？

每次 LLM 生成新 token 時，都需要回顧之前所有 token 的注意力資訊，這些資訊就儲存在 KV Cache 裡。上下文越長，KV Cache 就越肥。以 Llama-3.1 8B 為例，在 128K 上下文長度下，KV Cache 可以輕鬆吃掉 8GB 以上的 VRAM——比模型權重本身還多。

這就是為什麼你的 16GB 顯卡跑短對話沒問題，一開長文就當機的原因。

TurboQuant 如何用 3-bit 壓縮解決問題？

TurboQuant 的核心分成兩個階段：

第一階段：PolarQuant 旋轉降維

傳統量化直接把向量的每個數值壓縮，但不同維度的數值分布差異很大，壓了就掉精度。PolarQuant 的做法很聰明：

• 隨機正交旋轉：先把 KV 向量做一次隨機旋轉，讓能量均勻分散到所有維度
• 極座標轉換：把向量從直角座標轉成極座標（半徑 + 角度），省掉昂貴的正規化步驟
• 逐維度量化：旋轉後每個維度的分布變得可預測，就能用標準量化器安全壓縮

第二階段：QJL 誤差修正

Quantized Johnson-Lindenstrauss（QJL）演算法負責處理第一階段留下的殘差誤差。它只用 1-bit（+1 或 -1）來編碼誤差向量，再透過一個特殊的估計器，用高精度的 Query 搭配低精度的壓縮資料來計算注意力分數，達到零額外記憶體開銷。

實測數據有多強？

Google 在 Gemma、Mistral、Llama-3.1 8B 等模型上測試，結果相當驚人：

• 記憶體節省：KV Cache 記憶體用量減少 6 倍以上
• 速度提升：4-bit TurboQuant 在 NVIDIA H100 上的注意力計算速度比 32-bit 快 8 倍
• 精度維持：在 LongBench、Needle In A Haystack、ZeroSCROLLS、RULER、L-Eval 等基準測試中，3-bit 壓縮幾乎沒有精度損失
• 向量搜尋：在 GloVe 資料集上的召回率也優於 PQ 和 RabbiQ 等基線方法

這對一般開發者意味著什麼？

最直接的影響：你的 16GB Mac Mini 可能就能跑百萬 token 上下文的模型了。以前需要 A100 80GB 才能處理的長上下文推理，現在可能只需要一張消費級顯卡。

而且 TurboQuant 不需要重新訓練或微調模型，是純推理端的壓縮技術，直接套用就行。llama.cpp 社群已經在討論整合方案了。

我的觀察

試了才知道好不好用。TurboQuant 最讓我驚豔的不是數字本身，而是它解決問題的思路：與其硬砍數值精度，不如先把資料結構變得「好壓縮」，再壓。這種「旋轉 → 量化 → 誤差修正」的三段式設計，優雅又實用。

不過要注意：目前的實測主要集中在 8B 級別模型。在 70B 以上的超大模型、或是更極端的 2-bit 壓縮場景下表現如何，還需要更多社群驗證。但至少在 KV Cache 壓縮這個方向上，TurboQuant 已經設下了新的標準。

常見問題 FAQ

TurboQuant 和傳統 GPTQ/AWQ 量化有什麼不同？

GPTQ 和 AWQ 主要壓縮模型權重，TurboQuant 專門壓縮推理時的 KV Cache。兩者可以疊加使用——用 AWQ 壓權重，再用 TurboQuant 壓 KV Cache，記憶體省更多。

TurboQuant 需要重新訓練模型嗎？

不需要。TurboQuant 是純推理端技術，不需要任何重新訓練或微調，直接套用在現有模型上就能用。

哪些模型已經支援 TurboQuant？

Google 官方測試了 Gemma、Mistral 和 Llama-3.1 系列。llama.cpp 社群正在討論整合，預期很快會有更廣泛的模型支援。

3-bit 壓縮真的不會掉精度嗎？

在 Google 測試的基準上確實如此，但實際應用中的表現可能因任務類型和上下文長度而異。建議在自己的使用場景中實測驗證。

---

🇺🇸 Google TurboQuant Explained: 3-bit KV Cache Compression Cuts Memory 6x With Zero Accuracy Loss

If you have ever tried running a long-context LLM only to watch your GPU memory explode, you already know the villain: the KV cache. Google Research just presented TurboQuant at ICLR 2026, a compression algorithm that squeezes the KV cache down to 3 bits per element, slashing memory usage by 6x with zero accuracy loss. Here is how it actually works and why it matters.

What Is the KV Cache and Why Does It Bottleneck LLM Inference?

Every time a language model generates a new token, it revisits the attention information from all previous tokens stored in the key-value cache. The longer your context window, the fatter this cache grows. For Llama-3.1 8B at 128K context length, the KV cache alone can consume over 8 GB of VRAM — more than the model weights themselves.

That is why your 16 GB GPU handles short conversations fine but chokes on long documents.

How Does TurboQuant Achieve 3-bit Compression?

TurboQuant operates in two stages that work together elegantly:

Stage 1: PolarQuant Rotation

Standard quantization compresses each dimension directly, but value distributions vary wildly across dimensions, causing accuracy drops. PolarQuant takes a smarter approach:

• Random orthogonal rotation spreads vector energy uniformly across all dimensions
• Polar coordinate conversion replaces Cartesian coordinates with radius and angle, eliminating expensive normalization
• Per-dimension quantization becomes safe because rotated dimensions follow predictable distributions

Stage 2: QJL Error Correction

The Quantized Johnson-Lindenstrauss (QJL) algorithm handles residual errors from Stage 1. It encodes each error vector using just 1 bit (+1 or −1) and employs a special estimator that pairs high-precision queries with low-precision compressed data to compute attention scores at zero additional memory overhead.

What Do the Benchmarks Show?

Google tested TurboQuant on Gemma, Mistral, and Llama-3.1 8B across multiple benchmarks:

• Memory savings: 6x or greater reduction in KV cache memory
• Speed gains: 4-bit TurboQuant achieves up to 8x faster attention computation versus 32-bit on NVIDIA H100
• Accuracy preservation: Negligible quality loss on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval
• Vector search: Superior recall compared to PQ and RabbiQ baselines on the GloVe dataset

What Does This Mean for Developers?

The most immediate impact: a 16 GB Mac Mini could potentially run million-token context models. Long-context inference that previously demanded an A100 80 GB may soon fit on a consumer GPU.

TurboQuant requires no retraining or fine-tuning — it is a pure inference-time compression technique you apply directly. The llama.cpp community is already discussing integration.

My Take

You never know until you try. What impresses me most about TurboQuant is not the numbers but the design philosophy: instead of brute-force precision reduction, first transform the data structure to be compression-friendly, then compress. This rotate-quantize-correct pipeline is both elegant and practical.

One caveat: current benchmarks focus on 8B-class models. Performance on 70B+ models or more extreme 2-bit compression still needs community validation. But for KV cache compression, TurboQuant has set a new bar.

FAQ

How Is TurboQuant Different from GPTQ or AWQ?

GPTQ and AWQ compress model weights. TurboQuant compresses the inference-time KV cache. You can stack both — use AWQ for weights and TurboQuant for the cache — for even greater memory savings.

Does TurboQuant Require Retraining?

No. It is a pure inference-side technique that works out of the box on existing models without any retraining or fine-tuning.

Which Models Support TurboQuant?

Google officially tested Gemma, Mistral, and Llama-3.1. The llama.cpp community is working on broader integration, so wider model support is expected soon.

Does 3-bit Compression Really Preserve Accuracy?

On Google's benchmarks, yes. However, real-world results may vary depending on task type and context length. Always validate on your own use cases.

Sources / 資料來源

• Google Research Blog - TurboQuant
• Tom's Hardware - TurboQuant 6x Memory Reduction
• llama.cpp Discussion - TurboQuant Integration

常見問題 FAQ

TurboQuant 和傳統 GPTQ/AWQ 量化有什麼不同？

GPTQ 和 AWQ 壓縮模型權重，TurboQuant 專門壓縮推理時的 KV Cache，兩者可以疊加使用。

TurboQuant 需要重新訓練模型嗎？

不需要，TurboQuant 是純推理端技術，直接套用在現有模型上即可。

哪些模型支援 TurboQuant？

Google 測試了 Gemma、Mistral、Llama-3.1，llama.cpp 社群正在整合更多模型支援。

3-bit 壓縮真的不會掉精度嗎？

在 Google 基準測試中幾乎無損，但實際表現可能因任務和上下文長度而異，建議自行驗證。

延伸閱讀 / Related Articles

• Google AI Edge Gallery 實測：手機離線跑 Gemma 4，AI Agent 不用網路也能動 | Google AI Edge Gallery Hands-On: Running Gemma 4 Offline on Your Phone — AI Agents Without Internet
• Google Gemma 4 開源模型實測：31B 打贏 400B 對手，手機也能跑 AI Agent | Google Gemma 4 Hands-On: A 31B Open Model That Beats 400B Rivals and Runs on Your Phone
• GLM-5.1 開源模型實測：754B 參數跑 8 小時不停，SWE-Bench Pro 登頂的真相 | GLM-5.1 Hands-On: The 754B Open-Source Model That Codes for 8 Hours Straight

---

AI 工具觀察站 — 每日精選 AI Agent 與工具趨勢
AI Tool Observer — Daily curated AI Agent & tool trends