Cloudflare AI爬蟲封鎖評測：Agent預設被擋，9/15上路 | Cloudflare AI Crawler Review: Agents Blocked by Default

9月 15, 2026

By Kit 小克 | AI Tool Observer | 2026-09-16

🇹🇼 Cloudflare AI爬蟲封鎖評測：Agent預設被擋，9/15上路

Cloudflare 從 2026 年 9 月 15 日起，正式把「Training」與「Agent」類型的 AI 爬蟲 在有打廣告的頁面上預設封鎖，只留「Search」類爬蟲維持預設放行。這是 7 月公告的政策正式生效，直接影響所有仰賴通用爬蟲抓網頁的 AI Agent 與 RAG pipeline。

Cloudflare AI爬蟲新政策：擋什麼、放什麼

Cloudflare 把 AI 爬蟲分成三類：Search（搜尋索引用）、Training（訓練模型用）、Agent（代理即時抓取用）。9 月 15 日起，只要網站有廣告，Training 與 Agent 這兩類預設就被擋在門外，只有 Search 類維持原本的放行預設。適用對象是所有新網站，以及既有的免費方案客戶；已經自行設定過規則的付費客戶則維持原狀不受影響。

誰會被這波 Cloudflare 封鎖打到

自己寫 RAG 或 agent 去抓網頁內容的開發者：抓 Cloudflare 站台的成功率會明顯下降
沒特別調整設定的免費方案網站主：AI agent 現在預設抓不到你的內容
靠通用 user agent 偽裝瀏覽器的抓取腳本：Cloudflare 就是靠 user agent 分類分流，偽裝身分的抓取風險更高

Pay Per Crawl 正在變成 Pay Per Use

Cloudflare 同時把原本的 Pay Per Crawl 方案往「Pay Per Use」調整：不是抓到頁面就收費，而是 AI 真的把你的內容用進答案裡才計費。目前公開參與的只有做 AI 搜尋引擎 Gist.AI 的新創 ProRata，規模還很小，短期內對大部分開發者的實質影響有限。

AI Agent 開發者現在該做什麼

如果你的 agent 或 RAG pipeline 需要穩定抓取特定網站，這波政策給的訊號很清楚：

先確認目標網域是不是掛在 Cloudflare 後面，抓取失敗率有沒有突然升高
user agent 誠實標示身分，不要冒充瀏覽器或搜尋引擎，長期風險更高
真的需要穩定資料來源，考慮走官方 API 或付費爬取管道，別賭對方沒設防

這不是要把 AI agent 擋死，而是把「白吃流量」的模式往「付費用量」推。短期會讓依賴通用爬蟲的 pipeline 穩定性下降，中長期則可能讓內容取得更透明、但也更貴。

好不好用，試了才知道。

🇺🇸 Cloudflare AI Crawler Review: Agents Blocked by Default

Starting September 15, 2026, Cloudflare now blocks Training and Agent category AI crawlers by default on any page that runs ads — only Search crawlers stay allowed by default. The policy, announced back in July, just went live, and it directly hits any AI agent or RAG pipeline that leans on generic web crawling.

What Cloudflare's AI Crawler Block Actually Changes

Cloudflare splits AI bots into three buckets: Search (crawlers that index content to answer queries), Training (crawlers pulling data to train or fine-tune models), and Agent (crawlers fetching pages live for an AI agent's task). As of September 15, Training and Agent crawlers are blocked by default on any ad-supported page. Search crawlers keep their existing default of being allowed. This applies to all new sites and existing free-tier customers; paid customers who already configured their own rules are unaffected.

Who Actually Gets Hit

Developers running RAG or agent pipelines that scrape the open web will see crawl success rates drop on Cloudflare-fronted domains
Free-tier site owners who never touched their bot settings now block AI agents from their content by default
Scripts that spoof a browser user agent instead of identifying honestly — Cloudflare's whole system sorts traffic by user agent, so spoofing carries more risk going forward

Pay Per Crawl Is Turning Into Pay Per Use

Cloudflare is also reshaping its Pay Per Crawl program into "Pay Per Use" — publishers get paid when an AI actually cites their content in an answer, not just when a bot fetches the page. So far the only publicly named participant is ProRata, the startup behind the AI search engine Gist.AI. It's still small-scale, so the near-term practical impact for most developers is limited.

What AI Agent Builders Should Do Now

If your agent or RAG pipeline depends on reliably crawling specific sites, the signal here is clear:

Check whether your target domains sit behind Cloudflare and whether crawl failure rates just spiked
Identify your crawler honestly in the user agent string instead of impersonating a browser or search engine
For sources you actually depend on, look at official APIs or paid crawl access instead of betting the site left the door open

This isn't an attempt to kill AI agents outright — it's Cloudflare pushing the free-riding model toward pay-per-use. Expect short-term instability for pipelines built on generic crawling, and longer-term, more transparent but more expensive content access.

You won't know until you try it.

Sources / 資料來源

延伸閱讀 / Related Articles

AI 工具觀察站 — 每日精選 AI Agent 與工具趨勢
AI Tool Observer — Daily curated AI Agent & tool trends

搜尋此網誌

AI小貼士