Fix a blocked knowledge page fetch — AI Website Chat Agent

What to do when a page cannot be refetched because your website security blocks the knowledge crawler.

Why this happens

Some websites protect public pages with Cloudflare, a WAF, bot protection, rate limits, or robots.txt rules. When that protection blocks our knowledge fetcher, we keep the URL as URL-only and do not use it as grounded assistant knowledge until the page text can be fetched or uploaded.

What the portal tried

One selected public page was requested, not a broad crawl.
Browser-like request headers were used.
The page was left as URL-only instead of being marked as indexed content.

Send this to your web developer/host

Please allow the AI Website Chat Agent crawler/server to fetch the selected public help/product pages without a Cloudflare/WAF challenge. Keep admin, checkout, account, and private pages blocked. If you need exact crawler/server details, ask the site owner to request them from AI Website Chat Agent support.

How to fix Cloudflare or WAF blocks

Allowlist the AI Website Chat Agent crawler/server in Cloudflare, your host firewall, or your bot-protection tool for the public help/product pages you want indexed.
In Cloudflare, create a WAF/Bot rule that skips managed challenge/Bot Fight Mode for the allowed crawler/IP/path, or temporarily lower protection for the selected public pages while importing.
Keep admin, checkout, account, and private pages blocked; only allow public knowledge pages.

If allowlisting is not possible

Upload an export/file containing the page text.
Add manual Q&A content.
Import a product feed for catalogue data.
Ask support for the current crawler/server details to pass to your web developer or hosting provider.