Select pages and update knowledge
Understand page selection, searchable content, refreshes, and source status messages.
On this page
Discovered, selected, and searchable are different
| Term | Meaning |
|---|---|
| Discovered pages | Public URLs found within the source scope, including some URLs whose content could not be read. |
| Selected pages | URLs included by the current page and section rules. This is your desired knowledge scope. |
| Searchable pages | Content currently present in the assistant's search index. This can include a previously indexed version. |
Duplicate content, canonical URLs, unreadable pages, indexing work, and pending removals can make these counts differ. A checked box alone does not confirm that a page is searchable or that an older indexed copy has already been removed.
Select individual pages or sections
Open a source in Knowledge. Expand the page hierarchy and select or exclude an individual page or an entire section. Section choices apply to pages below that path, including newly discovered pages. A more specific page or section rule can override a broader choice. Changing a section selection replaces overrides within that section.
New top-level paths are included by default. New pages below an excluded section inherit that exclusion. A section is a path group; it need not itself be a webpage.
Activate a ready source
For a source awaiting configuration, review the selection and choose Activate source. Activation uses the retained discovery result to prepare selected readable content for search. If those discovery files are unavailable, discover the website again first.
Rediscover pages or crawl now?
Rediscover pages updates the visible URL inventory. For an active source, rediscovery alone does not refresh the assistant's indexed content.
Crawl now rediscovers the public pages and refreshes selected knowledge content. Use it after changing page selection or publishing content you want the assistant to use. Active sources also refresh daily. Check Next refresh, Last attempt, and individual page statuses rather than expecting immediate updates after a website edit.
Discovery and indexing run asynchronously and can take time, especially on a large website. You can leave the portal page while work continues. A source can have healthy searchable pages alongside pages needing attention.
Understand page results
| Page status | What it means |
|---|---|
| Searchable | The assistant can search an indexed version of this page. |
| Indexing or Updating | Preparation or replacement is still in progress. |
| Could not read; not searchable | Fetching or extracting the page did not produce usable new content. |
| Could not read update; previous version searchable | The assistant can still use older content. |
| Update failed; older version searchable | Replacement failed after an older version was indexed. |
Handle incomplete or unsuccessful refreshes
Fix accessibility or source-content problems, then run Crawl now again. For robots denial, allow CloudflareBrowserRenderingCrawler in the source website's robots.txt. An incomplete crawl does not establish that every omitted page has been deleted from the website.
After updating important information, check the page status and test the assistant's answer.