URLs and crawling
Add a single page or crawl a whole site, what the crawler follows and skips, and how to re-crawl when the content changes.
Content that already lives on your website does not need to be re-typed. Add it as a URL source, either one page at a time or as a site crawl.
A single URL
Choose Add URL, paste the address, and optionally give it a title. evoriqa fetches the page, extracts its readable text, and processes it like any other source.
One URL source is one page. It does not follow links.
Crawl a site
Choose Crawl website and give the crawler a root URL. It discovers pages and creates one URL source per page.
| Behaviour | Detail |
|---|---|
| Discovery | Reads /sitemap.xml if present, then follows links breadth-first |
| Scope | Same origin as the root URL only |
| Page cap | Up to 50 pages per crawl (the default and the maximum) |
| Ordering | Pages that look high-value — faq, help, support, pricing, about, services, contact, and similar — are crawled first, so a capped crawl gets the useful pages |
| Content types | HTML and XML only; PDFs, images, and JSON endpoints are skipped |
| Duplicates | A URL this chatbot already has is skipped, never added twice |
| Rate limit | Up to 10 crawls per workspace per minute |
Every fetch — the first crawl and every later re-crawl — goes through a protected fetcher that refuses private and internal addresses, re-checks on every redirect hop, and caps the response size.
A crawl consumes one knowledge source per page, from the same plan allowance as your files and FAQs. If the crawl reaches your source limit part-way through, it stops there and keeps what it created rather than failing the whole run. Check the source count afterwards.
Re-crawling
Two different things refresh a URL source, and they are worth keeping apart:
- Re-crawl the site — run discovery again on the root URL. This is
additive: it creates sources for pages that appeared since last time, and never removes ones that disappeared. Use it when your site gained pages.
- Refresh one page — a single URL source re-fetches on its own cadence, or
immediately when you reprocess it. Use it when a page's content changed.
The second is covered in full on Reindexing and freshness, including the per-source daily/weekly/monthly cadence and the Stale badge.
When a crawl brings back nothing
- The page renders its content with JavaScript. The crawler reads the HTML
the server returns; a client-rendered page often has no text in it.
- The URL redirects off-origin. Discovery stays on the root's origin.
- The page is behind a login. Only publicly reachable pages can be fetched —
paste the content as manual text instead.
- The site has no sitemap and few internal links. Add the important pages as
single URLs.
Where to go next
Last updated