The organisation's site is usually already there
The website address given at signup is crawled automatically the moment the account is approved, provided the crawl-consent box was ticked on the waitlist form. It becomes the organisation's first knowledge base, and it also gives the Builder a brief on the business, so new agents reflect the organisation from the first draft. Without consent, nothing is crawled: the base shows Consent withheld until the site is added manually.
Add another site
Knowledge → Add knowledge, then the Website tab (the panel opens on
the relevant tab). A bare domain is fine;
https:// is added automatically.
Tick I'm authorised to have this site crawled. The button stays disabled until it is ticked.
The crawl starts immediately, with live progress on the list. See Knowledge bases.
One base per site: adding a host that already exists is refused. How many bases each plan includes is in Plans & limits.
What the crawler reads and skips
The crawler reads the text of the site's pages, then cleans, summarises and
indexes it. It also picks out contact details such as emails and phone numbers as it
goes. On public sites it honours robots.txt. Crawls are capped at a
sensible size, and each base holds up to 100 MB of extracted text.
Password-protected sites
A knowledge base can be behind a login, such as a staff area or an intranet. This site needs a login, ticked when adding the site, reveals the login type: Username & password form or HTTP Basic auth. It also reveals credential fields; form logins can also take a Login page address. If the login fails, the crawl fails with a check-your-credentials error on the row.
The password is stored encrypted, used only by the crawler, and never shown
again. The base carries a Private badge. Because the organisation has
explicitly authorised the content, authenticated crawls do not apply
robots.txt.
Keeping it fresh
Website bases are re-crawled automatically on a schedule. After a site update that should reach agents now, Re-crawl on the row queues a fresh crawl immediately.
Withdrawing consent
Crawl consent is revocable at any time. Deleting the base with Delete, then Confirm delete, withdraws it: everything stored from the crawl is purged permanently, and the site is not crawled again unless it is re-added.