Configuring domains

Often you will find that the content on your website is spread across multiple domains, whether it's a blog subdomain like blog.example.com or a completely different domain like another-example.com. In that case, you can add multiple domains to your Collection.

Add a domain to a Collection

1. Navigate to the Domains section.

2. Click "Add domain" from the top right of the page.

3. Enter the url of the domain you want to add.

4. Ensure the "Crawling" checkbox is checked if you want to crawl and index the content of the domain.

5. Click "Add".

The Crawler will immediately begin indexing pages from the new domain.

Domain configuration

You can configure settings for each domain in the following ways:

Crawling: If enabled, the crawler will periodically visit pages in this domain and update your Collection with any changes. If turned off, the pages on this domain will not be updated in your Collection.

Search from domain: If enabled, search requests coming from this domain are authorized. Any search interface embedded in this domain will be authorized to make search requests to this collection. A common use case when you will only have this turned on is when you want to allow staging websites or testing environments (e.g. Netlify or Codesandbox). In this case, you will add the domain of the testing environment (e.g. testsite.netlify.com) as an additional domain and turn off crawling for that domain.

Turning off crawling for a domain does not remove pages of that domain from the collection. If you want to remove the records from the domain but still keep the domain, you can add an exclude rule via the Crawler rules section.

Remove a domain from a Collection

1. Log in to the Console and select the relevant Collection

2. Navigate to the Domains section.

3. Click the "Delete" button for the domain you want to remove. Note that removing a domain also removes records of that domain from the collection.