# Excluding documents
{% hint style="info" %}
**Crawler Only**
Unlike search settings, documents excluded via the exclude rules are not stored in the search index. Therefore changes to the rules require a re-index and are not realtime. We generally recommend to instead filter documents via [Search settings](/documentation/fundamentals/search-settings.md).
{% endhint %}
To start, head to the [Crawler rules](https://app.search.io/collection/crawler/rules) section.
* Click the ‘Create rule’ button in the top right, then choose the ‘Exclude’ option.
* Select the field for your exclude rule (e.g. URL, domain, directory). The fields in the dropdown are generated based on the data structure defined in your [Schema](https://docs.search.io/documentation/fundamentals/indexing-data/schema).
* Select an operator to define the condition (e.g. contains, equals, does not equal, etc.) for the exclude rule.
* Enter a value for the field that you want to exclude. This may be a directory name, a domain, a full URL, or something else that defines the content you want to exclude. See the examples below.
{% hint style="info" %}
Exclude rules may take between 20 mins to a few hours to process.
{% endhint %}
### Excluding a specific URL
To exclude a specific URL:
* Select 'URL' from the first dropdown.
* Select 'Equals' from the second dropdown.
* Enter the full URL in the third field, e.g. *"*[*www.example.com/articles/bad-news.html"*](http://www.example.com/articles/bad-news.html%22)
* Click 'Create Rule'.
### Exclude a directory
You can exclude specific sections of your website to ensure they don't appear in search results.
For example, if you don't want to index the blog content under [*"www.example.com/news/blog"*](http://www.example.com/news/blog), you can set a rule to ignore everything under the *Second directory* if the name of that directory is **blog**.
* Select "Second directory" (also referred as 'dir2') field.
* Select 'Equals' from the second dropdown.
* Enter the directory name (e.g. "blog").
* Click 'Create Rule'.
### Exclude a file type
You can exclude specific file types from your search results by targeting the filename extension at the end of the URL e.g. PDF, DOC.
For example, to exclude all PDFs from your search results:
* Select the URL field.
* Select 'Ends with'.
* Add ‘.pdf’ in the text field.
* Click 'Create Rule'.
### Using exclude to only index a certain part of your website
To only make a specific section of your website searchable, you can use the **does not equal** condition to any content that not within that section: For example, to only index the documentation within [*"www.example.com/docs"*](http://www.example.com/docs), you can set up the following rule:
* Select the URL field.
* Select 'First directory'.
* Select 'Does not equal'.
* Add ‘docs' in the text field.
* Click 'Create Rule'.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.search.io/documentation/guides/content-websites/excluding-documents.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.