Website Crawler Integration
Search the public web and crawl websites to extract content for research and knowledge ingestion.
Capabilities
This integration enables workers to:
- Search the public web for current information (ranked results: title, url, snippet)
- Crawl single pages or entire websites
- Extract text content in markdown format
- Respect robots.txt directives
- Domain allowlist for security
- Configurable depth and page limits
- Rate limiting to be respectful to target sites
Setup
Follow these steps to connect Website Crawler:
- Navigate to Integrations and click 'Add Crawler'
- Configure allowed host domains
- Set default crawl parameters (depth, max pages)
- Test the crawler with a sample URL
- Assign workers to process crawled content
Security
- SSRF protection blocks access to private IP ranges
- Only configured domains can be crawled
- Respects robots.txt by default
- Rate limited to 1 request per second
- All crawl operations are audit logged
Credential Storage
All integration credentials are encrypted at rest using AES-256-GCM encryption. OAuth tokens are automatically refreshed when needed.
Supported Roles
The following roles can use this integration by default: