Website Crawler
The Website Crawler integration enables your AI workers to automatically extract and index content from public websites. It respects robots.txt, supports sitemaps, and converts web content to searchable knowledge documents.
What You Can Do
- 1Ingest public website content for knowledge base
- 2Research competitor websites and documentation
- 3Extract product information from catalogs
- 4Monitor public pages for changes
- 5Build training datasets from public sources
Data Access & Privacy
CreateWorker only accesses the data it needs to complete tasks. All access is logged and auditable.
- •Public website content (text only)
- •Page titles and metadata
- •Internal link structure
Approval Workflow
- ✓All actions require human approval by default
- ✓Every action is logged to your audit trail
- ✓Revoke access at any time from settings