Turn a website or webpage into searchable content in Vectara
The web crawler currently has 4 modes of operation:
- Single URL: provide the crawler with a URL and it will ingest it into Vectara.
- Sitemap: provide the crawler with a root page, and it will retrieve the sitemap(s) and index all links from the sitemap.
- RSS: provide the crawler with an RSS feed URL and it will find all of the direct links on the feed. It can be used to periodically sync content published there.
- Recursive: the most comprehensive mode; recursively attempting to find links on its own.