Crawler
What is Crawler?
A program that automatically visits websites and reads the pages.
Also known as: crawl, crawling, crawled, crawlers
Search enginesSearch engine A website used to search the internet, such as Google or Bing. and AIAI Computer programs that perform tasks normally requiring human intelligence, such as writing text or answering questions. companies use crawlers to map the internet. A site-wide crawl means that all pages of a site have been reviewed.
