A traditional web crawler (aka web spider or web robot) is a program that automatically and robotically browses the internet to do a variety of things, such as:
spider programs for search engines where a copy of all the visited pages are stored for later processing or indexing,
automatic maintenance tasks, or harvesting email addresses for […]