Web crawler

A web crawler or spider is a computer program that automatically fetches the contents of a web page. The program then analyses the content, for example to index it by certain search terms. Search engines commonly use web crawlers.^[1]

Web Crawler Media

Architecture of a Web crawler
Evolution of Freshness and Age in a web crawler
High-level architecture of a standard Web crawler

Related pages

HTTrack – a web crawler released in 1998

References

↑ Masanès, Julien (February 15, 2007). Web Archiving. Springer. p. 1. ISBN 978-3-54046332-0. Retrieved April 24, 2014.^{[dead link]}

This short article about technology can be made longer. You can help Wikipedia by adding to it.

[GoogleBooks-9237380-1] Masanès, Julien (February 15, 2007). Web Archiving. Springer. p. 1. ISBN 978-3-54046332-0. Retrieved April 24, 2014.^{[dead link]}

[1]