Crawling the Infinite Web: Five Levels Are Enough

  title={Crawling the Infinite Web: Five Levels Are Enough},
  author={Ricardo A. Baeza-Yates and Carlos Castillo},
A large amount of publicly available Web pages are generated dynamically upon request, and contain links to other dynamically generated pages. This usually produces Web sites which can create arbitrarily many pages. This poses a problem to search engine managers: they need a rule to configure the search engine’s crawler in such a way that it stops downloading pages from each Web site at some depth. But how deep must the crawler go? In this article, several probabilistic models for browsing… CONTINUE READING
Highly Cited
This paper has 52 citations. REVIEW CITATIONS

12 Figures & Tables



Citations per Year

53 Citations

Semantic Scholar estimates that this publication has 53 citations based on the available data.

See our FAQ for additional information.