Efficient Crawling Through URL Ordering


In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this… (More)
DOI: 10.1016/S0169-7552(98)00108-1
View Slides



Citations per Year

908 Citations

Semantic Scholar estimates that this publication has 908 citations based on the available data.

See our FAQ for additional information.