Skip to search formSkip to main contentSkip to account menu

Distributed web crawling

Known as: Distributed crawling, Distributed search, Distributed web crawler 
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web… 
Wikipedia (opens in a new tab)

Related topics

Related topics

6 relations
Distributed computingFAROOInternetRobots exclusion standard

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
Highly Cited
2012
Highly Cited
2012
A Study of Recommending Locations on Location-Based Social Network by Collaborative Filtering
The development of location-based social networking (LBSN) services is growing rapidly these days. Users of LBSN services are… 
Review
2010
Review
2010
WAN-Based Distributed Web Crawling
There are three core issues recognized for WAN-based distributed Web crawling systems: Web Partition, Agent collaboration and… 
Highly Cited
2009
Highly Cited
2009
Analysis of PPLive through active and passive measurements
The P2P-IPTV is an emerging class of Internet applications that is becoming very popular. The growing popularity of these rather… 
2008
2008
On the feasibility of geographically distributed web crawling
We identify the issues that are important in design of a geographically distributed Web crawler. The identified issues are… 
2004
2004
Minimizing the Network Distance in Distributed Web Crawling
Distributed crawling has shown that it can overcome important limitations of the centralized crawling paradigm. However, the… 
2004
2004
Local dominance using polar coordinates to enhance multiobjective evolutionary algorithms
In this paper, we propose a calculation method of local dominance and enhance multiobjective evolutionary algorithms by… 
2004
2004
Distributed location aware web crawling
Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal… 
2004
2004
On the Structure of Sequential Search: Beyond "No Free Lunch"
Novel results are obtained by investigating the search algorithms predicated in “no free lunch” (NFL) theorems rather than NFL… 
2003
2003
Temporal Ranking for Fresh Information Retrieval
In business, the retrieval of up-to-date, or fresh, information is very important. It is difficult for conventional search… 
1999
1999
Optimal Distributed Arc-Consistency
This paper presents DisAC-9, the first optimal distributed algorithm performing the arc-consistency of a constraint network. Our… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)