Skip to search formSkip to main contentSkip to account menu

Distributed web crawling

Known as: Distributed crawling, Distributed search, Distributed web crawler 
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web… 
Wikipedia (opens in a new tab)

Related topics

Related topics

6 relations
Distributed computingFAROOInternetRobots exclusion standard

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
Highly Cited
2012
Highly Cited
2012
A Study of Recommending Locations on Location-Based Social Network by Collaborative Filtering
The development of location-based social networking (LBSN) services is growing rapidly these days. Users of LBSN services are… 
Review
2010
Review
2010
WAN-Based Distributed Web Crawling
There are three core issues recognized for WAN-based distributed Web crawling systems: Web Partition, Agent collaboration and… 
Highly Cited
2009
Highly Cited
2009
Analysis of PPLive through active and passive measurements
The P2P-IPTV is an emerging class of Internet applications that is becoming very popular. The growing popularity of these rather… 
2008
2008
On the feasibility of geographically distributed web crawling
We identify the issues that are important in design of a geographically distributed Web crawler. The identified issues are… 
2004
2004
Minimizing the Network Distance in Distributed Web Crawling
Distributed crawling has shown that it can overcome important limitations of the centralized crawling paradigm. However, the… 
2004
2004
Local dominance using polar coordinates to enhance multiobjective evolutionary algorithms
In this paper, we propose a calculation method of local dominance and enhance multiobjective evolutionary algorithms by… 
2004
2004
Distributed location aware web crawling
Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal… 
2002
2002
Fresh Information Retrieval Using Cooperative Meta Search Engines
In web page retrievals, search engines are usually used. However, conventional search engines have a problem in that their update… 
1999
1999
Optimal Distributed Arc-Consistency
This paper presents DisAC-9, the first optimal distributed algorithm performing the arc-consistency of a constraint network. Our… 
1995
1995
An efficient distributed deadlock detection algorithm
We propose a distributed deadlock detection algorithm which detects cycles made of back edges in a distributed search tree. Our… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)