Skip to search formSkip to main contentSkip to account menu

Distributed web crawling

Known as: Distributed crawling, Distributed search, Distributed web crawler 
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web… 
Wikipedia (opens in a new tab)

Related topics

Related topics

6 relations
Distributed computingFAROOInternetRobots exclusion standard

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
2016
2016
Improvised Architecture for Distributed Web Crawling
Web crawlers are program, designed to fetch web pages for information retrieval system. Crawlers facilitate this process by… 
Review
2010
Review
2010
WAN-Based Distributed Web Crawling
There are three core issues recognized for WAN-based distributed Web crawling systems: Web Partition, Agent collaboration and… 
Highly Cited
2009
Highly Cited
2009
Analysis of PPLive through active and passive measurements
The P2P-IPTV is an emerging class of Internet applications that is becoming very popular. The growing popularity of these rather… 
2008
2008
On the feasibility of geographically distributed web crawling
We identify the issues that are important in design of a geographically distributed Web crawler. The identified issues are… 
2004
2004
Minimizing the Network Distance in Distributed Web Crawling
Distributed crawling has shown that it can overcome important limitations of the centralized crawling paradigm. However, the… 
2004
2004
Distributed location aware web crawling
Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal… 
2003
2003
Temporal Ranking for Fresh Information Retrieval
In business, the retrieval of up-to-date, or fresh, information is very important. It is difficult for conventional search… 
2002
2002
Fresh Information Retrieval Using Cooperative Meta Search Engines
In web page retrievals, search engines are usually used. However, conventional search engines have a problem in that their update… 
2001
2001
A dynamic reconfiguration model for a distributed web crawling system
A web crawling system employing a parallel and distributed architecture needs to have a mechanism to bring the whole system in a… 
1995
1995
An efficient distributed deadlock detection algorithm
We propose a distributed deadlock detection algorithm which detects cycles made of back edges in a distributed search tree. Our… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)