Skip to search formSkip to main contentSkip to account menu

Distributed web crawling

Known as: Distributed crawling, Distributed search, Distributed web crawler 
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web… 
Wikipedia (opens in a new tab)

Related topics

Related topics

6 relations
Distributed computingFAROOInternetRobots exclusion standard

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
2019
2019
Grappler: Array of Bistable Elements For Pinching Net-Like Infrastructure to Low Gravity Bodies
Near-Earth asteroids and comets are ubiquitous and are key destinations for studying the formation of our solar system. These… 
2016
2016
Improvised Architecture for Distributed Web Crawling
Web crawlers are program, designed to fetch web pages for information retrieval system. Crawlers facilitate this process by… 
2013
2013
An enhanced MOEA/D using uniform directions and a pre-organization procedure
Multi-objective evolutionary algorithm based on decomposition (MOEA/D) has become increasingly popular in solving multi-objective… 
Review
2010
Review
2010
WAN-Based Distributed Web Crawling
There are three core issues recognized for WAN-based distributed Web crawling systems: Web Partition, Agent collaboration and… 
2010
2010
Exploiting Location-aware Mechanism for Distributed Web Crawling over DHTs
Inspired by the concept of internet computing, DHT-based distributed Web crawling model is proposed to solve the bottlenecks of… 
Highly Cited
2009
Highly Cited
2009
Analysis of PPLive through active and passive measurements
The P2P-IPTV is an emerging class of Internet applications that is becoming very popular. The growing popularity of these rather… 
2009
2009
DESIGN AND IMPLEMENTATION OF DISTRIBUTED WEB-CRAWLING SYSTEM
In this paper,the auther analyzed the principle of Tiny Search Engine,an experimental search engine provided by the Net Lab of… 
2008
2008
On the feasibility of geographically distributed web crawling
We identify the issues that are important in design of a geographically distributed Web crawler. The identified issues are… 
2007
2007
PUCC Architecture, Protocols and Applications
Compared with traditional Internet technologies, peer-to-peer has functions to realize resource discovery, resource sharing, and… 
2004
2004
On the Structure of Sequential Search: Beyond "No Free Lunch"
Novel results are obtained by investigating the search algorithms predicated in “no free lunch” (NFL) theorems rather than NFL… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)