Skip to search form
Skip to main content
Skip to account menu
Semantic Scholar
Semantic Scholar's Logo
Search 236,830,544 papers from all fields of science
Search
Sign In
Create Free Account
Distributed web crawling
Known as:
Distributed crawling
, Distributed search
, Distributed web crawler
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web…
Expand
Wikipedia
(opens in a new tab)
Create Alert
Alert
Related topics
Related topics
6 relations
Distributed computing
FAROO
Internet
Robots exclusion standard
Expand
Papers overview
Semantic Scholar uses AI to extract papers important to this topic.
2019
2019
Grappler: Array of Bistable Elements For Pinching Net-Like Infrastructure to Low Gravity Bodies
Juliana Cherston
,
Paul Strohmeier
,
J. Paradiso
AIAA Scitech Forum
2019
Corpus ID: 92987336
Near-Earth asteroids and comets are ubiquitous and are key destinations for studying the formation of our solar system. These…
Expand
2016
2016
Improvised Architecture for Distributed Web Crawling
Tilak Patidar
,
Aditya Ambasth
2016
Corpus ID: 19500551
Web crawlers are program, designed to fetch web pages for information retrieval system. Crawlers facilitate this process by…
Expand
2015
2015
A Study on Distributed Crawling-Based Overhead Optimization
Hanah Kim
,
Jongwon Lee
,
Tae-Hak Ban
,
Hoe-Kyung Jung
2015
Corpus ID: 111202631
Network technology has been in the process of development with IT. Data on the network is increasing rapidly and there are also…
Expand
2013
2013
An enhanced MOEA/D using uniform directions and a pre-organization procedure
Rui Wang
,
Zhang Tao
,
Bo Guo
IEEE Congress on Evolutionary Computation
2013
Corpus ID: 35958145
Multi-objective evolutionary algorithm based on decomposition (MOEA/D) has become increasingly popular in solving multi-objective…
Expand
Review
2010
Review
2010
WAN-Based Distributed Web Crawling
Xu Xiao
2010
Corpus ID: 63778309
There are three core issues recognized for WAN-based distributed Web crawling systems: Web Partition, Agent collaboration and…
Expand
2010
2010
Exploiting Location-aware Mechanism for Distributed Web Crawling over DHTs
Xiao Xu
,
Weizhe Zhang
,
Hongli Zhang
,
Binxing Fang
Journal of Computers
2010
Corpus ID: 32259722
Inspired by the concept of internet computing, DHT-based distributed Web crawling model is proposed to solve the bottlenecks of…
Expand
2009
2009
DESIGN AND IMPLEMENTATION OF DISTRIBUTED WEB-CRAWLING SYSTEM
Zhang Liang
,
W. Chun
2009
Corpus ID: 112788910
In this paper,the auther analyzed the principle of Tiny Search Engine,an experimental search engine provided by the Net Lab of…
Expand
2008
2008
Toward developing an optimal cooperative search algorithm for multiple unmanned aerial vehicles
P. DeLima
,
D. Pack
International Symposium on Collaborative…
2008
Corpus ID: 12480703
In search and surveillance operations, a team of small unmanned aerial vehicles (UAVs) can provide a robust solution that…
Expand
2007
2007
PUCC Architecture, Protocols and Applications
H. Sumino
,
N. Ishikawa
,
Shingo Murakami
,
Takeshi Kato
,
J. Hjelm
4th IEEE Consumer Communications and Networking…
2007
Corpus ID: 18801584
Compared with traditional Internet technologies, peer-to-peer has functions to realize resource discovery, resource sharing, and…
Expand
2004
2004
On the Structure of Sequential Search: Beyond "No Free Lunch"
T. M. English
EvoCOP
2004
Corpus ID: 6974257
Novel results are obtained by investigating the search algorithms predicated in “no free lunch” (NFL) theorems rather than NFL…
Expand