Skip to search formSkip to main contentSkip to account menu

Web crawler

Known as: Webcrawler, Crawl site, RBSE 
A Web crawler is an Internet bot which systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). Web… 
Wikipedia (opens in a new tab)

Related topics

Related topics

50 relations
Ajax (programming)Apache HadoopApache NutchApache Storm

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
Review
2013
Review
2013
Web Crawler: A Review
Information Retrieval deals with searching and retrieving information within the documents and it also searches the online… 
Highly Cited
2010
Highly Cited
2010
A focused crawler for Dark Web forums
The unprecedented growth of the Internet has given rise to the Dark Web, the problematic facet of the Web associated with… 
Highly Cited
2009
Highly Cited
2009
Web Crawler Architecture
  • Marc Najork
  • Encyclopedia of Database Systems
  • 2009
  • Corpus ID: 29031380
Definition A web crawler is a program that, given one or more seed URLs, downloads the web pages associated with these URLs… 
Highly Cited
2008
Highly Cited
2008
Investigating web services on the world wide web
Searching for Web service access points is no longer attached to service registries as Web search engines have become a new major… 
Highly Cited
2007
Highly Cited
2007
A Measurement Study of a Large-Scale P2P IPTV System
An emerging Internet application, IPTV, has the potential to flood Internet access and backbone ISPs with massive amounts of new… 
Highly Cited
2002
Highly Cited
2002
Design and implementation of a high-performance distributed Web crawler
Broad Web search engines as well as many more specialized search tools rely on Web crawlers to acquire large collections of pages… 
Highly Cited
2001
Highly Cited
2001
A web crawler design for data mining
The content of the web has increasingly become a focus for academic research. Computer programs are needed in order to conduct… 
Highly Cited
2001
Highly Cited
2001
Peer-to-peer architecture case study: Gnutella network
  • M. Ripeanu
  • Proceedings First International Conference on…
  • 2001
  • Corpus ID: 444337
Despite recent excitement generated by the P2P paradigm and despite surprisingly fast deployment of some P2P applications, there… 
Review
2000
Review
2000
SPSS 13.0 Guide to Data Analysis
The SPSS 16.0 Guide to Data Analysis is a friendly introduction to both data analysis and SPSS, the worlds leading desktop… 
Highly Cited
1999
Highly Cited
1999
Mercator: A scalable, extensible Web crawler
This paper describes Mercator, a scalable, extensible Web crawler written entirely in Java. Scalable Web crawlers are an… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)