Skip to search formSkip to main contentSkip to account menu

Web crawler

Known as: Webcrawler, Crawl site, RBSE 
A Web crawler is an Internet bot which systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). Web… 
Wikipedia (opens in a new tab)

Related topics

Related topics

50 relations
Ajax (programming)Apache HadoopApache NutchApache Storm

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
2011
2011
Design and Implementation of Scalable, Fully Distributed Web Crawler for a Web Search Engine
Web is a context in which traditional Information Retrieval methods are challenged. Given the volume of the Web and its speed of… 
2009
2009
Implementation of Web Crawler
The World Wide Web is an interlinked collection of billions of documents formatted using HTML. Ironically the very size of this… 
Highly Cited
2008
Highly Cited
2008
DistanceRank: An intelligent ranking algorithm for web pages
Highly Cited
2007
Highly Cited
2007
Mining templates from search result records of search engines
Metasearch engine, Comparison-shopping and Deep Web crawling applications need to extract search result records enwrapped in… 
Highly Cited
2006
Highly Cited
2006
A Crawler-based Study of Spyware in the Web
Malicious spyware poses a significant threat to desktop security and integrity. This paper examines that threat from an Internet… 
Highly Cited
2006
Highly Cited
2006
Development of a biomimetic miniature robotic crawler
The paper presents the development of segmented artificial crawlers endowed with passive hook-shaped frictional microstructures… 
Highly Cited
2005
Highly Cited
2005
DR-NEGOTIATE - a system for automated agent negotiation with defeasible logic-based strategies
Highly Cited
2004
Highly Cited
2004
The use of guidelines to automatically verify Web accessibility
Accessibility is one of the key challenges that the Internet must currently face to guarantee universal inclusion. Accessible Web… 
Highly Cited
2003
Highly Cited
2003
Development of "Souryu I & II" -Connected Crawler Vehicle for Inspection of Narrow and Winding Space
Highly Cited
2001
Highly Cited
2001
MODELING PEER-TO-PEER NETWORK TOPOLOGIES THROUGH “ SMALL-WORLD ” MODELS AND POWER LAWS
I INTRODUCTION The recent emergence of novel network applications such as Gnutella, Freenet, and Napster has reincarnated the… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)