Skip to search formSkip to main contentSkip to account menu

Robots exclusion standard

Known as: Robot Exclusion Protocol, Robots exclusion protocol, Robots exclusion file 
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with… 
Wikipedia (opens in a new tab)

Related topics

Related topics

24 relations
.htaccessApache NutchAutomated Content Access ProtocolDistributed web crawling

Broader (1)

World Wide Web

Papers overview

Semantic Scholar uses AI to extract papers important to this topic.
Highly Cited
2017
Highly Cited
2017
RCrawler: An R package for parallel web crawling and scraping
2012
2012
Hotel Information Exposure in Cyberspace: The Case of Hong Kong
Search engines are an everyday tool for Internet surfing. They are also a critical factor that affects e-business performance… 
2009
2009
Copyright and Copy-Reliant Technology
This article studies the rise of copy-reliant technologies - technologies such as Internet search engines and plagiarism… 
Review
2008
Review
2008
A larger scale study of robots.txt
A website can regulate search engine crawler access to its content using the robots exclusion protocol, specified in its robots… 
2007
2007
The North Carolina State Government Website Archives: A case study of an American government Web archiving project
The North Carolina State Archives and State Library of North Carolina collaborated to develop the North Carolina State Government… 
Highly Cited
2006
Highly Cited
2006
Academic Data Collection in Electronic Environments: Defining Acceptable Use of Internet Resources
Academic researchers access commercial web sites to collect research data. This research practice is likely to increase. Is this… 
2006
2006
ANALYSIS OF THE USAGE STATISTICS OF ROBOTS EXCLUSION STANDARD
Robots Exclusion standard [4] is a de-facto standard that is used to inform the crawlers, spiders or web robots about the… 
2005
2005
Static Analysis of Programs Using Omega Algebra with Tests
Recently, Kozen has proposed a framework based on Kleene algebra with tests for verifying that a program satisfies a security… 
Highly Cited
2004
Highly Cited
2004
Discovery of Web Robot Sessions Based on their Navigational Patterns
Web robots are software programs that automatically traverse the hyperlink structure of the World Wide Web in order to locate and… 
1999
1999
CoBWeb-a crawler for the Brazilian Web
One of the key components of current Web search engines is the document collector. The paper describes CoBWeb, an automatic… 
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)