Crawler-Friendly Web Servers

  title={Crawler-Friendly Web Servers},
  author={Onn Brandman and Junghoo Cho and H. Garcia-Molina and N. Shivakumar},
  journal={SIGMETRICS Perform. Evaluation Rev.},
  • Onn Brandman, Junghoo Cho, +1 author N. Shivakumar
  • Published 2000
  • Computer Science
  • SIGMETRICS Perform. Evaluation Rev.
  • In this paper we study how to make web servers (e.g., Apache) more crawler friendly. Current web servers offer the same interface to crawlers and regular web surfers, even though crawlers and surfers have very different performance requirements. We evaluate simple and easy-to-incorporate modifications to web servers so that there are significant bandwidth savings. Specifically, we propose that web servers export meta-data archives decribing their content. 
    72 Citations

    Figures and Topics from this paper.

    Crawlets: Agents for High Performance Web Search Engines
    • 29
    • Highly Influenced
    Scheduling algorithms for Web crawling
    • 70
    • PDF
    Internet search engine freshness by Web server help
    • V. Gupta, R. Campbell
    • Computer Science
    • Proceedings 2001 Symposium on Applications and the Internet
    • 2001
    • 30
    • PDF
    Cooperation schemes between a Web server and a Web search engine
    • C. Castillo
    • Computer Science
    • Proceedings of the IEEE/LEOS 3rd International Conference on Numerical Simulation of Semiconductor Optoelectronic Devices (IEEE Cat. No.03EX726)
    • 2003
    • 10
    • PDF
    Design of an Efficient Migrating Crawler based on Sitemaps
    • 6
    • PDF
    A Co-operative Web Services Paradigm for Supporting Crawlers
    • 7
    • Highly Influenced
    • PDF
    An Efficient Technique to Reduce Network Load during Web Crawling


    Accessible at
    • Harvest user's manual,
    • 1996
    Harvest user 's manual, Jan
    • 1996