Chapter 5 Practical Web Crawling

    Abstract

    Chapters ?? and ?? presented the model and architecture implemented in the WIRE crawler. When we tested our implementation, we found that there were several problems of Web crawling which did not become evident until a large crawl was executed. We are interested in documenting these problems for two reasons: • To help other crawler designers, because most… (More)

    2 Figures and Tables

    Topics

    • Presentations referencing similar topics