Web Search for a Planet: The Google Cluster Architecture

  title={Web Search for a Planet: The Google Cluster Architecture},
  author={L. Barroso and J. Dean and Urs H{\"o}lzle},
  journal={IEEE Micro},
Amenable to extensive parallelization, Google's web search application lets different queries run on different processors and, by partitioning the overall index, also lets a single query use multiple processors. to handle this workload, Googless architecture features clusters of more than 15,000 commodity-class PCs with fault tolerant software. This architecture achieves superior performance at a fraction of the cost of a system built from fewer, but more expensive, high-end servers. 
Motivating a Distributed System of Commodity Machines 1
This report examines the price/performance benefit of using a large cluster of commodity machines rather than server level hardware for certain large scale software applications. A number of toolsExpand
Basic issues on the processing of web queries
This study finds that load unbalance at low query arrival rates can be controlled with a simple measure of randomizing the distribution of documents among the index servers, the broker is not a bottleneck, and disk utilization is higher than CPU utilization. Expand
Challenges on Distributed Web Retrieval
This paper surveys and organizes recent research results, outlining the main challenges of designing a distributed Web retrieval system. Expand
Query Processing in Highly-Loaded Search Engines
A novel dropping strategy is introduced, based on machine learned performance predictors to select the queries to drop in order to sustain the largest possible query rate with a relative degradation in effectiveness. Expand
Load balancing for term-distributed parallel retrieval
Methods for load balancing in term-distributed parallel architectures are examined, and a suite of techniques for reducing net querying costs are proposed, which allow a 30% improvement in query throughput when tested on an eight-node parallel computer system. Expand
A Hybrid Distributed Architecture for Indexing
Test results confirmed that indexing performance is directly related to the size of the hybrid grid and intranet networking does not play a major role, and a system-efficiency and cost-effectiveness comparison of a grid and a multiprocessor machine showed that for workloads of modest to large sizes, the grid architecture delivers better throughput per unit cost than the multiprocessionor. Expand
ROAR: increasing the flexibility and performance of distributed search
Rendezvous On a Ring (ROAR) is introduced, a novel distributed algorithm that enables on-the-fly re-configuration of the partitioning level that can add and remove servers without stopping the system, cope with server failures, and provide good load-balancing even with a heterogeneous server pool. Expand
Towards a Distributed Search Engine
The main goal is to reduce the cost of a Web search engine while keeping all the benefits of a centralized search engine in spite of the intrinsic network latency imposed by Internet. Expand
Automatic management of partitioned, replicated search services
The distributed search architecture that underlies Twitter user search, a service for discovering relevant accounts on the popular microblogging service, makes use of the principle that eliminates the distinction between failure and other anticipated service disruptions, which leads to greater robustness and fault-tolerance. Expand
Exploiting Hybrid Parallelism in Web Search Engines
An hybrid technique based on MPI and OpenMP which has been devised to take advantage of the multithreading facilities provided by CMP nodes for search engines under high query traffic is proposed. Expand


Memory system characterization of commercial workloads
A detailed performance study of three important classes of commercial workloads: online transaction processing (OLTP), decision support systems (DSS), and Web index search, which characterizes the memory system behavior of these workloads through a large number of architectural experiments augmented with full system simulations to determine the impact of architectural trends. Expand
The Anatomy of a Large-Scale Hypertextual Web Search Engine
This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want. Expand
A Single-Chip Multiprocessor
Presents the case for billion-transistor processor architectures that will consist of chip multiprocessors (CMPs): multiple (four to 16) simple, fast processors on one chip, and all processors share a larger level-two cache. Expand
Piranha: a scalable architecture based on single-chip multiprocessing
This paper describes the Piranha system, a research prototype being developed at Compaq that aggressively exploits chip multiprocessing by integrating eight simple Alpha processor cores along with aExpand
Hyper-threading technology architecture and microarchitecture : a hyperhtext history
“ TPC Benchmark C Full Disclosure Report for IBM eserver xSeries 440 using Microsoft SQL Server 2000 Enterprise Edition and Microsoft Windows . NET Datacenter Server 2003 , TPC - C Version 5 . 0
  • “ Hyper - Threading Technology Architecture and Microarchitecture : A Hypertext History , ” Intel Technology J .
  • 2002
, and K . Olukotun , “ A Single - Chip Multiprocessor
  • Piranha : A Scalable Architecture Based on Single - Chip Multiprocessing , ” Proc . 27 th ACM Int ’ l Symp . Computer Architecture
  • 2000
A Single-Chip Multiprocessor," Computer
  • vol. 30,
  • 1997
TPC Benchmark C Full Disclosure Report for IBM eserver xSeries 440 using Microsoft SQL Server 2000 Enterprise Edition and Microsoft Windows .NET Datacenter Server 2003, TPC-C Version 5
  • TPC Benchmark C Full Disclosure Report for IBM eserver xSeries 440 using Microsoft SQL Server 2000 Enterprise Edition and Microsoft Windows .NET Datacenter Server 2003, TPC-C Version 5