Goods: Organizing Google's Datasets

Abstract

Enterprises increasingly rely on structured datasets to run their businesses. These datasets take a variety of forms, such as structured files, databases, spreadsheets, or even services that provide access to the data. The datasets often reside in different storage systems, may vary in their formats, may change every day. In this paper, we present GOODS, a project to rethink how we organize structured datasets at scale, in a setting where teams use diverse and often idiosyncratic ways to produce the datasets and where there is no centralized system for storing and querying them. GOODS extracts metadata ranging from salient information about each dataset (owners, timestamps, schema) to relationships among datasets, such as similarity and provenance. It then exposes this metadata through services that allow engineers to find datasets within the company, to monitor datasets, to annotate them in order to enable others to use their datasets, and to analyze relationships between them. We discuss the technical challenges that we had to overcome in order to crawl and infer the metadata for billions of datasets, to maintain the consistency of our metadata catalog at scale, and to expose the metadata to users. We believe that many of the lessons that we learned are applicable to building large-scale enterprise-level data-management systems in general.

DOI: 10.1145/2882903.2903730

Extracted Key Phrases

8 Figures and Tables

0102020162017
Citations per Year

Citation Velocity: 10

Averaging 10 citations per year over the last 2 years.

Learn more about how we calculate this metric in our FAQ.

Cite this paper

@inproceedings{Halevy2016GoodsOG, title={Goods: Organizing Google's Datasets}, author={Alon Y. Halevy and Flip Korn and Natalya Fridman Noy and Christopher Olston and Neoklis Polyzotis and Sudip Roy and Steven Euijong Whang}, booktitle={SIGMOD Conference}, year={2016} }