• DOI:10.14778/3229863.3240491
• Corpus ID: 51991005

Open Data Integration

@article{Miller2018OpenDI,
  title={Open Data Integration},
  author={Ren{\'e}e J. Miller},
  journal={Proc. VLDB Endow.},
  year={2018},
  volume={11},
  pages={2130-2139}
}

• Renée J. Miller
• Published 1 August 2018
• Computer Science
• Proc. VLDB Endow.

Open data plays a major role in supporting both governmental and organizational transparency. Many organizations are adopting
Open Data Principles
promising to make their open data complete, primary, and timely. These properties make this data tremendously valuable to data scientists. However, scientists generally do not have
a priori
knowledge about what data is available (its schema or content). Nevertheless, they want to be able to use open data and integrate it with other public or private data they are studying. Traditionally, data integration is done using a framework called
query discovery
where the main task is to discover a query (or transformation) that translates data from one form into another. The goal is to find the right operators to join, nest, group, link, and twist data into a desired form. We introduce a new paradigm for thinking about integration where the focus is on
data discovery,
but highly efficient internet-scale discovery that is driven by data analysis needs. We describe a research agenda and recent progress in developing scalable data-analysis or query-aware data discovery algorithms that provide high recall and accuracy over massive data repositories.