• DOI:10.1007/978-3-319-59665-5_4
• Corpus ID: 32495052

DDFlasks: Deduplicated Very Large Scale Data Store

@inproceedings{Maia2017DDFlasksDV,
  title={DDFlasks: Deduplicated Very Large Scale Data Store},
  author={Francisco Maia and Jo{\~a}o Paulo and F{\'a}bio Coelho and Francisco Neves and J. Pereira and R. Oliveira},
  booktitle={DAIS},
  year={2017}
}

• Francisco Maia, J. Paulo, +3 authors R. Oliveira
• Published in DAIS 19 June 2017
• Computer Science

With the increasing number of connected devices, it becomes essential to find novel data management solutions that can leverage their computational and storage capabilities. However, developing very large scale data management systems requires tackling a number of interesting distributed systems challenges, namely continuous failures and high levels of node churn. In this context, epidemic-based protocols proved suitable and effective and have been successfully used to build DataFlasks, an epidemic data store for massive scale systems. Ensuring resiliency in this data store comes with a significant cost in storage resources and network bandwidth consumption. Deduplication has proven to be an efficient technique to reduce both costs but, applying it to a large-scale distributed storage system is not a trivial task. In fact, achieving significant space-savings without compromising the resiliency and decentralized design of these storage systems is a relevant research challenge.