• DOI:10.1137/1.9781611972900.8
• Corpus ID: 14227041

Conjunctive Filter: Breaking the Entropy Barrier

@inproceedings{Okanohara2010ConjunctiveFB,
  title={Conjunctive Filter: Breaking the Entropy Barrier},
  author={Daisuke Okanohara and Yuichi Yoshida},
  booktitle={ALENEX},
  year={2010}
}

• Daisuke Okanohara, Yuichi Yoshida
• Published in ALENEX 16 January 2010
• Computer Science

We consider a problem for storing a map that associates a key with a set of values. To store n values from the universe of size m, it requires log2(mn) bits of space, which can be approximated as (1.44 + n) log2 m/n bits when n L m. If we allow e fraction of errors in outputs, we can store it with roughly n log2 1/e bits, which matches the entropy bound. Bloom filter is a well-known example for such data structures. Our objective is to break this entropy bound and construct more space-efficient data structures.

In this paper, we propose a novel data structure called a conjunctive filter, which supports conjunctive queries on k distinct keys for fixed k. Although a conjunctive filter cannot return the set of values itself associated with a queried key, it can perform conjunctive queries with O(1/√m) fraction of errors. Also, the consumed space is n/k log2 m bits and it is significantly smaller than the entropy bound n/2 log2 m when k ≥ 3.

We will show that many problems can be solved by using a conjunctive filter such as full-text search and database join queries. Also, we conducted experiments using a real-world data set, and show that a conjunctive filter answers conjunctive queries almost correctly using about 1/2 ~ 1/4 space as the entropy bound.