Corpus ID: 228376138

When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?

@article{Brown2020WhenIM,
  title={When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?},
  author={G. Brown and M. Bun and V. Feldman and A. Smith and Kunal Talwar},
  journal={ArXiv},
  year={2020},
  volume={abs/2012.06421}
}
  • G. Brown, M. Bun, +2 authors Kunal Talwar
  • Published 2020
  • Computer Science
  • ArXiv
  • Modern machine learning models are complex and frequently encode surprising amounts of information about individual inputs. In extreme cases, complex models appear to memorize entire input examples, including seemingly irrelevant information (social security numbers from text, for example). In this paper, we aim to understand whether this sort of memorization is necessary for accurate learning. We describe natural prediction problems in which every sufficiently accurate training algorithm must… CONTINUE READING
    1 Citations
    Privacy Analysis in Language Models via Training Data Leakage Report
    • Highly Influenced

    References

    SHOWING 1-10 OF 39 REFERENCES
    Does learning require memorization? a short tale about a long tail
    • 41
    • Highly Influential
    • PDF
    What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
    • 9
    • PDF
    Learners that Use Little Information
    • 42
    • Highly Influential
    • PDF
    Overparameterized neural networks implement associative memory
    • 4
    • PDF
    A Closer Look at Memorization in Deep Networks
    • 483
    • PDF
    Average-Case Information Complexity of Learning
    • 5
    • PDF
    The Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting Secrets
    • 138
    • PDF
    A Direct Sum Result for the Information Complexity of Learning
    • 8
    • PDF