A Deep Dive into Memorization in Deep Learning

Want to learn more about how, when and why machine learning, particularly deep learning systems memorize data? By studying memorization, you'll learn more about how machine learning systems really function, along with how privacy works from a technical point-of-view. You'll also be better able to decide how, when and where to use AI systems based on your new learnings.

This series aims to introduce the topics to a general audience, but there are plenty of links to dive deeper in each article. This page will be updated as the series is published.

The recommended reading order is as follows, but feel free to hop around!

• Introduction: Why study memorization in machine learning?
• Start with the Data: Machine Learning dataset distributions, history, and biases
• Encodings and embeddings: How does data get into machine learning systems?
• Gaming Evaluation: The evolution of deep learning training and evaluation
• How Memorization Happens: Repetition
• How Memorization Happens: Novelty
• How Memorization Happens: Overparametrized models

If you are a more visual or video learner, I've made a YouTube playlist to accompany the series.

I'm very open to feedback (positive, neutral or critical), questions (there are no stupid questions!) and creative re-use of this content. If you have any of those, please share it with me! This helps keep me inspired and writing. :)

Acknowledgements: I would like to thank Vicki Boykis, Damien Desfontaines and Yann Dupis for their feedback, corrections and thoughts on this series. Their input greatly contributed to improvements in my thinking and writing. Any mistakes, typos, inaccuracies or controversial opinions are my own.