Dense Associative Memories (DenseAMs) are generalizations of Hopfield networks that store training data at local minima of the energy landscape. When the amount of training data exceeds the critical memory storage capacity, new local minima (different from the training data) emerge. These emergent local minima, known as spurious states, can hinder memory retrieval. We examine diffusion models (DMs) through the DenseAM lens, viewing their generative process as an attempt of a memory retrieval. In the small data regime, DMs create distinct attractors for each training sample, akin to DenseAMs below the critical capacity. As the training set size increases, DMs transition to generalization. We identify a critical intermediate phase, predicted by DenseAM theory, characterized by the emergence of spurious states. In generative modeling, these previously overlooked states are no longer negative artifacts but rather are the first signs of generative capabilities. Their existence is demonstrated across a wide range of architectures and datasets.
The discrete, token-level counterpart of this picture is developed in Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data. For the associative memory tools used here, see Modern Methods in Associative Memory and Energy Transformer, or my blog post on energy-based modeling.
@article{pham2026emergence,
title={Emergence of diffusion models from associative memory through memorization and generalization},
author={Pham, Bao and Raya, Gabriel and Negri, Matteo and Zaki, Mohammed J. and Ambrogioni, Luca and Krotov, Dmitry},
journal={Nature Communications},
year={2026},
doi={10.1038/s41467-026-78727-y},
url={https://www.nature.com/articles/s41467-026-78727-y}
}