When do language diffusion models memorize their training data, and how to quantitatively assess their true generative regime? We address these questions by showing that Uniform-based Discrete Diffusion Models (UDDMs) fundamentally behave as Associative Memories with emergent creative capabilities. The core idea of an AM is to reliably recover stored data points as memories by establishing distinct basins of attraction around them. Historically, models like Hopfield networks use an explicit energy function to guarantee these stable attractors. We broaden this perspective by leveraging the observation that energy is not strictly necessary, as basins of attraction can also be formed via conditional likelihood maximization. By evaluating token recovery of training and test examples, we identify in UDDMs a sharp memorization-to-generalization transition governed by the size of the training dataset: as it increases, basins around training examples shrink and basins around unseen test examples expand, until both later converge to the same level. Crucially, we can detect this transition using only the conditional entropy of predicted token sequences: memorization is characterized by vanishing conditional entropy, while in the generalization regime the conditional entropy of most tokens remains finite. Thus, conditional entropy offers a practical probe for the memorization-to-generalization transition in deployed models.
Classical Associative Memories, and even their modern variants, rely on well-defined and explicit energy functions to define their attractors. Specifically, Hopfield networks and Dense Associative Memories define a functional \(E_\theta(x)\) whose local minima sit at the stored patterns. Because the retrieval dynamics descend \(E_\theta\), and \(E_\theta\) is bounded below and non-increasing along the trajectory, convergence to a fixed point is guaranteed. The energy is what certifies that the basins exist. However, in the discrete setting, we can no longer rely on energy functions because things are no longer continuously differentiable. In our paper, we show that in the simplest setting of pseudo-likelihood, involving spins, a simple model trained with this objective shows the same behaviour as a language diffusion model when it comes to token recovery.
Token recovery is the metric native to Associative Memory. Corrupt a sequence, let the model settle, and count how much of the original comes back; what this measures is the width of the basin around that sequence. Perplexity is the metric the language modelling community already reports. The two come from different traditions and measure different things, yet they mark the same event. The peak in perplexity (top row, coloured dashed line) falls at the dataset size where the recovery curves part ways (bottom row, grey dashed line): training recovery leaves its plateau at 100% while test recovery begins to climb. The takeaway is that the Associative Memory picture is not an interpretation layered on top of language diffusion models, but something already visible in a quantity these models are routinely evaluated on.
Token recovery tells us whether a basin exists, but measuring it requires the original data, since we have to know what the model was supposed to produce. Conditional entropy asks something far cheaper: how peaked is the model’s own predictive distribution over each token? As it turns out, the two questions have the same answer. Tokens that the model recovers sit at vanishing conditional entropy, the signature of a token pinned to the bottom of a basin, while the tokens it fails to recover spread across a broad band of finite entropy. Memorization is therefore visible in the entropy alone: a model that has stored its training data predicts that data with almost no uncertainty. Because this measurement needs nothing but the model’s own predictions, it can be run on a deployed model whose training set we cannot inspect.
This paper extends the memorization-to-generalization picture developed for continuous diffusion models in Memorization to Generalization: Emergence of Diffusion Models from Associative Memory to the discrete, token-level setting. For background on the associative memory tools used here, see Modern Methods in Associative Memory and my blog post on energy-based modeling.
@article{pham2026language,
title={Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data},
author={Pham, Bao and Zaki, Mohammed J and Ambrogioni, Luca and Krotov, Dmitry and Negri, Matteo},
journal={arXiv preprint arXiv:2604.26841},
year={2026}
}