Some accounts of common ground assume that successful communication requires detailed consideration of others’ knowledge. In two studies, we provide evidence for an alternative account that views common ground as being mediated in part through domain-general memory mechanisms. On each trial, participants heard prerecorded instructions from one of two speakers indicating which of two displayed pictures to select. During an initial associa- tion phase, each speaker repeatedly referred to different sets of pictures. In Experiment 1, we contrasted a “between- speaker mapping” condition, in which each speaker referred to only one picture from critical item pairs (e.g., the cat drinking milk vs. the cat sitting up), and a “within-speaker mapping” condition, in which each speaker referred to both pictures within each pair, although item categories differed across speakers. On subsequent test trials, we recorded participants' eye fixations to critical displays that included both items from a category pair. Prior to the linguistic point of disambiguation, participants in the between-speaker mapping condition were more likely to fixate on the picture previously described by the current speaker, suggesting that knowledge associated with the speaker was prompting expectations for which picture would be the intended target. In Experiment 2, we used two prerecorded speakers of the same gender to strengthen the claim that the relevant implicit memory associations are speaker-specific. These results demonstrate how domain-general memory associa- tions can be an important constraint upon language use.