Alterations in parietal and temporal white matter microstructure derived from diffusion tensor imaging occur in preclinical and clinical Alzheimer’s disease. Amyloid beta (Aβ) deposition and such white matter alterations are two pathological hallmarks of Alzheimer’s disease. However, the relationship between these pathologies is not yet understood, partly since conventional diffusion MRI methods cannot distinguish between cellular and extracellular processes. Thus, we studied Aβ-associated longitudinal diffusion MRI changes in Aβ-positive (N = 21) and Aβ-negative (N = 51) cognitively normal elderly obtained from the Alzheimer’s Disease Neuroimaging Initiative dataset using linear mixed models. Aβ-positivity was based on Alzheimer’s Disease Neuroimaging Initiative amyloid-PET recommendations using a standardized uptake value ratio cut-off of 1.11. We used free-water imaging to distinguish cellular and extracellular changes. We found that Aβ-positive subjects had increased baseline right uncinate fasciculus free-water fraction (FW), associated with worse baseline Alzheimer’s disease assessment scale scores. Furthermore, Aβ-positive subjects showed faster decrease in fractional anisotropy (FW-corrected) in the right uncinate fasciculus and faster age-dependent right inferior longitudinal fasciculus FW increases over time. Right inferior longitudinal fasciculus FW increases were associated with greater memory decline. Importantly, these results remained significant after controlling for gray and white matter volume and hippocampal volume. This is the first study to illustrate the influence of Aβ burden on early longitudinal (in addition to baseline) white matter changes in cognitively normal elderly individuals at-risk of Alzheimer’s disease, thus underscoring the importance of longitudinal studies in assessing microstructural alterations in individuals at risk of Alzheimer’s disease prior to symptoms onset.