INTRODUCTION: Diffusion tensor imaging detects early tissue alterations in Alzheimer’s disease and cerebral small vessel disease (SVD). However, the origin of diffusion alterations in SVD is largely unknown. METHODS: To gain further insight, we applied free water (FW) imaging to patients with genetically defined SVD (Cerebral Autosomal Dominant Arteriopathy with Subcortical Infarcts and Leukoencephalopathy [CADASIL], n = 57), sporadic SVD (n = 444), and healthy controls (n = 28). We modeled freely diffusing water in the extracellular space (FW) and measures reflecting fiber structure (tissue compartment). We tested associations between these measures and clinical status (processing speed and disability). RESULTS: Diffusion alterations in SVD were mostly driven by increased FW and less by tissue compartment alterations. Among imaging markers, FW showed the strongest association with clinical status (Rup to 34%, P < .0001). Findings were consistent across patients with CADASIL and sporadic SVD. DISCUSSION: Diffusion alterations and clinical status in SVD are largely determined by extracellular fluid increase rather than alterations of white matter fiber organization.