Brain imaging has significantly contributed to our understanding of the cerebellum being involved in recovery after non-cerebellar stroke. Due to its connections with supratentorial brain networks, acute stroke can alter the function and structure of the contralesional cerebellum, known as crossed cerebellar diaschisis (CCD). Data on the spatially precise distribution of structural CCD and their implications for persistent deficits after stroke are notably limited. In this cross-sectional study, structural MRI and clinical data were analyzed from 32 chronic stroke patients, at least 6 months after the event. We quantified lobule-specific contralesional atrophy, as a surrogate of structural CCD, in patients and healthy controls. Volumetric data were integrated with clinical scores of disability and motor deficits. Diaschisis-outcome models were adjusted for the covariables age, lesion volume, and damage to the corticospinal tract. We found that structural CCD was evident for the whole cerebellum, and particularly for lobules V and VI. Lobule VI diaschisis was significantly correlated with clinical scores, that is, volume reductions in contralesional lobule VI were associated with higher levels of disability and motor deficits. Lobule V and the whole cerebellum did not show similar diaschisis-outcome relationships across the spectrum of the clinical scores. These results provide novel insights into stroke-related cerebellar plasticity and might thereby promote lobule VI as a key area prone to structural CCD and potentially involved in recovery and residual motor functioning.