The distal fibula represents one of the most common fracture sites, and its epidemiology is characterized by a high incidence in both adolescence and the elderly. While fracture occurrence is influenced by trauma mechanism, a possible underlying skeletal microarchitectural deterioration in certain patient groups remains elusive. The purpose of this study was to determine the influence of age, sex, and overall skeletal status on fibular microarchitecture. We analyzed the microarchitecture of the distal fibula in 300 people by high-resolution peripheral quantitative computed tomography (HR-pQCT). Three areal bone mineral density (aBMD) groups (normal, osteopenia, osteoporosis; n = 100 per group) based on the concurrent assessment of aBMD by dual-energy X-ray absorptiometry (DXA) at the lumbar spine and total hip were established. Next to group comparisons, linear and non-linear regression analyses were carried out to assess the association between age, sex, BMI, tibial and fibular microarchitecture. While women had lower values for both trabecular bone volume fraction (BV/TVd, p < 0.001) and cortical thickness (Ct.Thd, p < 0.001) than men, osteoporosis by DXA negatively affected these parameters in both sexes. Remarkably, cortical but not trabecular microarchitecture declined with age, with a stronger decrease in females compared to males (Ct.Thd female -10.0 μm/year (95% CI: -12.2 to -7.7 μm/year), male -4.0 μm/year (95% CI: -6.3 to -1.7 μm/year)). Moderate positive associations between distal tibial and fibular microarchitecture were noted (e.g., BV/TVd R2 = 0.54, Ct.Thd R2 = 0.58). In summary, we here demonstrate the severe negative effects of age, female sex and osteoporosis on distal fibula bone mineralization and microarchitecture. The presented findings are likely to explain the higher susceptibility to distal fibula fractures in elderly women (independent of trauma mechanism). These alterations in fibular bone quality must be taken into account in the context of fracture prevention and treatment (e.g., osteosynthesis planning).