Hearing loss in later life has been associated with risk of dementia. The impact of risk factors for dementia may change during life, and it is unknown whether hearing loss early in midlife represents a risk factor for dementia. We examined whether hearing loss diagnosed in midlife was associated with an increased risk of dementia. A cohort comprising 942,567 Danish men enrolled in the mandatory conscription board examination was followed from conscription (age 19). Cognitive ability was measured at conscription, while hearing loss was ascertained either by physicians diagnosis at conscription or by the Danish National Patient Registry from 1977 to 2016 (ICD-8:388; 389; ICD-10:H90; H91). Differences in cognitive ability in relation to hearing loss at conscription were calculated using t test, while the risk of dementia associated with hearing loss was estimated using Cox regression with adjustment for cognitive ability, education, depression, diabetes, hypertension, and cerebrovascular disease. Men with hearing loss at conscription had about 2 points (corresponding to 0.20 SD) lower mean cognitive score than those without hearing loss. During follow-up, 59,834 men had a hearing loss diagnosis, while 9114 were diagnosed with dementia. Midlife hearing loss was associated with an increased rate of dementia diagnosed before age 60 (adjusted Hazard Ratio (HR) = 1.90 [95% CI 1.59–2.76]) or at a later age (adjusted HR = 1.15 [95% CI 1.06–1.25]). Our study supports the evidence that early identification and correction of hearing loss holds promise for prevention of dementia later in life.