Background The improved survival after breast cancer has prompted knowledge on the effect of a breast cancer diagnosis on health-related quality of life (HQoL). This study compared changes in HQoL among women from before to after breast cancer diagnosis with longitudinal changes among women who remained breast cancer-free. Material and methods The Danish Diet, Cancer and Health study included 57 053 cancer-free persons aged 50-64 years at baseline (1993-1997). We used data from first follow-up (1999-2002) and second follow-up (2010-2012) on HQoL [Medical Outcomes Survey, short form (SF-36)] obtained from 542 women aged 64-82 years with primary breast cancer (stages I-III) and a randomly matched sample of 729 women who remained breast cancer-free. Linear regression models were used to estimate the differences in changes in HQoL between women with and without breast cancer; the analyses were repeated with stratification according to age, comorbidity, partner support and time since diagnosis. Results Women with breast cancer reported significantly larger decreases in HQoL from before to after diagnosis than those who remained breast cancer-free (physical component summary, -2.0; 95% CI -2.8; -1.2, mental component summary, -1.5, 95% CI -2.3; -0.6). This association was significantly modified by comorbidity and time since diagnosis. Conclusions Women with breast cancer reported significantly larger HQoL declines than breast cancer-free women. Breast cancer diagnosis seems to have the greatest impact on HQoL closest to diagnosis and in women with comorbidity indicating that this group should be offered timely and appropriate follow-up care to prevent HQoL declines.