Objective: We examined whether childhood adversity experienced in early childhood (0–5 years) is related to body mass index (BMI) in childhood (6–7 years) and adolescence (12–15 years). Methods: This study combined data from the nationwide register-based DANLIFE study on childhood adversities with data on height and weight of school children in Copenhagen. Data were available for 53,401 children born in Denmark between 1980 and 1996. Children were divided into groups of early childhood adversity by applying group-based multi-trajectory modelling using their yearly count of childhood adversity in three dimensions (i.e., material deprivation, loss or threat of loss, and family dynamics) from 0–5 years. Direct and total associations between the early childhood adversity groups and BMI z-scores in childhood and adolescence were estimated using sex-stratified structural equation models. Results: Five exclusive and exhaustive groups of early childhood adversity were identified, which were characterized by low adversity (51%), moderate material deprivation (30%), high material deprivation (14%), loss or threat of loss (3%) and high adversity (2%). Boys and girls exposed to moderate or high material deprivation and loss or threat of loss had a slightly higher BMI z-score, especially in adolescence, compared with those in the low adversity group, with the strongest association found for girls in the loss or threat of loss group (b (95% CI) = 0.18 (0.10, 0.26)). Additionally, boys in the high adversity group had a slightly lower BMI z-score in childhood than boys in the low adversity group (b (95% CI) = −0.12 (−0.22, −0.02)). Conclusions: Whereas associations with BMI were found for children and adolescents exposed to material deprivation, loss or threat of loss, and high adversity, the effect sizes were generally small. Contrary to prevailing hypotheses, weight changes in childhood is probably not a major explanatory mechanism linking early childhood adversity with later-life morbidity.