In this report we analyzed genetically informative data to investigate within-person change and between-person differences in late-life cognitive abilities as a function of childhood social class. We used data from nine testing occasions spanning 28 y in the Swedish Adoption/Twin Study of Aging and parental social class based on the Swedish socioeconomic index. Cognitive ability included a general factor and the four domains of verbal, fluid, memory, and perceptual speed. Latent growth curve models of the longitudinal data tested whether level and change in cognitive performance differed as a function of childhood social class. Between-within twin-pair analyses were performed on twins reared apart to assess familial confounding. Childhood social class was significantly associated with mean-level cognitive performance at age 65 y, but not with rate of cognitive change. The association decreased in magnitude but remained significant after adjustments for level of education and the degree to which the rearing family was supportive toward education. A between-pair effect of childhood social class was significant in all cognitive domains, whereas within-pair estimates were attenuated, indicating genetic confounding. Thus, childhood social class is important for cognitive performance in adulthood on a population level, but the association is largely attributable to genetic influences.