Eye-movement metrics like fixation location and duration are increasingly being used in infancy research. We tested whether fixation durations during meaningful social stimulus viewing involve common or different familial influences than fixation durations during viewing of abstract stimulus. We analysed the duration of fixations, and the allocation of fixations to face and motion, from 536 dizygotic and monozygotic 5-month-old twins in: naturalistic scenes including low- and high-level social features, and abstract scenes only having low-level features. We observed significant genetic influences in both conditions (h²_naturalistic = 0.30, 95% confidence interval (CI) 0.14 to 0.44; h²_abstract = 0.25, 95% CI 0.09 to 0.39), while shared environmental influences were negligible. Although some genetic influences were shared between the two conditions, unique genetic factors were linked to naturalistic scene viewing, indicating that fixation durations index different phenomena dependent on the context. Heritability for face looking was moderate (h² = 0.19, 95% CI 0.03 to 0.34), and no familial influences were found for motion looking. Exploratory polygenic score analyses revealed no significant associations with fixation measures. This study underscores the dissociable genetic influences on infants’ visual exploration of abstract versus naturalistic stimuli and the importance of considering context when interpreting eye-tracking data.