Southern Africa was likely exclusively inhabited by San hunter-gatherers before similar to 2000 years ago. Around that time, East African groups assimilated with local San groups and gave rise to the Khoekhoe herders. Subsequently, Bantu-speaking farmers, arriving from the north (similar to 1800 years ago), assimilated and displaced San and Khoekhoe groups, a process that intensified with the arrival of European colonists similar to 350 years ago. In contrast to the western parts of southern Africa, where several Khoe-San groups still live today, the eastern parts are largely populated by Bantu speakers and individuals of non-African descent. Only a few scattered groups with oral traditions of Khoe-San ancestry remain. Advances in genetic research open up new ways to understand the population history of southeastern Africa. We investigate the genomic variation of the remaining individuals from two South African groups with oral histories connecting them to eastern San groups, i.e., the San from Lake Chrissie and the Duma San of the uKhahlamba-Drakensberg. Using similar to 2.2 million genetic markers, combined with comparative published data sets, we show that the Lake Chrissie San have genetic ancestry from both Khoe-San (likely the parallel to Xegwi San) and Bantu speakers. Specifically, we found that the Lake Chrissie San are closely related to the current southern San groups (i.e., the Karretjie people). Duma San individuals, on the other hand, were genetically similar to southeastern Bantu speakers from South Africa. This study illustrates how genetic tools can be used to assess hypotheses about the ancestry of people who seemingly lost their historic roots, only recalling a vague oral tradition of their origin.