High-energy nuclear collisions create a quark-gluon plasma, whose initial condition and subsequent expansion vary from event to event, impacting the distribution of the eventwise average transverse momentum [P([p(T)])]. Disentangling the contributions from fluctuations in the nuclear overlap size (geometrical component) and other sources at a fixed size (intrinsic component) remains a challenge. This problem is addressed by measuring the mean, variance, and skewness of P([p(T)]) in Pb-208 thorn Pb-208 and Xe-129 thorn Xe-129 collisions at root s(NN) = 5.02 and 5.44 TeV, respectively, using the ATLAS detector at the LHC. All observables show distinct features in ultracentral collisions, which are explained by a suppression of the geometrical component as the overlap area reaches its maximum. These results demonstrate a new technique to separate geometrical and intrinsic fluctuations, providing constraints on initial conditions and properties of the quark-gluon plasma, such as the speed of sound.