Predicting object displacement during tool–soil–object interaction is important for robotic excavation and manipulation tasks, yet existing methods are either computationally expensive or lack physical interpretability. This paper presents a novel three-level physics-informed design that systematically integrates domain knowledge through features, architecture, and loss function to model tool–soil–object interaction including granular dynamics. Trained on discrete element method simulations, our model demonstrates improved generalization to unseen object masses and lengths compared to black-box baselines, achieving lower prediction errors in extrapolative evaluation cases with improvements of up to ≈30% for extreme out-of-distribution scenarios.