Drug-drug interactions (DDIs) can lead to serious adverse effects and compromised treatment efficacy. As artificial intelligence models gain traction in DDI prediction, the interpretability of these models becomes crucial. Despite advancements in predictive performance, most DDI models lack adequate explainability, limiting their clinical utility. This study aims to enhance the interpretability of a deep learning DDI prediction model by applying a post-hoc Explainable Artificial Intelligence (XAI) method and evaluating the model’s reasoning in the context of domain knowledge. The analysis focused on a publicly available DDI model based on autoencoders and a deep neural network. Kernel SHAP was applied to the model using the original input space, which included feature vectors derived from structural, target gene, and gene ontology similarity matrices. DrugBank served as the reference for validating pharmacological relevance. Explanations were evaluated based on domain knowledge and visualized using SHAP tools. Several features highlighted by Kernel SHAP appeared to align with the underlying mechanisms of DDIs. Structural similarity features appeared frequently among the top contributors to the prediction. Often, high importance was assigned to identity-like structural similarity features, which raises concerns about potential shortcut learning. Target gene and gene ontology features were less consistent, sometimes reflecting mechanisms of interaction and other times appearing misleading due to limitations in similarity calculations. The study demonstrates that XAI methods, when supported by domain-informed interpretation, can uncover valuable insights into a model’s internal reasoning. It also raises important concerns regarding feature design and the necessity for more rigorous validation.