This paper introduces UJI-Butler, an innovative multi-robot framework that blends symbolic and non-symbolic artificial intelligence methods. Unlike previous systems, UJI-Butler integrates large language models (LLMs) with a knowledge base akin to RAG-based systems, while imposing logical reasoning on LLM-generated results. It facilitates multi-modal interaction with human users through speech, sign language, and physical interaction, fostering a human-in-the-loop learning paradigm. By acquiring new knowledge through verbal communication and mastering manipulation skills via human-lead-through programming, UJI-Butler enhances transparency and trust by incorporating human feedback during operations. Experimental results demonstrate that UJI-Butler’s combination of symbolic and non-symbolic AI offers intuitive interaction and accelerates the learning process with experience. It adeptly stores and utilizes knowledge gained from verbal communication, recognizing hand gestures for requests. Additionally, UJI-Butler successfully performs user-taught physical skills and generalizes them to varying object sizes and locations. The explicit nature of acquired knowledge enables seamless transferability to other platforms and modification by human users. The code of the whole project is available on Github, in addition, video demonstrations of the UJI-Butler system are available online in a Youtube Playlist.