The textile industry has a significant environmental impact while at the same time playing an important role in society. There is a need to improve the textile industry's sustainability. Textile SMEs encounter numerous challenges to reach sustainability. This paper aims at recognizing the challenges by collecting, analyzing, and discussing some theoretical and empirical data. At the end of this paper, six main groups of challenges are presented: lack of stakeholder knowledge about sustainability, lack of resources, lack of communication, and effective legislation. Also, recognizing the relevant needs and finding some conceptual solutions for textile SMEs to overcome the challenges and reach more sustainability are considered as future work of this study.