Objective: Although patients prefer describing their problems using words, mental health interventions are commonly evaluated using rating scales. Fortunately, recent advances in natural language processing (i.e., AI-methods) yield new opportunities to quantify people's own mental health descriptions. Our aim was to explore whether responses to open-ended questions, quantified using AI, provide additional value in measuring intervention outcomes compared to traditional rating scales.

Method: Swedish adolescents (N = 44) who received Internet-based Cognitive Behavioral Therapy (ICBT) for eight weeks completed (pre/post) scales measuring anxiety and depression and three open-ended questions (related to depression, anxiety and general mental health). The language responses were quantified using a large language model and quantitative methods to predict mental health as measured by rating scales, valence (i.e., words' positive/negative affectivity), and semantic content (i.e., meaning).

Results: Similar to the rating scales, language measures revealed statistically significant health improvements between pre and post measures such as reduced depression and anxiety symptoms and an increase in the use of words conveying positive emotions and different meanings (e.g., pre-intervention: “anxious”, depressed; post-intervention: “happy”, “the future”). Notably, the health changes identified through semantic content measures remained statistically significant even after accounting for the changes captured by the rating scales.

Conclusion: Language responses analyzed using AI-methods assessed outcomes with fewer items, demonstrating effectiveness and accuracy comparable to traditional rating scales. Additionally, this approach provided valuable insights into patients' well-being beyond mere symptom reduction, thus highlighting areas of improvement that rating scales often overlook. Since patients often prefer using natural language to express their mental health, this method could complement, and address comprehension issues associated fixed-item questionnaires.