Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evaluation NMT And LLM Systems For Thai-English And Swedish-English Translation
Umeå University, Faculty of Science and Technology, Department of Computing Science.
2026 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE creditsStudent thesis
Abstract [en]

This study combines two evaluation methods: three automatic evaluation metrics (BLEU, chrF, and COMET) and LLM-as-a-judge to evaluate translation quality for the low-resource language pair Thai-English. Moreover, this study also investigates translation performance by comparing with the high-resource language pair Swedish-English. Within the scope of the study, four large language model (LLM) systems (Gemini 2.5 Flash, GPT-5.4 mini, Qwen2-72B, Llama 3.3-70B) and one neural machine translation (NMT) system (Google Translate) were used to translate 32 texts drawn from the official governmental and diplomatic domain with a focus on terminological accuracy and semantic preservation. The results show that Gemini 2.5 Flash outperforms Google Translate and all other LLM- based translation systems on Thai-English translation across all three automatic metrics and LLM-as-a-judge. In contrast, Google Translate outperforms Gemini 2.5 Flash across BLEU and chrF on Swedish-English translation. These findings suggest that LLM systems are better suited for translation in low-resource languages and the combination of the evaluation methods provides a complementary evaluation of translation quality.

Place, publisher, year, edition, pages
2026. , p. 34
Series
UMNAD ; 1654
Keywords [en]
Low-resource languages, Thai-English translation, Large language models, Neural machine translation, LLM-as-a-judge, Automatic evaluation metrics
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:umu:diva-256091OAI: oai:DiVA.org:umu-256091DiVA, id: diva2:2080042
External cooperation
Parsd
Educational program
Bachelor of Science Programme in Computing Science
Supervisors
Examiners
Available from: 2026-06-26 Created: 2026-06-26 Last updated: 2026-06-26Bibliographically approved

Open Access in DiVA

fulltext(2812 kB)646 downloads
File information
File name FULLTEXT01.pdfFile size 2812 kBChecksum SHA-512
0fcc0098fa2e64d42c74ee95c0d7ad846e3910e866c3c85bcaf8a779d8544b61a4bb5e98b5ca6114f2f33f4db8364eedd1629487d65360480fd62c8029a27767
Type fulltextMimetype application/pdf

By organisation
Department of Computing Science
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 457 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf