Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Recall is the Proper Evaluation Metric for Word Segmentation
Uppsala University, Disciplinary Domain of Humanities and Social Sciences, Faculty of Languages, Department of Linguistics and Philology.
Uppsala University, Disciplinary Domain of Humanities and Social Sciences, Faculty of Languages, Department of Linguistics and Philology.
Uppsala University, Disciplinary Domain of Humanities and Social Sciences, Faculty of Languages, Department of Linguistics and Philology.
2017 (English)In: Proceedings of the The 8th International Joint Conference on Natural Language Processing, Taipei: Asian Federation of Natural Language Processing, 2017, p. 86-90Conference paper, Published paper (Refereed)
Abstract [en]

We extensively analyse the correlations and drawbacks of conventionally employed evaluation metrics for word segmentation. Unlike in standard information retrieval, precision favours under-splitting systems and therefore can be misleading in word segmentation. Overall, based on both theoretical and experimental analysis, we propose that precision should be excluded from the standard evaluation metrics and that the evaluation score obtained by using only recall is sufficient and better correlated with the performance of word segmentation systems.

Place, publisher, year, edition, pages
Taipei: Asian Federation of Natural Language Processing, 2017. p. 86-90
National Category
Language Technology (Computational Linguistics)
Identifiers
URN: urn:nbn:se:uu:diva-335946OAI: oai:DiVA.org:uu-335946DiVA, id: diva2:1164403
Conference
The Eighth International Joint Conference on Natural Language Processing, Taipei, November 27 – December 1, 2017.
Available from: 2017-12-11 Created: 2017-12-11 Last updated: 2018-04-13Bibliographically approved
In thesis
1. Segmenting and Tagging Text with Neural Networks
Open this publication in new window or tab >>Segmenting and Tagging Text with Neural Networks
2018 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Segmentation and tagging of text are important preprocessing steps for higher-level natural language processing tasks. In this thesis, we apply a sequence labelling framework based on neural networks to various segmentation and tagging tasks, including sentence segmentation, word segmentation, morpheme segmentation, joint word segmentation and part-of-speech tagging, and named entity transliteration. We apply a general neural CRF model to different tasks by designing specific tag sets. In addition, we explore effective ways of representing input characters, such as utilising concatenated n-grams and sub-character features, and use ensemble decoding to mitigate the effects of random parameter initialisation.

The segmentation and tagging models are evaluated in a truly multilingual setup with more than 70 datasets. The experimental results indicate that the proposed neural CRF model is effective for segmentation and tagging in general as state-of-the-art accuracies are achieved on datasets in different languages, genres, and annotation schemes for various tasks. For word segmentation, we propose several typological factors to statistically characterise the difficulties posed by different languages and writing systems. Based on this analysis, we apply language-specific settings to the segmentation system for higher accuracy. Our system achieves substantially better results on languages that are more difficult to segment when compared to previous work. Moreover, we investigate conventionally adopted evaluation metrics for segmentation tasks. We propose that precision should be excluded and using recall alone is more adequate for sentence segmentation and word segmentation. The segmentation and tagging tools implemented along with this thesis are publicly available as experimental frameworks for future development as well as preprocessing tools for higher-level NLP tasks.

Abstract [sv]

Segmentering och taggning av text är grundläggande analyssteg som möjliggör mer avancerade tillämpningar inom språkteknologi. I denna avhandling tillämpas ett ramverk för sekvensanalys baserat på neurala nätverk på ett antal olika segmenterings- och taggningsproblem, inklusive meningssegmentering, ordsegmentering, morfemsegmentering, förenad ordsegmentering och ordklasstaggning, samt translitterering av namn. Vi tillämpar en generell neural CRF-modellen på olika problem genom att definiera olika tagguppsättningar. Vi utforskar olika sätt att representera de tecken som utgör indata till processen, såsom hopslagna teckensekvenser och grafiska särdrag, och vi använder ensemble-avkodning för att mildra effekten av slumpmässig initialisering av parametrar.

Modellerna för segmentering och taggning utvärderas på flera språk med hjälp av mer än 70 olika datamängder. De experimentella resultaten visar att den föreslagna neurala CRF-modellen är effektiv för segmentering och taggning i allmänhet, med slagkraftiga resultat för olika uppgifter, språk, genrer och annotationsscheman. För ordsegmentering föreslår vi ett antal typologiska faktorer som kan användas för att statistiskt analysera de utmaningar som ges av olika språk och skriftsystem. Denna analys kan sedan läggas till grund för språkspecifika inställningar som förbättrar segmenteringens kvalitet. Vårt system uppnår väsentligt bättre resultat än tidigare metoder på språk som är svåra att segmentera. Till sist diskuterar vi utvärderingsmetoder för segmenteringsproblem och föreslår att precision ska uteslutas till förmån för enbart täckning vid utvärdering av menings- och ordsegmentering. De verktyg för segmentering och taggning som utvecklats i samband med avhandlingsarbetet är allmänt tillgängliga för fortsatt forskning och som grundläggande analysverktyg för mer avancerade tillämpningar av språkteknologi.

Place, publisher, year, edition, pages
Uppsala: Acta Universitatis Upsaliensis, 2018. p. 76
Series
Studia Linguistica Upsaliensia, ISSN 1652-1366 ; 21
Keywords
neural networks, sequence labelling, multilinguality, word segmentation, sentence segmentation, morpheme segmentation, transliteration, joint word segmentation and POS tagging
National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:uu:diva-348129 (URN)978-91-513-0340-6 (ISBN)
Public defence
2018-06-09, Humanistiska teatern, Thunbergsvägen 3, Uppsala, 10:15 (English)
Opponent
Supervisors
Available from: 2018-05-16 Created: 2018-04-10 Last updated: 2018-05-16

Open Access in DiVA

fulltext(166 kB)33 downloads
File information
File name FULLTEXT01.pdfFile size 166 kBChecksum SHA-512
b94d629b72fe0633738cce2636436a59d57694245575a42547bbdacdf931dedb14b70a51275d988f537c7970a2bf09f1d44750cc5e60884a24f669f23b9fe9f5
Type fulltextMimetype application/pdf

Other links

Fulltext

Search in DiVA

By author/editor
Shao, YanHardmeier, ChristianNivre, Joakim
By organisation
Department of Linguistics and Philology
Language Technology (Computational Linguistics)

Search outside of DiVA

GoogleGoogle Scholar
Total: 33 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 67 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf