Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Attention Mechanisms for Transition-based Dependency Parsing
Uppsala universitet, Humanistisk-samhällsvetenskapliga vetenskapsområdet, Språkvetenskapliga fakulteten, Institutionen för lingvistik och filologi.
2019 (Engelska)Självständigt arbete på avancerad nivå (masterexamen), 20 poäng / 30 hpStudentuppsats (Examensarbete)
Abstract [en]

Transition-based dependency parsing is known to compute the syntactic structure of a sentence efficiently, but is less accurate to predict long-distance relations between tokens as it lacks global information about the sentence. Our main contribution is the integration of attention mechanisms to replace the static token selection with a dynamic approach that takes the complete sequence into account. Though our experiments confirm that our approach fundamentally works, our models do not outperform the baseline parser. We further present a line of follow-up experiments to investigate these results. Our main conclusion is that the BiLSTM of the traditional parser is already powerful enough to encode the required global information into each token, eliminating the need for an attention-driven approach.

Our secondary results indicate that the attention models require a neural network with a higher capacity to potentially extract more latent information from the word embeddings and the LSTM than the traditional parser. We further show that positional encodings are not useful for our attention models, though BERT-style positional embeddings slightly improve the results. Finally, we experiment with replacing the LSTM with a Transformer-encoder to test the impact of self-attention. The results are disappointing, though we think that more future research should be dedicated to this.

For our work, we implement a UUParser-inspired dependency parser from scratch in PyTorch and extend it with, among other things, full GPU support and mini-batch processing. We publish the code under a permissive open source license at https://github.com/jgontrum/parseridge.

Ort, förlag, år, upplaga, sidor
2019.
Nyckelord [en]
natural language processing, language technology, dependency parsing, transition-based parsing, parsing, attention
Nationell ämneskategori
Datorsystem Språkteknologi (språkvetenskaplig databehandling)
Identifikatorer
URN: urn:nbn:se:uu:diva-395491OAI: oai:DiVA.org:uu-395491DiVA, id: diva2:1362881
Utbildningsprogram
Masterprogram i språkteknologi
Presentation
2019-10-07, 22:47 (Engelska)
Handledare
Examinatorer
Tillgänglig från: 2019-10-25 Skapad: 2019-10-21 Senast uppdaterad: 2019-10-29Bibliografiskt granskad

Open Access i DiVA

Attention Mechanisms for Transition-based Dependency Parsing(451 kB)39 nedladdningar
Filinformation
Filnamn FULLTEXT01.pdfFilstorlek 451 kBChecksumma SHA-512
7f21676341036d2b4febabae0cda1254ef01b4dd4d2cc270a672c305d886a7e94db8ce1514cbd07c9e457fa42a5007bb961da2e3ccf5a3fce7dd474eda2eaf50
Typ fulltextMimetyp application/pdf

Av organisationen
Institutionen för lingvistik och filologi
DatorsystemSpråkteknologi (språkvetenskaplig databehandling)

Sök vidare utanför DiVA

GoogleGoogle Scholar
Totalt: 39 nedladdningar
Antalet nedladdningar är summan av nedladdningar för alla fulltexter. Det kan inkludera t.ex tidigare versioner som nu inte längre är tillgängliga.

urn-nbn

Altmetricpoäng

urn-nbn
Totalt: 108 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf