Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Challenges in face expression recognition from video
University of Skövde, School of Informatics. University of Skövde, The Informatics Research Centre. (Interaction Lab)
University of Skövde, School of Informatics. University of Skövde, The Informatics Research Centre. (Interaction Lab)ORCID iD: 0000-0002-6568-9342
2017 (English)In: SweDS 2017: The 5th Swedish Workshop on Data Science / [ed] Alexander Schliep, 2017Conference paper, Oral presentation with published abstract (Refereed)
Abstract [en]

Identication of emotion from face expressions is a relatively well understood problem where state-of-the-art solutions perform almost as well as humans. However, in many practical applications, disruptingfactors still make identication of face expression a very challenging problem. Within the project DREAM1- Development of Robot Enhanced Therapy for Children with Autism Spectrum Disorder (ASD), we areidentifying face expressions from children with ASD, during therapy. Identied face expressions are usedboth in the online system, to guide the behavior of the robot, and o-line, to automatically annotate videofor measurements of clinical outcomes.

This setup puts several new challenges on the face expression technology. First of all, in contrast tomost open databases of face expressions comprising adult faces, we are recognizing emotions from childrenbetween the age of 4 to 7 years. Secondly, children with ASD may show emotions dierently, compared totypically developed children. Thirdly, the children move freely during the intervention and, despite the useof several cameras tracking the face of the child from dierent angles, we rarely have a full frontal view ofthe face. Fourthly, and nally, the amount of native data is very limited.

Although we have access to extensive video recorded material from therapy sessions with ASD children,potentially constituting a very valuable dataset for both training and testing of face expression implemen-tations, this data proved to be dicult to use. A session of 10 minutes of video may comprise only a fewinstances of expressions e.g. smiling. As such, although we have many hours of video in total, the data isvery sparse and the number of clear face expressions is still rather small for it to be used as training data inmost machine learning (ML) techniques.

We therefore focused on the use of synthetic datasets for transfer learning, trying to overcome thechallenges mentioned above. Three techniques were evaluated: (1) convolutional neural networks for imageclassication by analyzing separate video frames, (2) recurrent neural networks for sequence classication tocapture facial dynamics, and (3) ML algorithms classifying pre-extracted facial landmarks.

The performance of all three models are unsatisfactory. Although the proposed models were of highaccuracy, approximately 98%, while classifying a test set, they performed poorly on the real-world data.This was due to the usage of a synthetic dataset which had mostly a frontal view of faces. The models whichhave not seen similar examples before failed to classify them correctly. The accuracy decreased drasticallywhen the child rotated her head or covered a part of her face. Even if the frame clearly captured a facialexpression, ML algorithms were not able to provide a stable positive classication rate. Thus, elaborationon training datasets and designing robust ML models are required. Another option is to incorporate voiceand gestures of the child into the model to classify emotional state as a complex concept.

Place, publisher, year, edition, pages
2017.
National Category
Computer Engineering
Research subject
Interaction Lab (ILAB)
Identifiers
URN: urn:nbn:se:his:diva-14581OAI: oai:DiVA.org:his-14581DiVA, id: diva2:1166244
Conference
SweDS 2017: The 5th Swedish Workshop on Data Science. Data Science Division Department of Computer Science and Engineering University of Gothenburg Chalmers, Gothenburg, Sweden, December 12–13, 2017
Projects
DREAM, FP7 funded project, #611391Available from: 2017-12-14 Created: 2017-12-14 Last updated: 2018-01-13Bibliographically approved

Open Access in DiVA

fulltext(258 kB)16 downloads
File information
File name FULLTEXT01.pdfFile size 258 kBChecksum SHA-512
3c0077274be2283de6c0f9ca3c6d1ae0e363966bb86fecdae6ee3c2db25a7647998198ec2c46831ddf464bffcfd646388af2438ea6a98d5965ce60589ac9a4d0
Type fulltextMimetype application/pdf

Other links

Abstracts

Search in DiVA

By author/editor
Redyuk, SergeyBilling, Erik A.
By organisation
School of InformaticsThe Informatics Research Centre
Computer Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 16 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 654 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf