Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Deep probabilistic models for sequential and hierarchical data
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Division of Systems and Control. Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Automatic control.ORCID iD: 0000-0002-1539-6314
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Consider the problem where we want a computer program capable of recognizing a pedestrian on the road. This could be employed in a car to automatically apply the brakes to avoid an accident. Writing such a program is immensely difficult but what if we could instead use examples and let the program learn what characterizes a pedestrian from the examples. Machine learning can be described as the process of teaching a model (computer program) to predict something (the presence of a pedestrian) with help of data (examples) instead of through explicit programming.

This thesis focuses on a specific method in machine learning, called deep learning. This method can arguably be seen as sole responsible for the recent upswing of machine learning in academia as well as in society at large. However, deep learning requires, in human standards, a huge amount of data to perform well which can be a limiting factor.  In this thesis we describe different approaches to reduce the amount of data that is needed by encoding some of our prior knowledge about the problem into the model. To this end we focus on sequential and hierarchical data, such as speech and written language.

Representing sequential output is in general difficult due to the complexity of the output space. Here, we make use of a probabilistic approach focusing on sequential models in combination with a deep learning structure called the variational autoencoder. This is applied to a range of different problem settings, from system identification to speech modeling.

The results come in three parts. The first contribution focus on applications of deep learning to typical system identification problems, the intersection between the two areas and how they can benefit from each other. The second contribution is on hierarchical data where we promote a multiscale variational autoencoder inspired by image modeling. The final contribution is on verification of probabilistic models, in particular how to evaluate the validity of a probabilistic output, also known as calibration.

Place, publisher, year, edition, pages
Uppsala: Acta Universitatis Upsaliensis, 2022. , p. 87
Series
Digital Comprehensive Summaries of Uppsala Dissertations from the Faculty of Science and Technology, ISSN 1651-6214 ; 2139
Keywords [en]
Machine learning, Deep learning, Sequential modelling
National Category
Signal Processing
Research subject
Electrical Engineering with specialization in Signal Processing
Identifiers
URN: urn:nbn:se:uu:diva-470433ISBN: 978-91-513-1478-5 (print)OAI: oai:DiVA.org:uu-470433DiVA, id: diva2:1649337
Public defence
2022-05-24, Sonja Lyttkens, 101121, Ångström, Lägerhyddsvägen 1, Uppsala, 09:00 (English)
Opponent
Supervisors
Available from: 2022-05-02 Created: 2022-04-04 Last updated: 2022-06-14
List of papers
1. Data-driven impulse response regularization via deep learning
Open this publication in new window or tab >>Data-driven impulse response regularization via deep learning
2018 (English)Conference paper, Published paper (Refereed)
Series
IFAC-PapersOnLine, ISSN 2405-8963 ; 51:15
National Category
Signal Processing
Identifiers
urn:nbn:se:uu:diva-366186 (URN)10.1016/j.ifacol.2018.09.081 (DOI)000446599200002 ()
Conference
SYSID 2018, July 9–11, Stockholm, Sweden
Available from: 2018-10-08 Created: 2018-11-22 Last updated: 2022-04-04Bibliographically approved
2. Deep convolutional networks in system identification
Open this publication in new window or tab >>Deep convolutional networks in system identification
Show others...
2019 (English)In: Proc. 58th IEEE Conference on Decision and Control, IEEE, 2019, p. 3670-3676Conference paper, Published paper (Refereed)
Abstract [en]

Recent developments within deep learning are relevant for nonlinear system identification problems. In this paper, we establish connections between the deep learning and the system identification communities. It has recently been shown that convolutional architectures are at least as capable as recurrent architectures when it comes to sequence modeling tasks. Inspired by these results we explore the explicit relationships between the recently proposed temporal convolutional network (TCN) and two classic system identification model structures; Volterra series and block-oriented models. We end the paper with an experimental study where we provide results on two real-world problems, the well-known Silverbox dataset and a newer dataset originating from ground vibration experiments on an F-16 fighter aircraft.

Place, publisher, year, edition, pages
IEEE, 2019
National Category
Control Engineering
Identifiers
urn:nbn:se:uu:diva-397528 (URN)10.1109/CDC40024.2019.9030219 (DOI)000560779003058 ()978-1-7281-1398-2 (ISBN)
Conference
CDC 2019, December 11–13, Nice, France
Funder
Swedish Foundation for Strategic Research , RIT15-0012Swedish Research Council, 621-2016-06079
Available from: 2020-03-12 Created: 2019-11-21 Last updated: 2022-04-04Bibliographically approved
3. Learning deep autoregressive models for hierarchical data
Open this publication in new window or tab >>Learning deep autoregressive models for hierarchical data
2021 (English)In: IFAC PapersOnLine, Elsevier BV Elsevier, 2021, Vol. 54, no 7, p. 529-534Conference paper, Published paper (Refereed)
Abstract [en]

We propose a model for hierarchical structured data as an extension to the stochastic temporal convolutional network. The proposed model combines an autoregressive model with a hierarchical variational autoencoder and downsampling to achieve superior computational complexity. We evaluate the proposed model on two different types of sequential data: speech and handwritten text. The results are promising with the proposed model achieving state-of-the-art performance.

Place, publisher, year, edition, pages
ElsevierElsevier BV, 2021
Keywords
Deep learning, variational autoencoders, nonlinear systems
National Category
Computer Vision and Robotics (Autonomous Systems)
Identifiers
urn:nbn:se:uu:diva-457738 (URN)10.1016/j.ifacol.2021.08.414 (DOI)000696396200091 ()
Conference
19th IFAC Symposium on System Identification (SYSID), JUL 13-16, 2021, Padova, ITALY
Funder
Swedish Research CouncilKjell and Marta Beijer Foundation
Available from: 2021-11-12 Created: 2021-11-12 Last updated: 2024-01-15Bibliographically approved
4. Evaluating model calibration in classification
Open this publication in new window or tab >>Evaluating model calibration in classification
Show others...
2019 (English)In: 22nd International Conference on Artificial Intelligence and Statistics, 2019, p. 3459-3467Conference paper, Published paper (Refereed)
Series
Proceedings of Machine Learning Research, ISSN 2640-3498 ; 89
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:uu:diva-397519 (URN)000509687903053 ()
Conference
AISTATS 2019, April 16–18, Naha, Japan
Available from: 2019-04-25 Created: 2019-11-21 Last updated: 2023-04-26Bibliographically approved

Open Access in DiVA

UUThesis_Andersson,C_2022(2870 kB)519 downloads
File information
File name FULLTEXT01.pdfFile size 2870 kBChecksum SHA-512
c04f02c3cbb7e0260ceae484c384a7e6546a7655709001565b7cfdfd251fc51da6b6b3e7a50d659b7be7be57577f83f8d3d0cc03666b67b5bbb6039f2e64b3c9
Type fulltextMimetype application/pdf
UUThesis_Andersson,C_2022(2763 kB)402 downloads
File information
File name FULLTEXT02.pdfFile size 2763 kBChecksum SHA-512
f0cc385cdba1cff9d01431148e19afc3029b075c0df2aa4de24e51d88953742474ea3d8854b69f2a87dfc05c68890b1c405ac8867396ffe4a4626e3fa25b129c
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Andersson, Carl
By organisation
Division of Systems and ControlAutomatic control
Signal Processing

Search outside of DiVA

GoogleGoogle Scholar
Total: 922 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 684 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf