Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
In Pursuit of Ideal Model Selection for High-Dimensional Linear Regression
KTH, School of Electrical Engineering and Computer Science (EECS). (Information Science and Engineering)
2019 (English)Doctoral thesis, monograph (Other academic)
Abstract [en]

The fundamental importance of model specification has motivated researchers to study different aspects of this problem. One of which is the task of model selection from the set of available competing models. In this regard, several successful model selection criteria have been developed for the classical setting in which the number of measurements is much larger than the parameter space. However, when the number of measurements is comparable with the size of the dimension of the parameter space, these criteria are too liberal and prone to overfitting.

In this thesis, we consider the problem of model selection for the high-dimensional setting in which the number of measurements is much smaller than the dimension of the parameter space. Inspired by previous work in this area, we propose a new model selection criterion based on the Fisher information. We analyze the performance of our criterion as the number of measurements increases to infinity as well as when the noise variance decreases to zero. We prove that the proposed criterion is consistent in selecting the true model in both scenarios. Besides, we conceive a computationally affordable algorithm to execute our model selection criterion. This algorithm utilizes the solution path of Lasso to narrow the set of all plausible combinatorial models down to a few ones. Interestingly, this algorithm also can be used for choosing the regularization parameter in the Lasso estimator properly. The empirical results support our theoretical findings. We also practice the task of model selection in situations where there are multiple measurement vectors available. Here, we also allow the elements of the noise vector to be spatially correlated. For such situations, we propose a non-negative Lasso estimator that is inspired by covariance matching techniques. Here, to tune the corresponding regularization parameter, we use our model selection criterion that has been introduced earlier. Empirical results show that our non-negative Lasso estimator can correctly select the true model when a relatively small number of measurement vectors are available. Moreover, the empirical results show that our proposed method is rather insensitive to a high correlation between the columns of the design matrix. In the last part of the thesis, we apply some of the theories and tools developed for model selection in the previous chapters to the problem of change point detection for noisy piecewise constant signals. In more details, we first consider the previously proposed change point estimation method, fused Lasso, and explain why it cannot guarantee the detection of the true change points. Then, we propose a normalized version of fused Lasso that is obtained by normalizing the columns of the sensing matrix of the Lasso equivalent. We analyze the performance of the proposed method, and in particular, we show that it is consistent in detecting change points as the noise variance tends to zero. Finally, we show numerical experiments that support our theoretical findings.

Place, publisher, year, edition, pages
Stockholm, Sweden: KTH Royal Institute of Technology, 2019. , p. 120
Series
TRITA-EECS-AVL ; 2019:15
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Research subject
Electrical Engineering
Identifiers
URN: urn:nbn:se:kth:diva-244010ISBN: 978-91-7873-100-8 (print)OAI: oai:DiVA.org:kth-244010DiVA, id: diva2:1288142
Public defence
2019-03-08, F3, Lindstedtsvägen 26, KTH Main Campus, Stockholm, 10:15 (English)
Opponent
Supervisors
Note

QC 20190213

Available from: 2019-02-13 Created: 2019-02-12 Last updated: 2019-03-05Bibliographically approved

Open Access in DiVA

fulltext(1191 kB)84 downloads
File information
File name FULLTEXT01.pdfFile size 1191 kBChecksum SHA-512
7c49f7543418f47b3026a0999fb8165be2e8cb178a91225e3ab3fb4befbb5f15938bda9979584e4453836b6ac4a326e378c29b65f49b75c9d7a59f0426e74b6c
Type fulltextMimetype application/pdf

By organisation
School of Electrical Engineering and Computer Science (EECS)
Electrical Engineering, Electronic Engineering, Information Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 84 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 297 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf