Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Non-Gaussian Statistical Modelsand Their Applications
KTH, School of Electrical Engineering (EES), Sound and Image Processing.
2011 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Statistical modeling plays an important role in various research areas. It provides away to connect the data with the statistics. Based on the statistical properties of theobserved data, an appropriate model can be chosen that leads to a promising practicalperformance. The Gaussian distribution is the most popular and dominant probabilitydistribution used in statistics, since it has an analytically tractable Probability DensityFunction (PDF) and analysis based on it can be derived in an explicit form. However,various data in real applications have bounded support or semi-bounded support. As the support of the Gaussian distribution is unbounded, such type of data is obviously notGaussian distributed. Thus we can apply some non-Gaussian distributions, e.g., the betadistribution, the Dirichlet distribution, to model the distribution of this type of data.The choice of a suitable distribution is favorable for modeling efficiency. Furthermore,the practical performance based on the statistical model can also be improved by a bettermodeling.

An essential part in statistical modeling is to estimate the values of the parametersin the distribution or to estimate the distribution of the parameters, if we consider themas random variables. Unlike the Gaussian distribution or the corresponding GaussianMixture Model (GMM), a non-Gaussian distribution or a mixture of non-Gaussian dis-tributions does not have an analytically tractable solution, in general. In this dissertation,we study several estimation methods for the non-Gaussian distributions. For the Maxi-mum Likelihood (ML) estimation, a numerical method is utilized to search for the optimalsolution in the estimation of Dirichlet Mixture Model (DMM). For the Bayesian analysis,we utilize some approximations to derive an analytically tractable solution to approxi-mate the distribution of the parameters. The Variational Inference (VI) framework basedmethod has been shown to be efficient for approximating the parameter distribution byseveral researchers. Under this framework, we adapt the conventional Factorized Approx-imation (FA) method to the Extended Factorized Approximation (EFA) method and useit to approximate the parameter distribution in the beta distribution. Also, the LocalVariational Inference (LVI) method is applied to approximate the predictive distributionof the beta distribution. Finally, by assigning a beta distribution to each element in thematrix, we proposed a variational Bayesian Nonnegative Matrix Factorization (NMF) forbounded support data.

The performances of the proposed non-Gaussian model based methods are evaluatedby several experiments. The beta distribution and the Dirichlet distribution are appliedto model the Line Spectral Frequency (LSF) representation of the Linear Prediction (LP)model for statistical model based speech coding. For some image processing applications,the beta distribution is also applied. The proposed beta distribution based variationalBayesian NMF is applied for image restoration and collaborative filtering. Comparedto some conventional statistical model based methods, the non-Gaussian model basedmethods show a promising improvement.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology , 2011. , xii, 49 p.
Series
Trita-EE, ISSN 1653-5146
National Category
Telecommunications Electrical Engineering, Electronic Engineering, Information Engineering Computer and Information Sciences
Identifiers
URN: urn:nbn:se:kth:diva-47408ISBN: 978-91-7501-158-5 (print)OAI: oai:DiVA.org:kth-47408DiVA: diva2:455048
Public defence
2011-12-05, E1, Lindstedsvägen 3, KTH, Stockholm, 09:00 (English)
Opponent
Supervisors
Note
QC 20111115Available from: 2011-11-15 Created: 2011-11-08 Last updated: 2018-01-12Bibliographically approved
List of papers
1. Bayesian Estimation of Beta Mixture Models with Variational Inference
Open this publication in new window or tab >>Bayesian Estimation of Beta Mixture Models with Variational Inference
2011 (English)In: IEEE Transaction on Pattern Analysis and Machine Intelligence, ISSN 0162-8828, E-ISSN 1939-3539, Vol. 33, no 11, 2160-2173 p.Article in journal (Refereed) Published
Abstract [en]

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutionsto simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation tothe prior/posterior distribution of the parameters in the beta distribution and propose an analytically tractable (closed-form) Bayesianapproach to the parameter estimation. The approach is based on the variational inference (VI) framework. Following the principles ofthe VI framework and utilizing the relative convexity bound, the extended factorized approximation method is applied to approximate thedistribution of the parameters in BMM. In a fully Bayesian model where all the parameters of the BMM are considered as variables andassigned proper distributions, our approach can asymptotically find the optimal estimate of the parameters posterior distribution. Also,the model complexity can be determined based on the data. The closed-form solution is proposed so that no iterative numericalcalculation is required. Meanwhile, our approach avoids the drawback of overfitting in the conventional expectation maximizationalgorithm. The good performance of this approach is verified by experiments with both synthetic and real data.

Keyword
Bayesian Estimation, Maximum Likelihood Estimation, Beta Distribution, Mixture Modeling, Variational Inference, Factorized Approximation
National Category
Computer and Information Sciences
Research subject
SRA - ICT
Identifiers
urn:nbn:se:kth:diva-33677 (URN)10.1109/TPAMI.2011.63 (DOI)000294910000004 ()2-s2.0-80053127168 (Scopus ID)
Note
QC 20110929Available from: 2011-05-16 Created: 2011-05-13 Last updated: 2018-01-12Bibliographically approved
2. Approximating the predictive distribution of the beta distribution with the local variational method
Open this publication in new window or tab >>Approximating the predictive distribution of the beta distribution with the local variational method
2011 (English)In: IEEE Intl. Workshop on Machine Learning for Signal Processing, 2011Conference paper, Published paper (Refereed)
Abstract [en]

In the Bayesian framework, the predictive distribution is obtained by averaging over the posterior parameter distribution. When there is a small amount of data, the uncertainty of the parameters is high. Thus with the predictive distribution, a more reliable result can be obtained in the applications as classification, recognition, etc. In the previous works, we have utilized the variational inference framework to approximate the posterior distribution of the parameters in the beta distribution by minimizing the Kullback-Leibler divergence of the true posterior distribution from the approximating one. However, the predictive distribution of the beta distribution was approximated by a plug-in approximation with the posterior mean, regardless of the parameter uncertainty. In this paper, we carry on the factorized approximation introduced in the previous work and approximate the beta function by its first order Taylor expansion. Then the upper bound of the predictive distribution is derived by exploiting the local variational method. By minimizing the upper bound of the predictive distribution and after normalization, we approximate the predictive distribution by a probability density function in a closed form. Experimental results shows the accuracy and efficiency of the proposed approximation method.

Keyword
Bayesian Estimation, Beta Distribution, Local Variational Method, Predictive Distribution
National Category
Electrical Engineering, Electronic Engineering, Information Engineering Computer and Information Sciences
Identifiers
urn:nbn:se:kth:diva-47401 (URN)10.1109/MLSP.2011.6064567 (DOI)000298259900022 ()2-s2.0-82455198854 (Scopus ID)978-1-4577-1623-2 (ISBN)
Conference
21st IEEE International Workshop on Machine Learning for Signal Processing (MLSP)
Note
Trita-EE 2011:28. QC 20120403Available from: 2011-11-08 Created: 2011-11-08 Last updated: 2018-01-12Bibliographically approved
3. Expectation propagation for estimating the parameters of the beta distribution
Open this publication in new window or tab >>Expectation propagation for estimating the parameters of the beta distribution
2010 (English)In: 2010 IEEE International Conference On Acoustics, Speech, And Signal Processing, 2010, 2082-2085 p.Conference paper, Published paper (Refereed)
Abstract [en]

Parameter estimation for the beta distribution is analytically intractable due to the integration expression in the normalization constant. For maximum likelihood estimation, numerical methods can be used to calculate the parameters. For Bayesian estimation, we can utilize different approximations to the posterior parameter distribution. A method based on the variational inference (VI) framework reported the posterior mean of the parameters analytically but the approximating distribution violated the correlation between the parameters. We now propose a method via the expectation propagation (EP) framework to approximate the posterior distribution analytically and capture the correlation between the parameters. Compared to the method based on VI, the EP based algorithm performs better with small amounts of data and is more stable.

Series
International Conference on Acoustics Speech and Signal Processing ICASSP, ISSN 1520-6149
Keyword
Beta Distribution, Expectation Propagation, Variational Inference, Importance Sampling, Laplace Approximation
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:kth:diva-32255 (URN)10.1109/ICASSP.2010.5495085 (DOI)000287096002016 ()2-s2.0-78049387838 (Scopus ID)978-1-4244-4296-6 (ISBN)
Conference
2010 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2010; Dallas, TX; 14 March 2010 through 19 March 2010
Note
QC 20110418Available from: 2011-04-18 Created: 2011-04-11 Last updated: 2011-11-15Bibliographically approved
4. Vector Quantization of LSF Parameters With a Mixture of Dirichlet Distributions
Open this publication in new window or tab >>Vector Quantization of LSF Parameters With a Mixture of Dirichlet Distributions
2013 (English)In: IEEE Transactions on Audio, Speech, and Language Processing, ISSN 1558-7916, E-ISSN 1558-7924, Vol. 21, no 9, 1777-1790 p.Article in journal (Refereed) Published
Abstract [en]

Quantization of the linear predictive coding parameters is an important part in speech coding. Probability density function (PDF)-optimized vector quantization (VQ) has been previously shown to be more efficient than VQ based only on training data. For data with bounded support, some well-defined bounded-support distributions (e.g., the Dirichlet distribution) have been proven to outperform the conventional Gaussian mixture model (GMM), with the same number of free parameters required to describe the model. When exploiting both the boundary and the order properties of the line spectral frequency (LSF) parameters, the distribution of LSF differences (Delta LSF) can be modelled with a Dirichlet mixture model (DMM). We propose a corresponding DMM based VQ. The elements in a Dirichlet vector variable are highly mutually correlated. Motivated by the Dirichlet vector variable's neutrality property, a practical non-linear transformation scheme for the Dirichlet vector variable can be obtained. Similar to the Karhunen-Loeve transform for Gaussian variables, this non-linear transformation decomposes the Dirichlet vector variable into a set of independent beta-distributed variables. Using high rate quantization theory and by the entropy constraint, the optimal inter-and intra-component bit allocation strategies are proposed. In the implementation of scalar quantizers, we use the constrained-resolution coding to approximate the derived constrained-entropy coding. A practical coding scheme for DVQ is designed for the purpose of reducing the quantization error accumulation. The theoretical and practical quantization performance of DVQ is evaluated. Compared to the state-of-the-art GMM-based VQ and recently proposed beta mixture model (BMM) based VQ, DVQ performs better, with even fewer free parameters and lower computational cost.

Keyword
Beta distribution, bounded support distribution, Dirichlet distribution, line spectral frequency, mixture modelling, neutrality property, Speech coding, vector quantization
National Category
Electrical Engineering, Electronic Engineering, Information Engineering Computer and Information Sciences
Identifiers
urn:nbn:se:kth:diva-47406 (URN)10.1109/TASL.2013.2238732 (DOI)000321906500001 ()2-s2.0-84880522911 (Scopus ID)
Note

QC 20130815. Updated from submitted to published.

Available from: 2011-11-08 Created: 2011-11-08 Last updated: 2018-01-12Bibliographically approved
5. Modelling Speech Line Spectral Frequencies with Dirichlet Mixture Models
Open this publication in new window or tab >>Modelling Speech Line Spectral Frequencies with Dirichlet Mixture Models
2010 (English)In: 11th Annual Conference of the International Speech Communication Association: Spoken Language Processing for All, INTERSPEECH 2010, 2010, 2370-2373 p.Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we model the underlying probability density function(PDF) of the speech line spectral frequencies (LSF) parameterswith a Dirichlet mixture model (DMM). The LSF parametershave two special features: 1) the LSF parameters havea bounded range; 2) the LSF parameters are in an increasingorder. By transforming the LSF parameters to the ΔLSF parameters,the DMM can be used to model the ΔLSF parametersand take advantage of the features mentioned above. Thedistortion-rate (D-R) relation is derived for the Dirichlet distributionwith the high rate assumption. A bit allocation strategyfor DMM is also proposed. In modelling the LSF parametersextracted from the TIMIT database, the DMM shows a betterperformance compared to the Gaussian mixture model, in termsof D-R relation, likelihood and model complexity. Since modellingis the essential and prerequisite step in the PDF-optimizedvector quantizer design, better modelling results indicate a superiorquantization performance.

Keyword
speech coding, line spectral frequencies, mixture models, Dirichlet distribution
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering Computer Sciences
Identifiers
urn:nbn:se:kth:diva-33679 (URN)000313086500205 ()2-s2.0-79959816308 (Scopus ID)978-1-61782-123-3 (ISBN)
Conference
11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010. Makuhari, Chiba. 26 September 2010 - 30 September 2010
Note

QC 20111118

Available from: 2011-05-13 Created: 2011-05-13 Last updated: 2018-01-12Bibliographically approved
6. BG-NMF: a variational Bayesian NMF model for bounded support data
Open this publication in new window or tab >>BG-NMF: a variational Bayesian NMF model for bounded support data
2011 (English)Article in journal (Other academic) Submitted
Abstract [en]

In this paper, we present a new Bayesian nonnegative matrix factor-ization (NMF) method for bounded support data. The distribution of thebounded support data is modelled with the beta distribution. The parametersof the beta density function are considered as latent variables and factorizedinto two matrices (the basis matrix and the excitation matrix). Further-more, each entry in the factorized matrices is assigned with a gamma prior.Thus, we name this method as beta-gamma NMF (BG-NMF). Usually, theestimation of the posterior distribution does not have a closed-form solu-tion. With the variational inference framework and by taking the relativeconvexity property of the log-inverse-beta function, we derive a closed-formsolution to approximate the posterior distribution of the entries in the basisand the excitation matrices. Also, a sparse BG-NMF can be carried outby adding the sparseness constraint to the gamma prior. Evaluations withsynthetic data and real life data demonstrate that the proposed method isefficient for source separation, missing data prediction, and collaborativefiltering problems.

National Category
Electrical Engineering, Electronic Engineering, Information Engineering Computer and Information Sciences
Identifiers
urn:nbn:se:kth:diva-47407 (URN)
Note
QS 2011 QS 20120316Available from: 2011-11-08 Created: 2011-11-08 Last updated: 2018-01-12Bibliographically approved

Open Access in DiVA

fulltext(1963 kB)3826 downloads
File information
File name FULLTEXT01.pdfFile size 1963 kBChecksum SHA-512
43e7bcbc01c12bb69a35072c1549c9ec64940464df42cf33cd197879e477684b5e410242b3b2132674d365e3e77ae26fe18cb6093ed7df1b2b45e5811f244e64
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Ma, Zhanyu
By organisation
Sound and Image Processing
TelecommunicationsElectrical Engineering, Electronic Engineering, Information EngineeringComputer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 3826 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 543 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf