Maximum Likelihood Estimation of the Parameters of Univariate Normal Mixture and Probit Ordinal Regression Models via the EM and ECM Algorithms

Abstract

The intent of this project is to provide a detailed introduction to the Expectation Maximization (EM) and Expectation Conditional-Maximization (ECM) algorithms. With the aim of presenting an interesting and sufficiently complete exposition of these methods, we base our investigation on two major points, namely the convergence properties of the EM algorithm, and the formulation of the ECM algorithm as a solution to the problem when the M-step of the EM algorithm is complicated. Our starting point is a review of the statistical theory that is necessary for the comprehension and development of our topics of interest. After describing the mathematical construction of the EM and ECM algorithms, we give a discussion about their main convergence results where we prove the monotonicity property of the EM algorithm. As an application of these methods, we consider the important problems of estimating the parameters of univariate normal mixture and probit ordinal regression models by the method of maximum likelihood (ML). Finally, on the basis of the obtained results, we make use of the programming language for statistical computing R to illustrate the convergence properties of the EM algorithm in the case of a mixture of two univariate normal distributions, and to examine the performance of the ECM algorithm applied to a probit ordinal regression model in the case of four classes.