Mixture model-based classification

__"This is a great overview of the field of model-based clustering and classification by one of its leading developers. McNicholas provides a resource that I am certain will be used by researchers in statistics and related disciplines for quite some time. The discussion of mixtures with heavy tails and asymmetric distributions will place this text as the authoritative, modern reference in the mixture modeling literature."__ (Douglas Steinley, University of Missouri) **Mixture Model-Based Classification** is the first monograph devoted to mixture model-based approaches to clustering and classification. This is both a book for established researchers and newcomers to the field. A history of mixture models as a tool for classification is provided and Gaussian mixtures are considered extensively, including mixtures of factor analyzers and other approaches for high-dimensional data. Non-Gaussian mixtures are considered, from mixtures with components that parameterize skewness and/or concentration, right up to mixtures of multiple scaled distributions. Several other important topics are considered, including mixture approaches for clustering and classification of longitudinal data as well as discussion about how to define a cluster **Paul D. McNicholas** is the Canada Research Chair in Computational Statistics at McMaster University, where he is a Professor in the Department of Mathematics and Statistics. His research focuses on the use of mixture model-based approaches for classification, with particular attention to clustering applications, and he has published extensively within the field. He is an associate editor for several journals and has served as a guest editor for a number of special issues on mixture models. Cover 1 Half Title 2 Title 3 Copyright 4 Dedication 5 Table of Contents 6 List of Figures 12 List of Tables 15 Preface 20 1 Introduction 22 1.1 Classification 22 1.2 Finite Mixture Models 24 1.3 Model-Based Clustering, Classification, and Discriminant Analysis 25 1.4 Comparing Partitions 27 1.5 R Packages 29 1.6 Datasets 30 1.7 Outline of the Contents of This Monograph 30 2 Mixtures of Multivariate Gaussian Distributions 32 2.1 Historical Development 32 2.2 Parameter Estimation 35 2.2.1 Model-Based Clustering 35 2.2.2 Model-Based Classification 37 2.2.3 Model-Based Discriminant Analysis 38 2.2.4 Initialization via Deterministic Annealing 40 2.2.5 Stopping Rules 40 2.3 Gaussian Parsimonious Clustering Models 44 2.4 Model Selection 46 2.5 Merging Gaussian Components 48 2.6 Illustrations 50 2.6.1 x2 Data 50 2.6.2 Banknote Data 53 2.6.3 Female Voles Data 54 2.6.4 Italian Olive Oil Data 56 2.7 Comments 61 3 Mixtures of Factor Analyzers and Extensions 63 3.1 Factor Analysis 63 3.1.1 The Model 63 3.1.2 An EM Algorithm for the Factor Analysis Model 64 3.1.3 Woodbury Identity 67 3.1.4 Comments 68 3.2 Mixture of Factor Analyzers 68 3.3 Parsimonious Gaussian Mixture Models 69 3.3.1 A Family of Eight Models 69 3.3.2 Parameter Estimation 70 3.3.3 Comments 74 3.4 Expanded Parsimonious Gaussian Mixture Models 74 3.4.1 A Family of Twelve Models 74 3.4.2 Parameter Estimation 75 3.5 Mixture of Common Factor Analyzers 78 3.5.1 The Model 78 3.5.2 Parameter Estimation 79 3.5.3 Discussion 82 3.6 Illustrations 83 3.6.1 x2 Data 83 3.6.2 Italian Wine Data 83 3.6.3 Italian Olive Oil Data 85 3.6.4 Alon Colon Cancer Data 86 3.7 Comments 88 4 Dimension Reduction and High-Dimensional Data 91 4.1 Implicit and Explicit Approaches 91 4.2 PGMM Family in High-Dimensional Applications 92 4.3 VSCC 93 4.4 clustvarsel and selvarclust 95 4.5 GMMDR 96 4.6 HD-GMM 98 4.7 Illustrations 99 4.7.1 Coffee Data 99 4.7.2 Leptograpsus Crabs 101 4.7.3 Banknote Data 103 4.7.4 Wisconsin Breast Cancer Data 103 4.7.5 Leukaemia Data 104 4.8 Comments 105 5 Mixtures of Distributions with Varying Tail Weight 106 5.1 Mixtures of Multivariate t-Distributions 106 5.2 Mixtures of Power Exponential Distributions 111 5.3 Illustrations 119 5.3.1 Overview 119 5.3.2 x2 Data 119 5.3.3 Body Data 119 5.3.4 Diabetes Data 120 5.3.5 Female Voles Data 122 5.3.6 Leptograpsus Crabs Data 122 5.4 Comments 124 6 Mixtures of Generalized Hyperbolic Distributions 125 6.1 Overview 125 6.2 Generalized Inverse Gaussian Distribution 125 6.2.1 A Parameterization 125 6.2.2 An Alternative Parameterization 126 6.3 Mixtures of Shifted Asymmetric Laplace Distributions 127 6.3.1 Shifted Asymmetric Laplace Distribution 127 6.3.2 Parameter Estimation 128 6.3.3 SAL Mixtures versus Gaussian Mixtures 131 6.4 Mixture of Generalized Hyperbolic Distributions 133 6.4.1 Generalized Hyperbolic Distribution 133 6.4.2 Parameter Estimation 136 6.5 Mixture of Generalized Hyperbolic Factor Analyzers 139 6.5.1 The Model 139 6.5.2 Parameter Estimation 140 6.5.3 Analogy with the Gaussian Solution 143 6.6 Illustrations 146 6.6.1 Old Faithful Data 146 6.6.2 Yeast Data 147 6.6.3 Italian Wine Data 149 6.6.4 Liver Data 150 6.7 A Note on Normal Variance-Mean Mixtures 150 6.8 Comments 151 7 Mixtures of Multiple Scaled Distributions 154 7.1 Overview 154 7.2 Mixture of Multiple Scaled t-Distributions 155 7.3 Mixture of Multiple Scaled SAL Distributions 157 7.4 Mixture of Multiple Scaled Generalized Hyperbolic Distributions 158 7.5 Mixture of Coalesced Generalized Hyperbolic Distributions 159 7.6 Cluster Convexity 161 7.7 Illustrations 164 7.7.1 Bankruptcy Data 164 7.7.2 Other Clustering Examples 166 7.7.3 Classification and Discriminant Analysis Examples 168 7.8 Comments 168 8 Methods for Longitudinal Data 170 8.1 Modified Cholesky Decomposition 170 8.2 Gaussian Mixture Modelling of Longitudinal Data 171 8.2.1 The Model 171 8.2.2 Model Fitting 172 8.2.2.1 VEA Model 172 8.2.2.2 EVI Model 176 8.2.3 Constraining Sub-Diagonals of Tg 178 8.2.3.1 VdEA Model 178 8.2.3.2 EdVI Model 180 8.2.4 Modelling the Component Means 181 8.3 Using t-Mixtures 182 8.4 Illustrations 184 8.4.1 Clustering 184 8.4.2 Classification 189 8.5 Comments 191 9 Miscellania 192 9.1 On the Definition of a Cluster 192 9.2 What Is the Best Way to Perform Clustering, Classification, and Discriminant Analysis? 195 9.3 Mixture Model Averaging 197 9.4 Robust Clustering 200 9.5 Clustering Categorical Data 202 9.6 Cluster-Weighted Models 203 9.7 Mixed-Type Data 204 9.7.1 A Mixture of Latent Variables Model 204 9.7.2 Illustration: Urinary System Disease Diagnosis 206 9.8 Alternatives to the EM Algorithm 208 9.8.1 Variational Bayes Approximations 208 9.8.2 Other Approaches 209 9.9 Challenges and Open Questions 210 Appendix 212 A.1 Linear Algebra Results 212 A.2 Matrix Calculus Results 213 A.3 Method of Lagrange Multipliers 214 References 215 Index 241 This work addresses classification using mixture models broadly. Unlike traditional treatments of the subject that heavily focus on unsupervised approaches, this book gives attention to unsupervised, semi-supervised, and supervised classification paradigms. Case studies illustrate both non-Gaussian and Gaussian approaches to model selection.