Matrix-based introduction to multivariate data analysis

This is the first textbook that allows readers who may be unfamiliar with matrices to understand a variety of multivariate analysis procedures in matrix forms. By explaining which models underlie particular procedures and what objective function is optimized to fit the model to the data, it enables readers to rapidly comprehend multivariate data analysis. Arranged so that readers can intuitively grasp the purposes for which multivariate analysis procedures are used, the book also offers clear explanations of those purposes, with numerical examples preceding the mathematical descriptions. Supporting the modern matrix formulations by highlighting singular value decomposition among theorems in matrix algebra, this book is useful for undergraduate students who have already learned introductory statistics, as well as for graduate students and researchers who are not familiar with matrix-intensive formulations of multivariate data analysis. The book begins by explaining fundamental matrix operations and the matrix expressions of elementary statistics. Then, it offers an introduction to popular multivariate procedures, with each chapter featuring increasing advanced levels of matrix algebra. Further the book includes in six chapters on advanced procedures, covering advanced matrix operations and recently proposed multivariate procedures, such as sparse estimation, together with a clear explication of the differences between principal components and factor analyses solutions. In a nutshell, this book allows readers to gain an understanding of the latest developments in multivariate data science. Preface to the Second Edition......Page 5 Preface to the First Edition......Page 8 Contents......Page 11 Elementary Statistics with Matrices......Page 18 1.1 Matrices......Page 19 1.2 Vectors......Page 21 1.3 Sum of Matrices and Their Multiplication by Scalars......Page 22 1.4 Inner Product and Norms of Vectors......Page 23 1.5 Product of Matrices......Page 24 1.7 Trace Operator and Matrix Norm......Page 27 1.8 Vectors and Matrices Filled with Ones or Zeros......Page 29 1.9 Special Square Matrices......Page 30 1.10 Bibliographical Notes......Page 32 2.1 Data Matrices......Page 33 2.2 Distributions......Page 35 2.4 Centered Scores......Page 36 2.5 Variances and Standard Deviations......Page 39 2.6 Standard Scores......Page 41 2.7 What Centering and Standardization Do for Distributions......Page 42 2.8 Matrix Representation......Page 43 2.9 Bibliographical Notes......Page 44 3.1 Scatter Plots and Correlations......Page 46 3.2 Covariances......Page 47 3.3 Correlation Coefficients......Page 49 3.4 Variable Vectors and Correlations......Page 51 3.5 Covariances and Correlations for Standard Scores......Page 52 3.6 Matrix Expressions of Covariances and Correlations......Page 53 3.7 Unbiased Covariances......Page 54 3.8 Centered Matrices......Page 55 3.9 Ranks of Matrices: Intuitive Introduction......Page 56 3.10 Ranks of Matrices: Mathematical Definition......Page 57 3.11 Bibliographical Notes......Page 59 Least Squares Procedures......Page 61 4.1 Prediction of a Dependent Variable by Explanatory Variables......Page 62 4.2 Least Squares Method......Page 65 4.3 Predicted and Error Values......Page 67 4.4 Proportion of Explained Variance and Multiple Correlation......Page 69 4.5 Interpretation of Regression Coefficients......Page 71 4.6 Standardization......Page 73 4.7 Geometric Derivation of Regression Coefficients......Page 74 4.8 Bibliographical Notes......Page 75 5.1 Reduction of Variables into Components......Page 78 5.2 Singular Value Decomposition......Page 80 5.3 Formulation with a Weight Matrix......Page 83 5.4 Constraints for Components......Page 84 5.5 Interpretation of Loadings......Page 86 5.7 Percentage of Explained Variance......Page 87 5.8 High-Dimensional Data Analysis......Page 89 5.9 Bibliographical Notes......Page 90 6.1 Reformulation with Different Constraints......Page 94 6.2 Maximizing the Sum of Variances......Page 95 6.3 Weighted Composite Scores with Maximum Variance......Page 97 6.4 Projecting Three-Dimensional Vectors onto Two-Dimensional Ones......Page 99 6.5 Visualization of Invisible Distributions......Page 102 6.6 Goodness of Projection......Page 105 6.7 Bibliographical Notes......Page 106 7.1 Membership Matrices......Page 108 7.2 Example of Clustering Results......Page 109 7.3 Formulation......Page 111 7.4 Iterative Algorithm......Page 113 7.5 Obtaining Cluster Features......Page 115 7.6 Obtaining Memberships......Page 116 7.7 Brief Description of Algorithm......Page 117 7.8 Bibliographical Notes......Page 118 Maximum Likelihood Procedures......Page 121 8.1 Model, Parameter, Objective Function, and Optimization......Page 122 8.2 Maximum Likelihood Method......Page 123 8.3 Probability Density Function......Page 126 8.4 Multivariate Normal Distribution......Page 127 8.5 Maximum Likelihood Method for Normal Variables......Page 130 8.6 Maximum Likelihood Estimates of Means and Covariances......Page 132 8.7 Model Selection......Page 134 8.8 Assessment of Between-Group Heterogeneity......Page 135 8.9 Bibliographical Notes......Page 138 9.1 From Multiple Regression Analysis to Path Analysis......Page 142 9.2 Matrix Expression......Page 147 9.3 Distributional Assumptions......Page 148 9.4 Likelihood for Covariance Structure Analysis......Page 149 9.5 Maximum Likelihood Estimation......Page 150 9.6 Estimated Covariance Structure......Page 151 9.8 Other and Extreme Models......Page 153 9.9 Model Selection......Page 155 9.10 Bibliographical Notes......Page 156 10 Confirmatory Factor Analysis......Page 160 10.2 Matrix Expression......Page 161 10.3 Distributional Assumptions for Common Factors......Page 167 10.4 Distributional Assumptions for Errors......Page 168 10.5 Maximum Likelihood Method......Page 169 10.6 Solutions......Page 170 10.7 Other and Extreme Models......Page 171 10.8 Model Selection......Page 172 10.9 Bibliographical Notes......Page 173 11.1 Causality Among Factors......Page 175 11.2 Observed Variables as Indicator of Factors......Page 176 11.4 Matrix Expression......Page 177 11.5 Distributional Assumptions......Page 182 11.6 Maximum Likelihood Method......Page 183 11.7 Solutions......Page 184 11.8 Model Selection......Page 185 11.9 Bibliographical Notes......Page 186 12.1 Example of Exploratory Factor Analysis Model......Page 188 12.2 Matrix Expression......Page 189 12.3 Distributional Assumptions......Page 190 12.5 Indeterminacy of EFA Solutions......Page 191 12.7 Interpretation of Loadings......Page 193 12.9 Selecting the Number of Factors......Page 195 12.10 Difference to Principal Component Analysis......Page 197 12.11 Bibliographical Notes......Page 201 Miscellaneous Procedures......Page 204 13.1 Geometric Illustration of Factor Rotation......Page 205 13.3 Rotation to Simple Structure......Page 208 13.4 Varimax Rotation......Page 211 13.5 Geomin Rotation......Page 212 13.6 Orthogonal Procrustes Rotation......Page 214 13.7 Bibliographical Notes......Page 215 14.1 Block Matrices......Page 218 14.2 Canonical Correlation Analysis......Page 221 14.3 Generalized Canonical Correlation Analysis......Page 224 14.4 Multivariate Categorical Data......Page 228 14.5 Multiple Correspondence Analysis......Page 229 14.6 Homogeneity Assumption......Page 231 14.7 Bibliographical Notes......Page 233 15.1 Modification of Multiple Correspondence Analysis......Page 236 15.2 Canonical Discriminant Analysis......Page 238 15.3 Minimum Distance Classification......Page 240 15.4 Maximum Probability Classification......Page 241 15.5 Normal Discrimination for Two Groups......Page 243 15.6 Interpreting Solutions......Page 245 15.7 Generalized Normal Discrimination......Page 248 15.8 Bibliographical Notes......Page 251 16.1 Linking Coordinates to Quasi-distances......Page 253 16.2 Illustration of an MDS Solution......Page 255 16.3 Iterative Algorithm......Page 256 16.4 Matrix Expression for Squared Distances......Page 257 16.5 Inequality for Distances......Page 259 16.6 Majorization Algorithm......Page 261 16.7 Bibliographical Notes......Page 262 Advanced Procedures......Page 265 17.1 Introductory Systems of Linear Equations......Page 266 17.2 Moore–Penrose Inverse and System of Linear Equations......Page 267 17.3 Singular Value Decomposition and the Moore–Penrose Inverse......Page 269 17.4 Least Squares Problem Solved with Moore–Penrose Inverse......Page 271 17.5 Orthogonal Complement Matrix......Page 273 17.6 Kronecker Product......Page 276 17.7 Khatri–Rao Product......Page 277 17.8 Vec Operator......Page 279 17.9 Hadamard Product......Page 280 17.10 Bibliographical Notes......Page 281 18.1 Matrix Decomposition Formulation......Page 283 18.2 Comparisons to Latent Variable Formulation......Page 286 18.3 Solution of Loadings and Unique Variances......Page 287 18.4 Iterative Algorithm......Page 288 18.5 Estimation of Covariances Between Variables and Factor Scores......Page 289 18.6 Estimation of Loadings and Unique Variances......Page 292 18.7 Identifiability of the Model Part and Residuals......Page 293 18.8 Factor Scores as Higher Rank Approximations......Page 295 18.9 Bibliographical Notes......Page 297 19.1 Motivational Examples......Page 300 19.2 Comparisons of Models......Page 301 19.3 Solutions and Decomposition of the Sum of Squares......Page 303 19.4 Larger Common Part of Principal Component Analysis......Page 307 19.5 Better Fit of Factor Analysis......Page 308 19.6 Largeness of Unique Variances in Factor Analysis......Page 309 19.7 Inequalities for Latent Variable Factor Analysis......Page 310 19.8 Inequalities After Nonsingular Transformation......Page 311 19.9 Proofs for Inequalities......Page 312 19.10 Bibliographical Notes......Page 313 20.1 Tucker3 and Parafac Models......Page 314 20.2 Hierarchical Relationships Among PCA and 3WPCA......Page 316 20.3 Parafac Solution......Page 322 20.4 Tucker3 Solution......Page 325 20.5 Unconstrained Parafac Algorithm......Page 327 20.6 Constrained Parafac Algorithm......Page 330 20.7 Tucker3 Algorithm: The Optimal Core Array......Page 332 20.8 Tucker3 Algorithm: Iterative Solution......Page 334 20.9 Three-Way Rotation in Tucker3......Page 336 20.10 Bibliographical Notes......Page 339 21.1 Illustration of Sparse Solution......Page 343 21.2 Penalized Least Squares Method and Lasso......Page 345 21.3 Coordinate Descent Algorithm for Lasso......Page 346 21.4 Selection of Penalty Weight......Page 350 21.5 L0 Sparse Regression......Page 353 21.6 Standard Regression in Ordinary and High-Dimensional Cases......Page 355 21.7 High-Dimensional Variable Selection by Sparse Regression......Page 358 21.8 Bibliographical Notes......Page 360 22.1 From Confirmatory FA to Sparse FA......Page 362 22.2 Formulation of Penalized Sparse LVFA......Page 364 22.3 Algorithm for Penalized Sparse LVFA......Page 365 22.4 M-Step for Penalized Sparse LVFA......Page 366 22.5 Using Penalized Sparse LVFA......Page 370 22.6 Formulation of Cardinality Constrained MDFA......Page 373 22.7 Algorithm for Cardinality Constrained MDFA......Page 375 22.8 Using Cardinality Constrained MDFA......Page 376 22.9 Sparse FA Versus Factor Rotation in Exploratory FA......Page 378 22.10 Bibliographical Notes......Page 380 A.1.1 Angles Between Vectors......Page 384 Outline placeholder......Page 0 A.1.2 Orthonormal Matrix......Page 385 A.1.3 Vector Space......Page 386 A.1.4 Projection Onto a Subspace......Page 389 A.2.1 Decomposition Using Averages......Page 390 A.2.2 Decomposition Using a Projection Matrix......Page 392 A.3.1 SVD: Extended Version......Page 393 A.3.2 SVD: Compact Version......Page 395 A.3.3 Other Expressions of SVD......Page 396 A.3.4 SVD and Eigenvalue Decomposition for Sysmmetric Matrices......Page 398 A.4.1 ten Berge’s Theorem with Suborthonormal Matrices......Page 399 A.4.2 Maximization of Trace Functions......Page 400 A.4.3 Reduced Rank Approximation......Page 403 A.4.4 Modified Reduced Rank Approximation......Page 404 A.4.5 Modified Versions of Maximizing Trace Functions......Page 408 A.4.6 Obtaining Symmetric Square Roots of Matrices......Page 411 A.5.1 Estimates of Means and Covariances......Page 412 A.5.2 Multiple Groups with Homogeneous Covariances......Page 414 A.6.1 General Methodology......Page 416 A.6.2 Gradient Algorithm for Single Parameter Cases......Page 418 A.6.3 Gradient Algorithm for Multiple Parameter Cases......Page 420 A.7.1 Definition of Scale Invariance......Page 421 A.7.2 Scale Invariance of Factor Analysis......Page 423 A.7.3 Scale Invariance of Path Analysis......Page 425 A.8.1 Joint, Conditional, and Marginal Probability Densities......Page 426 A.8.2 Expected Values......Page 427 A.8.3 Covariances as Expected Values......Page 428 A.8.5 EM Algorithm......Page 430 A.9 EM Algorithm for Factor Analysis......Page 432 A.9.2 Complete Data Log Likelihood......Page 433 A.9.3 Expected Complete Data Log Likelihood......Page 435 A.9.4 E-Step......Page 436 A.9.5 Updating Unique Variances in M-Step......Page 438 A.9.6 Updating Factor Covariance in M-Step for CFA......Page 439 A.9.7 Updating Loadings in M-Step for EFA and CFA......Page 440 A.9.8 Whole Steps in EFA and CFA......Page 441 A.9.9 Algorithm for Penalized Factor Analysis......Page 442 References......Page 444 Index......Page 451