Mathematics for machine learning

Foreword......Page 7 Part I Mathematical Foundations......Page 15 1 Introduction and Motivation......Page 17 1.1 Finding Words for Intuitions......Page 18 1.2 Two Ways to Read This Book......Page 19 1.3 Exercises and Feedback......Page 22 2 Linear Algebra......Page 23 2.1 Systems of Linear Equations......Page 25 2.2 Matrices......Page 28 2.3 Solving Systems of Linear Equations......Page 33 2.4 Vector Spaces......Page 41 2.5 Linear Independence......Page 46 2.6 Basis and Rank......Page 50 2.7 Linear Mappings......Page 54 2.8 Affine Spaces......Page 67 Exercises......Page 69 3 Analytic Geometry......Page 76 3.1 Norms......Page 77 3.2 Inner Products......Page 78 3.3 Lengths and Distances......Page 81 3.4 Angles and Orthogonality......Page 82 3.5 Orthonormal Basis......Page 84 3.6 Orthogonal Complement......Page 85 3.7 Inner Product of Functions......Page 86 3.8 Orthogonal Projections......Page 87 3.9 Rotations......Page 97 3.10 Further Reading......Page 100 Exercises......Page 101 4 Matrix Decompositions......Page 104 4.1 Determinant and Trace......Page 105 4.2 Eigenvalues and Eigenvectors......Page 111 4.3 Cholesky Decomposition......Page 120 4.4 Eigendecomposition and Diagonalization......Page 121 4.5 Singular Value Decomposition......Page 125 4.6 Matrix Approximation......Page 135 4.7 Matrix Phylogeny......Page 140 4.8 Further Reading......Page 141 Exercises......Page 143 5 Vector Calculus......Page 145 5.1 Differentiation of Univariate Functions......Page 147 5.2 Partial Differentiation and Gradients......Page 152 5.3 Gradients of Vector-Valued Functions......Page 155 5.4 Gradients of Matrices......Page 161 5.5 Useful Identities for Computing Gradients......Page 164 5.6 Backpropagation and Automatic Differentiation......Page 165 5.7 Higher-Order Derivatives......Page 170 5.8 Linearization and Multivariate Taylor Series......Page 171 Exercises......Page 176 6.1 Construction of a Probability Space......Page 178 6.2 Discrete and Continuous Probabilities......Page 184 6.3 Sum Rule, Product Rule, and Bayes' Theorem......Page 189 6.4 Summary Statistics and Independence......Page 192 6.5 Gaussian Distribution......Page 203 6.6 Conjugacy and the Exponential Family......Page 211 6.7 Change of Variables/Inverse Transform......Page 220 6.8 Further Reading......Page 227 Exercises......Page 228 7 Continuous Optimization......Page 231 7.1 Optimization Using Gradient Descent......Page 233 7.2 Constrained Optimization and Lagrange Multipliers......Page 239 7.3 Convex Optimization......Page 242 7.4 Further Reading......Page 252 Exercises......Page 253 Part II Central Machine Learning Problems......Page 255 8.1 Data, Models, and Learning......Page 257 8.2 Empirical Risk Minimization......Page 264 8.3 Parameter Estimation......Page 271 8.4 Probabilistic Modeling and Inference......Page 278 8.5 Directed Graphical Models......Page 284 8.6 Model Selection......Page 289 9 Linear Regression......Page 295 9.1 Problem Formulation......Page 297 9.2 Parameter Estimation......Page 298 9.3 Bayesian Linear Regression......Page 309 9.4 Maximum Likelihood as Orthogonal Projection......Page 319 9.5 Further Reading......Page 321 10 Dimensionality Reduction with Principal Component Analysis......Page 323 10.1 Problem Setting......Page 324 10.2 Maximum Variance Perspective......Page 326 10.3 Projection Perspective......Page 331 10.4 Eigenvector Computation and Low-Rank Approximations......Page 339 10.5 PCA in High Dimensions......Page 341 10.6 Key Steps of PCA in Practice......Page 342 10.7 Latent Variable Perspective......Page 345 10.8 Further Reading......Page 349 11 Density Estimation with Gaussian Mixture Models......Page 354 11.1 Gaussian Mixture Model......Page 355 11.2 Parameter Learning via Maximum Likelihood......Page 356 11.3 EM Algorithm......Page 366 11.4 Latent-Variable Perspective......Page 369 11.5 Further Reading......Page 374 12 Classification with Support Vector Machines......Page 376 12.1 Separating Hyperplanes......Page 378 12.2 Primal Support Vector Machine......Page 380 12.3 Dual Support Vector Machine......Page 389 12.4 Kernels......Page 394 12.5 Numerical Solution......Page 396 12.6 Further Reading......Page 398 References......Page 401 Index......Page 413