Pattern Recognition and Machine Learning (Information Science and Statistics)

The dramatic growth in practical applications for machine learning over the last ten years has been accompanied by many important developments in the underlying algorithms and techniques. For example, Bayesian methods have grown from a specialist niche to become mainstream, while graphical models have emerged as a general framework for describing and applying probabilistic techniques. The practical applicability of Bayesian methods has been greatly enhanced by the development of a range of approximate inference algorithms such as variational Bayes and expectation propagation, while new models based on kernels have had a significant impact on both algorithms and applications. This completely new textbook reflects these recent developments while providing a comprehensive introduction to the fields of pattern recognition and machine learning. It is aimed at advanced undergraduates or first-year PhD students, as well as researchers and practitioners. No previous knowledge of pattern recognition or machine learning concepts is assumed. Familiarity with multivariate calculus and basic linear algebra is required, and some experience in the use of probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory. The book is suitable for courses on machine learning, statistics, computer science, signal processing, computer vision, data mining, and bioinformatics. Extensive support is provided for course instructors, including more than 400 exercises, graded according to difficulty. Example solutions for a subset of the exercises are available from the book web site, while solutions for the remainder can be obtained by instructors from the publisher. The book is supported by a great deal of additional material, and the reader is encouraged to visit the book web site for the latest information. Coming soon: *For students, worked solutions to a subset of exercises available on a public web site (for exercises marked "www" in the text) *For instructors, worked solutions to remaining exercises from the Springer web site *Lecture slides to accompany each chapter *Data sets available for download Introduction. Example : Polynomial Curve Fitting ; Probability Theory ; Model Selection ; The Curse Of Dimensionality Decision Theory ; Information Theory -- Probability Distributions. Binary Vehicles ; Multinomial Variables ; The Gaussian Distribution ; The Exponential Family ; Nonparametric Methods -- Linear Models For Regression. Linear Basis Function Models ; The Bias-variance Decomposition ; Bayesian Linear Regression ; Bayesian Model Comparison ; The Evidence Approximation ; Limitations Of Fixed Basis Functions -- Linear Models For Classification. Discriminant Functions ; Probabilistic Generative Models ; Probabilistic Discrimitive Models ; The Laplace Approximation ; Bayesian Logistic Regression -- Neural Networks. Feed-forward Network Functions ; Network Training ; Error Backpropagation ; The Hessian Matrix ; Regularization In Neural Networks ; Mixture Density Networks ; Bayesian Neural Networks. Kernel Methods. Dual Representations ; Constructing Kernals ; Radial Basis Function Networks ; Gaussian Processes -- Sparse Kernel Machines. Maximum Margin Classifiers ; Relevance Vector Machines -- Graphical Models. Bayesian Networks ; Conditional Independence ; Markov Random Fields ; Inference In Graphical Models -- Mixture Models And Em. K-means Clustering ; Mixtures Of Gaussians ; An Alternative View Of Em ; The Em Algorithm In General -- Approximate Inference. Variational Inference ; Illustration : Variational Mixture Of Gaussians ; Variational Linear Regression ; Exponential Family Distributions ; Local Variational Methods ; Variational Logistic Regression ; Expectation Propagation -- Sampling Methods. Basic Sampling Algorithms ; Markov Chain Monte Carlo ; Gibbs Sampling ; Slice Sampling ; The Hybrid Monte Carlo Algorithm ; Estimating The Partition Function. Continuous Latent Variables. Principal Component Analysis ; Probabilistic Pca ; Kernel Pca ; Nonlinear Latent Variable Models -- Sequential Data. Markoc Models ; Hidden Markov Models ; Linear Dynamical Systems -- Combining Models. Bayesian Model Averaging ; Committees ; Boosting ; Tree-based Models ; Conditional Mixture Models -- Data Sets -- Probability Distributions -- Properties Of Matrices -- Calculus Of Variations -- Lagrange Multipliers. Christopher M. Bishop. Includes Bibliographical References (p. 711-728) And Index. The dramatic growth in practical applications for machine learning over the last ten years has been accompanied by many important developments in the underlying algorithms and techniques. For example, Bayesian methods have grown from a specialist niche to become mainstream, while graphical models have emerged as a general framework for describing and applying probabilistic techniques. The practical applicability of Bayesian methods has been greatly enhanced by the development of a range of approximate inference algorithms such as variational Bayes and expectation propagation, while new models based on kernels have had a significant impact on both algorithms and applications. This completely new textbook reflects these recent developments while providing a comprehensive introduction to the fields of pattern recognition and machine learning. It is aimed at advanced undergraduates or first-year PhD students, as well as researchers and practitioners. No previous knowledge of pattern recognition or machine learning concepts is assumed. Familiarity with multivariate calculus and basic linear algebra is required, and some experience in the use of probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory. The book is suitable for courses on machine learning, statistics, computer science, signal processing, computer vision, data mining, and bioinformatics. Extensive support is provided for course instructors, including more than 400 exercises, graded according to difficulty. Example solutions for a subset of the exercises are available from the book web site, while solutions for the remainder can be obtained by instructors from the publisher. The book is supported by a great deal of additional material, and the reader is encouraged to visit the book web site for the latest information. Christopher M. Bishop is Deputy Director of Microsoft Research Cambridge, and holds a Chair in Computer Science at the University of Edinburgh. He is a Fellow of Darwin College Cambridge, a Fellow of the Royal Academy of Engineering, and a Fellow of the Royal Society of Edinburgh. His previous textbook "Neural Networks for Pattern Recognition" has been widely adopted Pattern recognition has its origins in engineering, whereas machine learning grew out of computer science. However, these activities can be viewed as two facets of the same field, and together they have undergone substantial development over the past ten years. In particular, Bayesian methods have grown from a specialist niche to become mainstream, while graphical models have emerged as a general framework for describing and applying probabilistic models. Also, the practical applicability of Bayesian methods has been greatly enhanced through the development of a range of approximate inference algorithms such as variational Bayes and expectation propagation. Similarly, new models based on kernels have had a significant impact on both algorithms and applications. This new textbook reflects these recent developments while providing a comprehensive introduction to the fields of pattern recognition and machine learning. It is aimed at advanced undergraduates or first-year PhD students, as well as researchers and practitioners, and assumes no previous knowledge of pattern recognition or machine learning concepts. Knowledge of multivariate calculus and basic linear algebra is required, and some familiarity with probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory. Pattern Recognition and Machine Learning......Page 4 Preface......Page 7 Mathematical notation......Page 10 Contents......Page 12 1. Introduction ......Page 20 2. Probability Distributions ......Page 86 3. Linear Models for Regression ......Page 156 4. Linear Models for Classification ......Page 197 5. Neural Networks ......Page 243 6. Kernel Methods ......Page 309 7. Sparse Kernel Machines ......Page 342 8. Graphical Models ......Page 376 9. Mixture Models and EM ......Page 440 10. Approximate Inference ......Page 477 11. Sampling Methods ......Page 539 12. Continuous Latent Variables ......Page 575 13. Sequential Data ......Page 620 14. Combining Models ......Page 668 Appendix A. Data Sets ......Page 692 Appendix B. Probability Distributions ......Page 699 Appendix C. Properties of Matrices ......Page 708 Appendix D. Calculus of Variations ......Page 715 Appendix E. Lagrange Multipliers ......Page 718 References ......Page 722 Index ......Page 740 Pattern Recognition and Machine Learning 4 Preface 7 Mathematical notation 10 Contents 12 Chapters 20 1. Introduction 20 2. Probability Distributions 86 3. Linear Models for Regression 156 4. Linear Models for Classification 197 5. Neural Networks 243 6. Kernel Methods 309 7. Sparse Kernel Machines 342 8. Graphical Models 376 9. Mixture Models and EM 440 10. Approximate Inference 477 11. Sampling Methods 539 12. Continuous Latent Variables 575 13. Sequential Data 620 14. Combining Models 668 Appendix A. Data Sets 692 Appendix B. Probability Distributions 699 Appendix C. Properties of Matrices 708 Appendix D. Calculus of Variations 715 Appendix E. Lagrange Multipliers 718 References 722 Index 740 The field of pattern recognition has undergone substantial development over the years. This book reflects these developments while providing a grounding in the basic concepts of pattern recognition and machine learning. It is aimed at advanced undergraduates or first year PhD students, as well as researchers and practitioners