Fundamental Mathematical Concepts for Machine Learning in Science

This book is for individuals with a scientific background who aspire to apply machine learning within various natural science disciplines—such as physics, chemistry, biology, medicine, psychology and many more. It elucidates core mathematical concepts in an accessible and straightforward manner, maintaining rigorous mathematical integrity. For readers more versed in mathematics, the book includes advanced sections that are not prerequisites for the initial reading. It ensures concepts are clearly defined and theorems are proven where it's pertinent. Machine learning transcends the mere implementation and training of algorithms; it encompasses the broader challenges of constructing robust datasets, model validation, addressing imbalanced datasets, and fine-tuning hyperparameters. These topics are thoroughly examined within the text, along with the theoretical foundations underlying these methods. Rather than concentrating on particular algorithms this book focuses on the comprehensive concepts and theories essential for their application. It stands as an indispensable resource for any scientist keen on integrating machine learning effectively into their research. Numerous texts delve into the technical execution of machine learning algorithms, often overlooking the foundational concepts vital for fully grasping these methods. This leads to a gap in using these algorithms effectively across diverse disciplines. For instance, a firm grasp of calculus is imperative to comprehend the training processes of algorithms and neural networks, while linear algebra is essential for the application and efficient training of various algorithms, including neural networks. Absent a solid mathematical base, machine learning applications may be, at best, cursory, or at worst, fundamentally flawed. This book lays the foundation for a comprehensive understanding of machine learning algorithms and approaches. Preface Acknowledgements Contents Acronyms Chapter 1 Introduction 1.1 Introduction 1.2 Choice of Topics 1.3 Prerequisites 1.4 Book Structure 1.5 About This Book 1.6 Warnings, Info and Examples 1.7 Optional and Advanced Material 1.8 Further Exploration and Reading 1.9 References 1.10 Let us Start References Chapter 2 Machine Learning: History and Terminology 2.1 Brief History of Machine Learning 2.2 Machine Learning in Science 2.3 Types of Machine Learning References Chapter 3 Calculus and Optimisation for Machine Learning 3.1 Motivation 3.2 Concept of Limit 3.3 Derivative and its Properties 3.4 Partial Derivative 3.5 Gradient 3.6 Extrema of a Function 3.7 Optimisation for Machine Learning 3.8 Introduction to Optimisation for Neural Networks 3.8.1 First Definition of Learning 3.8.2 Constrained vs. Unconstrained Optimisation 3.9 Optimization Algorithms 3.9.1 Line Search and Trust Region Approaches 3.9.2 Steepest Descent 3.9.3 Additional Directions for the Line Search Approach 3.9.4 The Gradient Descent Algorithm 3.9.5 Choosing the Right Learning Rate 3.9.6 Variations of Gradient Descent 3.9.6.1 Mini-batch Gradient Descent 3.9.6.2 Stochastic Gradient Descent 3.9.7 How to Choose the Right Mini-batch Size 3.9.8 Stochastic Gradient Descent and Fractals 3.10 Conclusions References Chapter 4 Linear Algebra 4.1 Motivation 4.2 Vectors 4.2.1 Geometrical Interpretation of Vectors 4.2.2 Norm of Vectors 4.2.3 Dot Product 4.2.4 Cross Product 4.3 Matrices 4.3.1 Sum, Subtraction and Transpose 4.3.2 Multiplication of Matrices and Vectors 4.3.3 Inverse and Trace 4.3.4 Determinant 4.3.5 Matrix Calculus and Linear Regression 4.4 Relevance for Machine Learning 4.5 Eigenvectors and Eigenvalues 4.6 Principal Component Analysis 4.6.1 Basis of a Vector Space 4.6.2 Definition of a Vector Space 4.6.3 Linear Transformations (maps) 4.6.4 PCA Formalisation 4.6.5 Covariance Matrix 4.6.6 Overview of Assumptions 4.6.7 PCA with Eigenvectors and Eigenvalues 4.6.8 One Implementation Limitation References Chapter 5 Statistics and Probability for Machine Learning 5.1 Motivation 5.2 Random Experiments and Variables 5.3 Algebra of Sets 5.4 Probability 5.4.1 Relative Frequency Interpretation of Probability 5.4.2 Probability as a Set Function 5.4.3 Axiomatic Definition of Probability 5.4.4 Properties of Probability Functions 5.5 The Softmax Function 5.5.1 Softmax Range of Applications 5.6 Some Theorems about Probability Functions 5.7 Conditional Probability 5.8 Bayes Theorem 5.9 Bayes Error 5.10 Naïve Bayes Classifier 5.11 Distribution Functions 5.11.1 Cumulative Distribution Function (CDF) 5.11.2 Probability Density (PDF) and Mass Functions (PMF) 5.12 Expected Values and its Properties 5.13 Variance and its Properties 5.13.1 Properties 5.14 Normal Distribution 5.15 Other Distributions 5.16 The MSE and its Distribution 5.16.1 Moment Generating Functions 5.16.2 Central Limit Theorem 5.17 Central Limit Theorem without Mathematics References Chapter 6 Sampling Theory (a.k.a. Creating a Dataset Properly) 6.1 Introduction 6.2 Research Questions and Hypotheses 6.2.1 Research Questions 6.2.2 Hypothesis 6.2.3 Relevance of Hypothesis and Research Questions in Machine Learning 6.3 Survey Populations 6.4 Survey Samples 6.4.1 Non-probability Sampling 6.4.2 Probability Sampling 6.5 Stratification and Clustering 6.6 Random Sampling without Replacement 6.7 Random Sampling with Replacement 6.8 Bootstrapping 6.9 Random Stratified Sampling 6.10 Sampling in Machine Learning References Chapter 7 Model Validation and Selection 7.1 Introduction 7.2 Bias-Variance Tradeoff 7.3 Bias-Variance Tradeoff - a Mathematical Discussion 7.4 High-Variance Low-Bias regime 7.5 Low-Variance High-Bias regime 7.6 Overfitting and Underfitting 7.7 The Simple Split Approach (a.k.a. Hold-out Approach) 7.8 Data Leakage 7.8.1 Data Leakage with Correlated Observations 7.8.2 Stratified Sampling 7.9 Monte Carlo Cross-Validation 7.10 Monte-Carlo Cross Validation with Bootstrapping 7.11 k-Fold Cross Validation 7.12 The Leave-One-Out Approach 7.13 Choosing the Cross-Validation Approach 7.14 Model Selection 7.14.1 Model Selection with Supervised Learning 7.14.2 Model Selection with Unsupervised Learning 7.15 Qualitative Criteria for Model Selection References Chapter 8 Unbalanced Datasets and Machine Learning Metrics 8.1 Introduction 8.2 A Simple Example 8.3 Approaches to Deal with Unbalanced Datasets 8.3.1 Oversampling 8.3.2 (Random) Undersampling 8.4 Synthetic Minority Oversampling TEchnique (SMOTE) 8.5 Summary of Methods for Dealing with Unbalanced Datasets 8.6 Important Metrics 8.6.1 The Notion of Metric 8.6.1.1 .The MSE is a Metric 8.6.1.2 . 1 − a is a Metric 8.6.2 Confusion Matrix 8.6.3 Sensitivity and Specificity 8.6.4 Precision 8.6.5 Fβ-score 8.6.6 Balanced Accuracy 8.6.7 Receiving Operating Characteristic (ROC) Curve 8.6.8 Probability Interpretation of the AUC References Chapter 9 Hyper-parameter Tuning 9.1 Introduction 9.2 Black-box Optimisation 9.2.1 Notes on Black-box Functions 9.3 The Problem of Hyper-parameter Tuning 9.4 Sample Black-box Problem 9.4.1 Grid Search 9.4.2 Random Search 9.4.3 Coarse to Fine Optimisation 9.4.4 Sampling on a Logarithmic Scale 9.5 Overview of Approaches for Hyper-parameter Tuning References Chapter 10 Feature Importance and Selection 10.1 Introduction 10.2 Feature Importance Taxonomy 10.2.1 Filter Methods 10.2.2 Wrapper Methods 10.2.3 Embedded Methods 10.3 Forward Feature Selection 10.3.1 Forward Feature Selection Practical Example 10.4 Backward Feature Elimination 10.5 Permutation Feature Importance 10.6 Information Content Elimination 10.7 Summary 10.8 SHapley Additive exPlanations (SHAP) References Index