Deep Learning and Computational Physics

The main objective of this book is to introduce a student who is familiar with elementary math concepts to select topics in deep learning. It exploits strong connections between deep learning algorithms and the techniques of computational physics to achieve two important goals. First, it uses concepts from computational physics to develop an understanding of deep learning algorithms. Second, it describes several novel deep learning algorithms for solving challenging problems in computational physics, thereby offering someone who is interested in modeling physical phenomena with a complementary set of tools. It is intended for senior undergraduate and graduate students in science and engineering programs. It is used as a textbook for a course (or a course sequence) for senior-level undergraduate or graduate-level students. Preface Acknowledgements Contents About the Authors 1 Introduction 1.1 Computational Physics 1.2 Machine Learning 1.2.1 Examples of ML 1.2.2 Types of ML Algorithms Based Tasks 1.3 Artificial Intelligence, Machine Learning and Deep Learning 1.4 Machine Learning and Computational Physics 1.5 Computational Exercise 2 Introduction to Deep Neural Networks 2.1 MLP Architecture 2.2 Activation Functions 2.2.1 Linear Activation 2.2.2 Rectified Linear Unit (ReLU) 2.2.3 Leaky ReLU 2.2.4 Logistic Function 2.2.5 Tanh 2.2.6 Sine 2.3 Expressivity of a Network 2.3.1 Universal Approximation Results 2.4 Training, Validation and Testing of Neural Networks 2.5 Overfitting and How to Avoid It 2.5.1 Regularization 2.6 Gradient Descent 2.6.1 Convergence 2.7 Some Advanced Optimization Algorithms 2.7.1 Momentum Methods 2.7.2 Adam 2.7.3 Stochastic Optimization 2.8 Calculating Gradients Using Back-Propagation 2.9 Regression Versus Classification 2.10 Computational Exercise 2.10.1 Expressivity of Deep Neural Networks 2.10.2 Training an MLP for a Regression Problem 3 Residual Neural Networks 3.1 Vanishing Gradients in Deep Networks 3.2 ResNets 3.3 Connections with ODEs 3.4 Neural ODEs 4 Convolutional Neural Networks 4.1 Functions and Images 4.2 Convolutions of Functions 4.2.1 Example 1 4.2.2 Example 2 4.3 Discrete Convolutions 4.4 Connection to Finite Difference Approximations 4.5 Convolution Layers 4.5.1 Average and Max Pooling 4.5.2 Convolution for Inputs with Multiple Channels 4.6 Convolution Neural Network (CNN) 4.7 Transpose Convolution Layers 4.8 UpSampling 4.9 Image-to-Image Transformations 4.10 Computational Exercise: Convolutional Neural Networks (CNNs) 5 Solving PDEs with Neural Networks 5.1 Finite Difference Method 5.2 Spectral Collocation Method 5.3 Physics-Informed Neural Networks (PINNs) 5.4 Extending PINNs to a More General PDE 5.5 Error Analysis for PINNs 5.6 Data Assimilation Using PINNs 5.7 Some Existing PINN Formulations 5.8 Computational Exercise: Physics Informed Neural Networks (PINNs) 6 Operator Networks 6.1 Parametrized PDEs 6.2 Operators 6.3 Deep Operator Network (DeepONet) Architecture 6.3.1 Training DeepONets 6.3.2 Error Analysis for DeepONets 6.4 Physics-Informed DeepONets 6.5 DeepONets and Their Applications 6.6 Fourier Neural Operator (FNO) 6.6.1 Discretization of the Fourier Neural Operator 6.6.2 The Use of Fourier Transforms 6.7 Variationally Mimetic Operator Network (VarMiON) 6.7.1 Background 6.7.2 VarMiON Architecture 6.7.3 Training the VarMiON 6.7.4 Error Estimates of VarMiON Approximation 6.8 Mesh Graph Networks 6.8.1 Background 6.8.2 Architecture of MGNs 6.8.3 Training MGNs 6.9 Computational Exercise: Deep Operator Networks (DeepONets) 7 Generative Deep Learning 7.1 Generative Algorithms 7.2 Introductory Concepts in Probability 7.2.1 Random Variables 7.2.2 Cumulative Distribution Function 7.2.3 Probability Density Function 7.2.4 Examples of Important Random Variables 7.2.5 Expectation and Variance of RVs 7.2.6 Random Vectors 7.2.7 Joint Probability Density Function 7.2.8 Examples of Important Random Vectors 7.2.9 Expectation and Covariance of Random Vectors 7.2.10 Marginal and Conditional Probability Density Functions 7.3 Pure Generative Problem 7.3.1 GANs 7.3.2 Score-Based Diffusion Models 7.4 Conditional Generative Algorithms 7.4.1 Conditional GANs 7.4.2 Conditional Diffusion Models Appendix References