Information geometry

"The book provides a comprehensive introduction and a novel mathematical foundation of the field of information geometry with complete proofs and detailed background material on measure theory, Riemannian geometry and Banach space theory. Parametrised measure models are defined as fundamental geometric objects, which can be both finite or infinite dimensional. Based on these models, canonical tensor fields are introduced and further studied, including the Fisher metric and the Amari-Chentsov tensor, and embeddings of statistical manifolds are investigated.This novel foundation then leads to application highlights, such as generalizations and extensions of the classical uniqueness result of Chentsov or the Cramér-Rao inequality. Additionally, several new application fields of information geometry are highlighted, for instance hierarchical and graphical models, complexity theory, population genetics, or Markov Chain Monte Carlo. The book will be of interest to mathematicians who are interested in geometry, information theory, or the foundations of statistics, to statisticians as well as to scientists interested in the mathematical foundations of complex systems."-- Read more... Information Geometry, doi:10.1007/978-3-319-56478-4 Information Geometry 3 Preface 5 Acknowledgements 7 Contents 8 Chapter 1: Introduction 11 1.1 A Brief Synopsis 11 1.2 An Informal Description 16 1.2.1 The Fisher Metric and the Amari-Chentsov Structure for Finite Sample Spaces 17 1.2.2 Infinite Sample Spaces and Functional Analysis 18 1.2.3 Parametric Statistics 20 1.2.4 Exponential and Mixture Families from the Perspective of Differential Geometry 24 1.2.5 Information Geometry and Information Theory 25 1.3 Historical Remarks 27 1.4 Organization of this Book 30 Chapter 2: Finite Information Geometry 34 2.1 Manifolds of Finite Measures 34 2.2 The Fisher Metric 38 2.3 Gradient Fields 48 2.4 The m- and e-Connections 51 2.5 The Amari-Chentsov Tensor and the alpha-Connections 56 2.5.1 The Amari-Chentsov Tensor 56 2.5.2 The alpha-Connections 59 2.6 Congruent Families of Tensors 61 2.7 Divergences 77 2.7.1 Gradient-Based Approach 77 2.7.2 The Relative Entropy 79 2.7.3 The alpha-Divergence 82 2.7.4 The f-Divergence 85 2.7.5 The q-Generalization of the Relative Entropy 87 2.8 Exponential Families 88 2.8.1 Exponential Families as Affine Spaces 88 2.8.2 Implicit Description of Exponential Families 93 2.8.3 Information Projections 100 2.9 Hierarchical and Graphical Models 109 2.9.1 Interaction Spaces 110 2.9.2 Hierarchical Models 117 2.9.3 Graphical Models 121 Chapter 3: Parametrized Measure Models 129 3.1 The Space of Probability Measures and the Fisher Metric 129 3.2 Parametrized Measure Models 143 3.2.1 The Structure of the Space of Measures 147 3.2.2 Tangent Fibration of Subsets of Banach Manifolds 148 3.2.3 Powers of Measures 151 3.2.4 Parametrized Measure Models and k-Integrability 158 3.2.5 Canonical n-Tensors of an n-Integrable Model 172 3.2.6 Signed Parametrized Measure Models 176 3.3 The Pistone-Sempi Structure 178 3.3.1 e-Convergence 178 3.3.2 Orlicz Spaces 180 3.3.3 Exponential Tangent Spaces 184 Chapter 4: The Intrinsic Geometry of Statistical Models 193 4.1 Extrinsic Versus Intrinsic Geometric Structures 193 4.2 Connections and the Amari-Chentsov Structure 197 4.3 The Duality Between Exponential and Mixture Families 209 4.4 Canonical Divergences 218 4.4.1 Dual Structures via Divergences 218 4.4.2 A General Canonical Divergence 221 4.4.3 Recovering the Canonical Divergence of a Dually Flat Structure 223 4.4.4 Consistency with the Underlying Dualistic Structure 225 4.5 Statistical Manifolds and Statistical Models 227 4.5.1 Statistical Manifolds and Isostatistical Immersions 228 4.5.2 Monotone Invariants of Statistical Manifolds 231 4.5.3 Immersion of Compact Statistical Manifolds into Linear Statistical Manifolds 234 4.5.4 Proof of the Existence of Isostatistical Immersions 236 4.5.5 Existence of Statistical Embeddings 246 Chapter 5: Information Geometry and Statistics 248 5.1 Congruent Embeddings and Sufficient Statistics 248 5.1.1 Statistics and Congruent Embeddings 251 5.1.2 Markov Kernels and Congruent Markov Embeddings 260 5.1.3 Fisher-Neyman Sufficient Statistics 268 5.1.4 Information Loss and Monotonicity 270 5.1.5 Chentsov's Theorem and Its Generalization 275 5.2 Estimators and the Cramér-Rao Inequality 284 5.2.1 Estimators and Their Bias, Mean Square Error, Variance 284 5.2.2 A General Cramér-Rao Inequality 288 5.2.3 Classical Cramér-Rao Inequalities 293 5.2.4 Efficient Estimators and Consistent Estimators 294 Chapter 6: Fields of Application of Information Geometry 301 6.1 Complexity of Composite Systems 301 6.1.1 A Geometric Approach to Complexity 302 6.1.2 The Information Distance from Hierarchical Models 304 6.1.2.1 The Information Distance as a Complexity Measure 304 6.1.2.2 The Maximization of the Information Distance 308 6.1.3 The Weighted Information Distance 313 6.1.3.1 The General Scheme 313 6.1.3.2 The Size Hierarchy of Subsets: The TSE-Complexity 314 6.1.3.3 The Length Hierarchy of Subintervals: The Excess Entropy 318 6.1.4 Complexity of Stochastic Processes 323 6.1.4.1 Multi-Information Rate, Transfer Entropy, and Directed Information 323 6.1.4.2 The Information Distance from Hierarchical Models of Markov Kernels 325 6.2 Evolutionary Dynamics 333 6.2.1 Natural Selection and Replicator Equations 334 6.2.2 Continuous Time Limits 339 6.2.3 Population Genetics 342 6.3 Monte Carlo Methods 354 6.3.1 Langevin Monte Carlo 356 6.3.2 Hamiltonian Monte Carlo 357 6.4 Infinite-Dimensional Gibbs Families 360 Appendix A: Measure Theory 367 Appendix B: Riemannian Geometry 373 Appendix C: Banach Manifolds 386 References 392 Index 402 Nomenclature 407 Information Geometry Preface Acknowledgements Contents Chapter 1: Introduction 1.1 A Brief Synopsis 1.2 An Informal Description 1.2.1 The Fisher Metric and the Amari-Chentsov Structure for Finite Sample Spaces 1.2.2 In nite Sample Spaces and Functional Analysis 1.2.3 Parametric Statistics 1.2.4 Exponential and Mixture Families from the Perspective of Differential Geometry 1.2.5 Information Geometry and Information Theory 1.3 Historical Remarks 1.4 Organization of this Book Chapter 2: Finite Information Geometry 2.1 Manifolds of Finite Measures 2.2 The Fisher Metric 2.3 Gradient Fields2.4 The m- and e-Connections 2.5 The Amari-Chentsov Tensor and the alpha-Connections 2.5.1 The Amari-Chentsov Tensor 2.5.2 The alpha-Connections 2.6 Congruent Families of Tensors 2.7 Divergences 2.7.1 Gradient-Based Approach 2.7.2 The Relative Entropy 2.7.3 The alpha-Divergence 2.7.4 The f-Divergence 2.7.5 The q-Generalization of the Relative Entropy 2.8 Exponential Families 2.8.1 Exponential Families as Af ne Spaces 2.8.2 Implicit Description of Exponential Families 2.8.3 Information Projections 2.9 Hierarchical and Graphical Models 2.9.1 Interaction Spaces 2.9.2 Hierarchical Models2.9.3 Graphical Models Chapter 3: Parametrized Measure Models 3.1 The Space of Probability Measures and the Fisher Metric 3.2 Parametrized Measure Models 3.2.1 The Structure of the Space of Measures 3.2.2 Tangent Fibration of Subsets of Banach Manifolds 3.2.3 Powers of Measures 3.2.4 Parametrized Measure Models and k-Integrability 3.2.5 Canonical n-Tensors of an n-Integrable Model 3.2.6 Signed Parametrized Measure Models 3.3 The Pistone-Sempi Structure 3.3.1 e-Convergence 3.3.2 Orlicz Spaces 3.3.3 Exponential Tangent Spaces Chapter 4: The Intrinsic Geometry of Statistical Models4.1 Extrinsic Versus Intrinsic Geometric Structures 4.2 Connections and the Amari-Chentsov Structure 4.3 The Duality Between Exponential and Mixture Families 4.4 Canonical Divergences 4.4.1 Dual Structures via Divergences 4.4.2 A General Canonical Divergence 4.4.3 Recovering the Canonical Divergence of a Dually Flat Structure 4.4.4 Consistency with the Underlying Dualistic Structure 4.5 Statistical Manifolds and Statistical Models 4.5.1 Statistical Manifolds and Isostatistical Immersions 4.5.2 Monotone Invariants of Statistical Manifolds4.5.3 Immersion of Compact Statistical Manifolds into Linear Statistical Manifolds 4.5.4 Proof of the Existence of Isostatistical Immersions 4.5.5 Existence of Statistical Embeddings Chapter 5: Information Geometry and Statistics 5.1 Congruent Embeddings and Suf cient Statistics 5.1.1 Statistics and Congruent Embeddings 5.1.2 Markov Kernels and Congruent Markov Embeddings 5.1.3 Fisher-Neyman Suf cient Statistics 5.1.4 Information Loss and Monotonicity 5.1.5 Chentsov's Theorem and Its Generalization "The book provides a comprehensive introduction and a novel mathematical foundation of the field of information geometry with complete proofs and detailed background material on measure theory, Riemannian geometry and Banach space theory. Parametrised measure models are defined as fundamental geometric objects, which can be both finite or infinite dimensional. Based on these models, canonical tensor fields are introduced and further studied, including the Fisher metric and the Amari-Chentsov tensor, and embeddings of statistical manifolds are investigated.This novel foundation then leads to application highlights, such as generalizations and extensions of the classical uniqueness result of Chentsov or the Cramér-Rao inequality. Additionally, several new application fields of information geometry are highlighted, for instance hierarchical and graphical models, complexity theory, population genetics, or Markov Chain Monte Carlo. The book will be of interest to mathematicians who are interested in geometry, information theory, or the foundations of statistics, to statisticians as well as to scientists interested in the mathematical foundations of complex systems."-- Provided by publisher "The book provides a comprehensive introduction and a novel mathematical foundation of the field of information geometry with complete proofs and detailed background material on measure theory, Riemannian geometry and Banach space theory. Parametrised measure models are defined as fundamental geometric objects, which can be both finite or infinite dimensional. Based on these models, canonical tensor fields are introduced and further studied, including the Fisher metric and the Amari-Chentsov tensor, and embeddings of statistical manifolds are investigated. This novel foundation then leads to application highlights, such as generalizations and extensions of the classical uniqueness result of Chentsov or the Cramér-Rao inequality. Additionally, several new application fields of information geometry are highlighted, for instance hierarchical and graphical models, complexity theory, population genetics, or Markov Chain Monte Carlo. The book will be of interest to mathematicians who are interested in geometry, information theory, or the foundations of statistics, to statisticians as well as to scientists interested in the mathematical foundations of complex systems."