Information Theory: A Tutorial Introduction (2nd Edition)

Originally developed by Claude Shannon in the 1940s, information theory laid the foundations for the digital revolution, and is now an essential tool in telecommunications, genetics, linguistics, brain sciences, and deep space communication. In this richly illustrated book, accessible examples are used to introduce information theory in terms of everyday games like ‘20 questions’ before more advanced topics are explored. Online MatLab and Python computer programs provide hands-on experience of information theory in action, and PowerPoint slides give support for teaching. Written in an informal style, with a comprehensive glossary and tutorial appendices, this text is an ideal primer for novices who wish to learn the essential principles and applications of information theory. This Second Edition includes two new chapters on on the theory and applications of Transfer Entropy and Rate Distortion Theory. Preface What Is Information? Introduction Entropy of Discrete Variables Introduction Ground Rules and Terminology Shannon's Desiderata Information, Surprise and Entropy Evaluating Entropy Properties of Entropy Independent and Identically Distributed Values Bits, Shannons, and Bans Summary The Source Coding Theorem Introduction Capacity of a Discrete Noiseless Channel Shannon's Source Coding Theorem Calculating Information Rates Data Compression Huffman Coding The Entropy of English Letters Why the Theorem is True Kolmogorov Complexity Summary The Noisy Channel Coding Theorem Introduction Joint Distributions Mutual Information Conditional Entropy Noise and Cross-Talk Noisy Pictures and Coding Efficiency Error Correcting Codes Shannon's Noisy Channel Coding Theorem Rate Distortion Theory The Noisy Typewriter Why the Theorem is True Summary Entropy of Continuous Variables Introduction The Trouble With Entropy Differential Entropy Under-Estimating Entropy Properties of Differential Entropy Maximum Entropy Distributions Making Sense of Differential Entropy What is Half a Bit of Information? Summary Mutual Information: Continuous Introduction Joint Distributions Conditional Distributions and Entropy Mutual Information and Conditional Entropy Mutual Information is Invariant Kullback–Leibler Divergence and Bayes Summary Channel Capacity: Continuous Introduction Channel Capacity The Gaussian Channel Error Rates of Noisy Channels Using a Gaussian Channel Mutual Information and Correlation The Fixed Range Channel Summary Rate Distortion Theory Introduction Informal Summary Rate Distortion Theory The Binary Rate Distortion Function The Gaussian Rate Distortion Function Deterministic versus Stochastic Encoding Image Compression Example Applications Summary Transfer Entropy Introduction Transfer Entropy The Pendulum Numerical Example Summary Thermodynamic Entropy and Information Introduction Physics, Entropy and Disorder Information and Thermodynamic Entropy Ensembles, Macrostates and Microstates Pricing Information: The Landauer Limit The Second Law of Thermodynamics Maxwell's Demon Quantum Computation Summary Information As Nature's Currency Introduction Satellite TVs, MP3 and All That Does Sex Accelerate Evolution? The Human Genome: How Much Information? Enough DNA to Wire Up a Brain? Are Brains Good at Processing Information? A Very Short History of Information Theory Summary Further Reading Glossary Mathematical Symbols Logarithms Probability Density Functions Averages From Distributions The Rules of Probability The Gaussian Distribution Key Equations Bibliography