Deep and Shallow : Machine Learning in Music and Audio

Providing an essential and unique bridge between the theories of signal processing, machine learning, and artificial intelligence (AI) in music, this book provides a holistic overview of foundational ideas in music, from the physical and mathematical properties of sound to symbolic representations. Combining signals and language models in one place, this book explores how sound may be represented and manipulated by computer systems, and how our devices may come to recognize particular sonic patterns as musically meaningful or creative through the lens of information theory. Introducing popular fundamental ideas in AI at a comfortable pace, more complex discussions around implementations and implications in musical creativity are gradually incorporated as the book progresses. Each chapter is accompanied by guided programming activities designed to familiarize readers with practical implications of discussed theory, without the frustrations of free-form coding. Surveying state-of-the art methods in applications of deep neural networks to audio and sound computing, as well as offering a research perspective that suggests future challenges in music and AI research, this book appeals to both students of AI and music, as well as industry professionals in the fields of machine learning, music, and AI. Cover Endorsements Half Title Series Page Title Page Copyright Page Dedication Contents Preface Chapter 1: Introduction to Sounds of Music 1.1. From Sound to Features 1.2. Representation of Sound and Music Data 1.3. Acoustics and Basics of Notes 1.4. Perception and Cognition: Anticipation and Principles of Music Information Dynamics 1.5. Exercises Chapter 2: Noise: The Hidden Dynamics of Music 2.1. Noise, Aleatoric Music, and Generative Processes in Music 2.2. History of Mathematical Theory of Music and Compositional Algorithms 2.3. Computer Modeling of Musical Style and Machine Improvisation 2.4. Markov Models and Language Models for Music 2.5. Music Information Dynamics 2.6. Exercises Chapter 3: Communicating Musical Information 3.1. Music as Information Source 3.2. Lempel-Ziv Algorithm and Musical Style 3.3. Lossy Prediction Using Probabilistic Suffix Tree 3.4. Improved Suffix Search Using Factor Oracle Algorithm 3.5. Modeling of Musical Style 3.6. Estimating Music Information Dynamics Using Lempel-Ziv Complexity 3.7. Exercises Chapter 4: Understanding and (Re)Creating Sound 4.1. Introduction to Digital Signal Processing 4.2. Short-Time Fourier Analysis and Perfect Reconstruction (COLA) 4.3. Griffin-Lim Phase Reconstruction 4.4. Filters and Transforms 4.5. Voice as a Source-Filter Model 4.6. Information Rate in Spectral Analysis 4.7. Exercises Chapter 5: Generating and Listening to Audio Information 5.1. Concatenative and Recombinant Audio Synthesis 5.2. Audio Oracle 5.3. Audio Symbolization Using Music Information Dynamics 5.4. Accompaniment Using Query-Based Improvisation 5.5. Computational Aesthetics 5.6. Exercises Chapter 6: Artificial Musical Brains 6.1. Neural Network Models of Music 6.2. Viewing Neural Networks in a Musical Frame 6.3. Learning Audio Representations 6.4. Audio-Basis Using PCA and Matrix Factorization 6.5. Representation Learning with Auto-Encoder 6.6. Exercises Chapter 7: Representing Voices in Pitch and Time 7.1. Tokenization 7.2. Recurrent Neural Network for Music 7.3. Convolutional Neural Networks for Music and Audio 7.4. Pretrained Audio Neural Networks 7.5. Exercises Chapter 8: Noise Revisited: Brains that Imagine 8.1. Why study generative modeling? 8.2. Mathematical Definitions 8.3. Variational Methods: Autoencoder, Evidence Lower Bound 8.4. Generating Music and Sound with Variational Autoencoder 8.5. Discrete Neural Representation with Vector-Quantized VAE 8.6. Generative Adversarial Networks 8.7. Exercises Chapter 9: Paying (Musical) Attention 9.1. Introduction 9.2. Transformers and Memory Models in RNN 9.3. MIDI Transformers 9.4. Spectral Transformers 9.5. Attention, Memory, and Information Dynamics 9.6. Exercises Chapter 10: Last Noisy Thoughts, Summary, and Conclusion 10.1. Music Communication Revisited: Information Theory of VAE 10.2. The Big Picture: Deep Music Information Dynamics 10.3. The Future of AI in Music and Man-Machine Creative Interaction Appendix A: Introduction to Neural Network Frameworks: Keras, Tensorflow, Pytorch Appendix B: Summary of Programming Examples and Exercises Appendix C: Software Packages for Music and Audio Representation and Analysis Appendix D: Free Music and Audio-Editing Software Appendix E: Datasets Appendix F: Figure Attributions References Index