Neuromorphic Computing Principles and Organization

This book focuses on neuromorphic computing principles and organization and how to build fault-tolerant scalable hardware for large and medium scale spiking neural networks with learning capabilities. In addition, the book describes in a comprehensive way the organization and how to design a spike-based neuromorphic system to perform network of spiking neurons communication, computing, and adaptive learning for emerging AI applications. The book begins with an overview of neuromorphic computing systems and explores the fundamental concepts of artificial neural networks. Next, we discuss artificial neurons and how they have evolved in their representation of biological neuronal dynamics. Afterward, we discuss implementing these neural networks in neuron models, storage technologies, inter-neuron communication networks, learning, and various design approaches. Then, comes the fundamental design principle to build an efficient neuromorphic system in hardware. The challenges that need to be solved toward building a spiking neural network architecture with many synapses are discussed. Learning in neuromorphic computing systems and the major emerging memory technologies that promise neuromorphic computing are then given. A particular chapter of this book is dedicated to the circuits and architectures used for communication in neuromorphic systems. In particular, the Network-on-Chip fabric is introduced for receiving and transmitting spikes following the Address Event Representation (AER) protocol and the memory accessing method. In addition, the interconnect design principle is covered to help understand the overall concept of on-chip and off-chip communication. Advanced on-chip interconnect technologies, including si-photonic three-dimensional interconnects and fault-tolerant routing algorithms, are also given. The book also covers the main threats of reliability and discusses several recovery methods for multicore neuromorphic systems. This is important for reliable processing in several embedded neuromorphic applications. A reconfigurable design approach that supports multiple target applications via dynamic reconfigurability, network topology independence, and network expandability is also described in the subsequent chapters. The book ends with a case study about a real hardware-software design of a reliable three-dimensional digital neuromorphic processor geared explicitly toward the 3D-ICs biological brain’s three-dimensional structure. The platform enables high integration density and slight spike delay of spiking networks and features a scalable design. We present methods for fault detection and recovery in a neuromorphic system as well. Neuromorphic Computing Principles and Organization is an excellent resource for researchers, scientists, graduate students, and hardware-software engineers dealing with the ever-increasing demands on fault-tolerance, scalability, and low power consumption. It is also an excellent resource for teaching advanced undergraduate and graduate students about the fundamentals concepts, organization, and actual hardware-software design of reliable neuromorphic systems with learning and fault-tolerance capabilities. Preface Acknowledgments Contents Acronyms 1 Introduction to Neuromorphic Computing Systems 1.1 Introduction 1.2 Design Challenges 1.3 Neural Networks 1.3.1 Artificial Neural Networks 1.3.2 Spiking Neural Networks 1.4 Learning in Spiking Neural Networks 1.5 Synapse Memory Technologies 1.6 Neurons Communication Network 1.7 Neuromorphic System Design Domains 1.8 Chapter Summary References 2 Neuromorphic System Design Fundamentals 2.1 Introduction 2.1.1 Spiking Neural Networks 2.1.2 Neural Coding Schemes 2.1.2.1 Rate Coding 2.1.2.2 Temporal Coding 2.2 Spiking Neuron Models 2.2.1 Hodgkin-Huxley Model 2.2.2 Izhikevich Model 2.2.3 Leaky Integrate and Fire Model 2.3 Learning Algorithms 2.3.1 Supervised Learning 2.3.2 Unsupervised Learning 2.3.2.1 Spike Timing Dependent Plasticity (STDP) 2.3.2.2 Spike Driven Synaptic Plasticity (SDSP) 2.4 Synapse Memory 2.4.1 SRAM 2.4.2 eDRAM 2.4.3 Memristor 2.5 Inter-Neuron Communication Schemes 2.5.1 AER—Address Event Representation 2.6 Neuromorphic Spike Routing 2.7 Chapter Summary References 3 Learning in Neuromorphic Systems 3.1 Learning Methods 3.2 Conversion from ANN to SNN 3.2.1 Converted SNNs 3.2.2 Challenges of ANN Conversion 3.3 Supervised Learning 3.3.1 Tempotron 3.3.2 ReSuMe 3.3.3 SpikeProp Algorithm 3.3.4 Approximate Derivative Method (ADM) 3.4 Unsupervised Learning 3.4.1 Pair-Based STDP Learning Rule 3.4.2 Triplet STDP Learning Rule 3.4.3 Reward-Modulated STDP Learning 3.4.4 Other Variants of STDP Learning Rule 3.5 Chapter Summary References 4 Emerging Memory Devices for Neuromorphic Systems 4.1 Introduction 4.2 Memory Technology 4.2.1 SRAM 4.2.2 eDRAM 4.2.3 STT-RAM 4.2.4 RRAM and Resistive Crossbar 4.2.5 Phase Change Memory 4.2.6 Other Memory Technologies 4.3 Memory Organization 4.4 Memory for Neuromorphic Systems 4.4.1 Neuron State Memory 4.4.2 Synapse Memory 4.4.2.1 SRAM Synapse Memory 4.4.2.2 Non-volatile Synapse Memory 4.4.2.3 NVM In-memory Computing 4.5 Dynamic NVM Synapse 4.5.1 Learning Related NVM 4.5.2 Conductance Drift in NVM 4.6 Chapter Summary References 5 Communication Networks for Neuromorphic Systems 5.1 Introduction 5.2 Neural Communication 5.3 Interconnect for Inter-Neural Communication 5.3.1 SpiNNaker 5.3.2 TrueNorth 5.3.3 Loihi 5.4 Interconnect Design Principles 5.4.1 OSI Model for Network-on-Chip 5.4.1.1 Application and Presentation Layers 5.4.1.2 Session and Transport Layers 5.4.1.3 Network Layer 5.4.1.4 Link and Data Layers 5.4.2 Network Topologies 5.4.2.1 Major Types of Topologies 5.4.2.2 2D Mesh Topology 5.4.2.3 3D Mesh Topology 5.4.3 Application Mapping 5.4.4 Communication Architecture 5.4.4.1 Switching Technique 5.4.4.2 Packet Routing 5.4.4.3 Flow Control 5.4.4.4 Quality of Service 5.4.4.5 Router Design 5.4.4.6 Link Design 5.5 Advanced Interconnects Multicore Neuromorphic Systems 5.5.1 Three Dimensional On-chip Interconnect 5.5.1.1 3D-NoC Versus 2D-NoC 5.5.1.2 Routing Algorithms 5.5.1.3 3D-NoC Router Architecture Design 5.5.1.4 Input-Port Module Design 5.5.1.5 Switch Allocator Design 5.5.1.6 Stall-Go Flow Control Architecture 5.5.1.7 Matrix-Arbiter Scheduling Architecture 5.5.1.8 Crossbar Design 5.5.1.9 Network Interface Architecture 5.5.2 Photonic On-chip Interconnect for High-Bandwidth Multicore SoCs 5.5.2.1 Photonic Communication Building Blocks 5.5.2.2 Design Challenges 5.5.2.3 Fault Models 5.5.2.4 Fault-Tolerant Photonic Network-on-Chip 5.5.3 Network Interface 5.6 Chapter Summary References 6 Fault-Tolerant Neuromorphic System Design 6.1 Introduction 6.1.1 Measure of Fault Tolerance 6.1.2 Type of Faults and Behavior 6.1.3 Impact of Faults on Neuromorphic System 6.2 Conventional Computing System Fault Tolerance 6.2.1 Hardware Approach 6.2.2 Information Redundancy 6.2.2.1 Parity Code 6.2.2.2 Hamming Code 6.2.3 Software Approach 6.3 Fault-Tolerance for Neuromorphic Computing 6.3.1 Memory Protection 6.3.2 Communication Protection 6.3.3 Computation Protection 6.3.4 SNN Mapping for Tolerating Errors 6.3.5 Fault-Tolerant Remapping for Neuromorphic Computing 6.3.5.1 Problem Formulation 6.3.5.2 Max-Flow Min-Cut Based Algorithm 6.3.5.3 Evaluation 6.4 Chapter Summary References 7 Reconfigurable Neuromorphic Computing System 7.1 Introduction 7.2 Fault-Tolerant Neural Networks 7.2.1 Learning-Based Approach 7.2.2 Architecture-Based Approach 7.2.3 Hybrid-Based Approach 7.3 Inter-Neuron Communication Network 7.4 Reconfigurable Neuromorphic System Building Blocks 7.4.1 Spiking Neuron Processing Core 7.4.1.1 LIF Neuron 7.4.1.2 Crossbar 7.4.1.3 Learning Algorithm 7.4.2 Network Interface 7.4.3 Fault-Tolerant Multicast 3D Router 7.5 Fault-Tolerant Spike Routing Algorithm 7.5.1 Shortest Path K-means Multicast Spike Routing Algorithm 7.5.2 Fault-Tolerant K-means Multicast Spike Routing Algorithm 7.6 Mapping 7.7 Complexity Analysis 7.8 Chapter Summary References 8 Case Study: Real Hardware-Software Design of 3D-NoC-Based Neuromorphic System 8.1 Introduction 8.2 R-NASH System 8.3 R-NASH Hardware 8.3.1 R-NASH Hardware Building Blocks 8.3.2 Spiking Neural Processing Core (SNPC) 8.3.3 Network Interface 8.3.4 Crossbar 8.3.4.1 LIF Neuron 8.3.5 Controlling 8.3.6 Inter-Neural Interconnect 8.4 R-NASH Learning 8.4.1 Off-chip Learning 8.4.2 Online Learning with STDP 8.5 R-NASH Initial Mapping 8.5.1 Genetic Algorithm 8.5.1.1 Initialization 8.5.2 Selection 8.5.3 Crossover 8.5.4 Mutation 8.5.4.1 Finalization 8.5.5 Genetic Algorithm for Neurons Mapping on R-NASH Hardware 8.6 R-NASH Run-Time Maintenance 8.6.1 Data Integrity Protection 8.6.2 Communication Protection 8.6.3 Fault-Tolerant Neurons Mapping Scheme 8.7 R-NASH Evaluation Results 8.7.1 Initial Mapping Evaluation 8.7.1.1 Mapping over Different 3D-NoC Sizes 8.7.1.2 Mapping over Different Node Sizes 8.7.1.3 Comparison Between 3D and 2D in Initial Mapping 8.7.2 Fault-Tolerant Mapping 8.7.3 Hardware Complexity 8.7.4 System Validation 8.7.4.1 Offline Feed-Forward Network 8.7.5 Unsupervised STDP 8.8 Chapter Summary References 9 Survey of Neuromorphic Systems 9.1 Introduction 9.2 Software Emulation Approach 9.2.1 SpiNNaker 9.2.1.1 Model 9.2.1.2 SpiNNaker Architecture 9.2.1.3 Other Modules 9.2.1.4 Communication in SpiNNaker 9.2.1.5 SpiNNaker Software Platform 9.3 Digital Hardware Design Approach 9.3.1 IBM TrueNorth 9.3.1.1 TrueNorth Neurosynaptic Architecture 9.3.1.2 Interconnect 9.3.1.3 TrueNorth Software 9.3.2 Intel Loihi 9.4 Analog and Mixed-Signal Hardware Approach 9.4.1 NeuroGrid 9.4.1.1 Neurogrid Software 9.4.1.2 Neurogrid Hardware 9.4.1.3 Neurogrid Communication 9.4.1.4 Transmitter and Receiver 9.5 Chapter Summary References Index