Data Storage Architectures and Technologies

Data is a core asset in the current development of information technology and needs to be stored efficiently and reliably to serve many important real-world applications such as the Internet, big data, artificial intelligence, and high-performance computing. Generations of researchers and practitioners have continued to innovate the design of storage systems to achieve the goals of high performance, ease of use, and high reliability. This textbook provides a thorough and comprehensive introduction to the field of data storage. With 14 chapters, the book not only covers the basics of storage devices, storage arrays, storage protocols, key-value stores, file systems, network storage architecture, distributed storage systems, storage reliability, storage security, and data protection, but also provides in-depth discussions on advanced topics such as storage maintenance, storage solutions, and storage technology trends and developments (e.g., in-storage computing, persistent memory system, blockchain storage, and in-network storage system). For each section, the authors have attempted to provide the latest current academic and industry research progress that will help readers deepen their understanding and application of basic data storage concepts. This textbook is ideal for storage courses targeting upper-level undergraduate or graduate students in computer science and related disciplines. It also serves as a valuable reference for technical professionals. Foreword 1 Foreword 2 Building Excellent Data Infrastructure by Mobilizing Storage Industry Professionals Preface Acknowledgments Summary Contents Chapter 1: Data Storage Background 1.1 The Importance of Data Storage 1.2 The Objectives of Data Storage 1.2.1 High Performance 1.2.2 High Usability 1.2.3 High Reliability 1.2.4 Other Objectives References Chapter 2: Storage Disks and Media 2.1 HDDs 2.1.1 Components and Structure 2.1.2 Performance 2.1.3 Firmware 2.1.3.1 Address Mapping Zones Remappings Skews 2.1.3.2 HDD Cache Write Buffer Read Cache and Prefetch 2.1.3.3 HDD Scheduling FCFS Algorithm SSTF Algorithm SCAN Algorithm LOOK Algorithm 2.2 SSDs 2.2.1 Flash Memory Cells and Structure 2.2.2 FTL 2.2.2.1 Address Mapping Page-Level Address Mapping Block-Level Address Mapping Hybrid Address Mapping 2.2.2.2 Garbage Collection 2.2.2.3 Wear Leveling 2.2.2.4 ECC 2.3 Main Memory 2.3.1 DRAM Components and Structure 2.3.2 DRAM Refresh 2.3.3 Memory Controllers 2.3.4 Nonvolatile Memory 2.3.4.1 PCM 2.3.4.2 RRAM 2.3.4.3 MRAM 2.4 Other Storage Media 2.4.1 Optical Storage 2.4.2 Tapes 2.5 Summary 2.6 Practice Questions References Chapter 3: Storage Arrays 3.1 Hardware Architecture 3.1.1 System Architecture 3.1.2 Controller Module 3.1.3 Interface Module 3.1.4 Disk Enclosure and Disk Module 3.1.5 Heat Dissipation Module 3.2 Software Architecture 3.2.1 RAID Subsystem 3.2.1.1 RAID Algorithm Principles 3.2.1.2 RAID Data Update 3.2.1.3 RAID Space Management 3.2.2 Cache Mirroring Subsystem 3.2.2.1 How Does the Cache Mirroring Subsystem Improve Storage Array Performance? 3.2.2.2 How Does the Cache Mirroring Subsystem Improve the Reliability of Storage Arrays? 3.2.2.3 New Trends in Cache Mirroring Subsystems 3.3 High Performance and Reliability Design 3.3.1 Application Scenarios 3.3.1.1 Database 3.3.1.2 Virtualization Platform 3.3.2 High-Reliability Subsystems for Redundant Failover 3.3.2.1 Fast Controller Failure Detection 3.3.2.2 Cache Mirroring Technology (Including Continuous Mirroring) 3.3.2.3 Fast Controller Failover 3.3.3 High-Performance Cluster Subsystems 3.3.3.1 Unbalanced Architecture 3.3.3.2 Balanced Architecture 3.3.4 Redirect-on-Write and Garbage Collection Technologies 3.4 Summary 3.5 Practice Questions References Chapter 4: Storage Protocols 4.1 SCSI Protocol 4.1.1 SCSI Protocol Overview 4.1.2 SCSI Service Model 4.1.3 SCSI Command Sets 4.1.4 SCSI Read/Write Process Analysis 4.2 SCSI Link Bearer Protocols 4.2.1 SAS Protocol 4.2.2 Fibre Channel Protocol 4.2.3 iSCSI Protocol 4.3 NVMe Protocol 4.3.1 NVMe Device Model 4.3.2 NVMe Queue Model 4.3.3 NVMe Command Set 4.3.4 NVMe over PCIe 4.4 NVMe over Fabrics 4.4.1 NVMe over RDMA 4.4.2 NVMe over TCP 4.4.3 NVMe over FC 4.5 Compute Express Link Protocol 4.5.1 CXL Overview 4.5.2 CXL Type 1 Devices 4.5.3 CXL Type 2 Devices 4.5.4 CXL Type 3 Devices 4.6 Summary 4.7 Practice Questions References Chapter 5: Key-Value Stores 5.1 Basic Operations 5.2 Key-Value Indexes 5.2.1 Hash Indexes 5.2.1.1 Hash Conflict 5.2.1.2 Extendable Hashing Table 5.2.1.3 Cuckoo Hashing 5.2.1.4 Using Hash Tables as Indexes 5.2.2 B+ Tree Indexes 5.2.3 LSM Tree Indexes 5.2.3.1 Bloom Filter 5.2.3.2 Comparison of the LSM Tree and B+ Tree 5.3 Data Layouts 5.3.1 In-Place Updated Data Layout 5.3.2 Log-Structured Data Layout 5.4 Crash Consistency 5.4.1 WALs 5.4.2 Shadow Pages 5.5 Summary 5.6 Practice Questions References Chapter 6: File Systems 6.1 Basic Operations 6.2 File System Implementation 6.2.1 A Simple File System 6.2.2 Namespace Management 6.2.3 Storage Management 6.2.3.1 Space Allocation Contiguous Allocation Linked Allocation Indexed Allocation 6.2.3.2 Free Space Management Management Using a Bitmap Management Using a Linked List Issues that May Occur During Free Block Management 6.3 File System Instance: ext2 6.3.1 Disk Data Structure 6.3.2 Directory Tree Structure 6.3.3 Data Block Addressing 6.4 Summary 6.5 Practice Questions References Chapter 7: Network Storage Architectures 7.1 DAS 7.2 NAS 7.2.1 Architecture Characteristics 7.2.2 Network-Based File Access Protocols 7.2.2.1 Protocol Type 7.2.2.2 I/O Operation Process 7.2.3 Application Scenarios 7.3 SAN 7.3.1 Architecture Characteristics 7.3.2 Core Components 7.3.2.1 Management Software 7.3.2.2 File System 7.3.3 Application Scenarios 7.3.4 Comparing NAS and SAN 7.4 Object Storage 7.4.1 Architecture Characteristics 7.4.2 Core Components 7.5 Parallel Storage 7.5.1 Architecture Characteristics 7.5.2 Key Technologies 7.6 P2P Storage 7.6.1 Architecture Characteristics 7.6.2 Key Technologies 7.6.2.1 Structured Overlay Network 7.6.2.2 Data Distribution Technology 7.6.2.3 Data Redundancy Technology 7.6.3 Application Scenarios 7.7 Cloud Storage 7.7.1 Architecture Characteristics 7.7.2 Application Scenarios 7.7.2.1 Public Cloud 7.7.2.2 Private Cloud 7.7.2.3 Hybrid Cloud 7.8 Storage Virtualization 7.8.1 Basic Concepts 7.8.2 Key Technologies 7.9 Software-Defined Storage (SDS) 7.9.1 Basic Concepts 7.9.2 Representative Systems 7.9.3 Key Challenges 7.10 Hyperconverged Infrastructure 7.10.1 Basic Concepts 7.10.2 Key Technologies 7.10.3 Representative Systems 7.10.4 Concept Comparison 7.11 Summary 7.12 Practice Questions References Chapter 8: Distributed Storage Systems 8.1 Typical Architecture of Distributed Storage Systems 8.2 Key Measurement Indicators of Distributed Storage Systems 8.2.1 Performance 8.2.2 Scalability 8.2.3 Consistency 8.2.4 Availability 8.3 Distributed Key-Value Store Systems 8.3.1 Typical Distributed Key-Value Store Systems 8.3.1.1 Dynamo System Interfaces and Semantics Replication Mechanism and Read/Write Process 8.3.1.2 RAMCloud System Interfaces and Semantics Replication Mechanism and Read/Write Process 8.3.2 Key Technologies of Distributed Key-Value Store Systems 8.3.2.1 Distributed Replication Technology 8.3.2.2 Distributed Secondary Index Technology 8.4 Distributed Object Storage Systems 8.4.1 Typical Distributed Object Storage Systems 8.4.1.1 Swift Cluster Architecture Data Organization Data Read/Write Process 8.4.1.2 Ceph RADOS Cluster Architecture Data Organization Data Read/Write Process 8.4.2 Key Technologies of Distributed Object Storage Systems 8.4.2.1 Data Distribution 8.4.2.2 Load Balancing 8.5 Distributed Block Storage Systems 8.5.1 Typical Distributed Block Storage Systems 8.5.1.1 Amazon EBS 8.5.1.2 Blizzard 8.5.2 Key Technologies of Distributed Block Storage Systems 8.5.2.1 Caching Optimization 8.5.2.2 Parallelism 8.6 Distributed File Systems 8.6.1 Typical Distributed File Systems 8.6.1.1 GFS Cluster Architecture Directory Tree Structure File Creation Process File Read/Write Process 8.6.1.2 InfiniFS Cluster Architecture Directory Tree Structure File Creation Process Data Read and Write Parsing 8.6.2 Key Technologies of Distributed File Systems 8.6.2.1 Directory Tree Management 8.6.2.2 Fast Path Resolution 8.7 Summary 8.8 Practice Questions References Chapter 9: Storage Reliability 9.1 Overview of Storage Reliability 9.1.1 Reliability Metrics and Calculation 9.1.2 Layered Reliability Design 9.1.3 Reliability vs. Availability 9.2 Disk Reliability 9.2.1 Analysis of Disk Error Characteristics 9.2.1.1 HDD Error Characteristics 9.2.1.2 SSD Error Characteristics 9.2.2 Disk Failure Warning and Monitoring 9.2.2.1 S.M.A.R.T. Attribute Values 9.2.2.2 ML-Based Disk Fault Prediction Algorithm 9.2.2.3 Bad Sector Detection and Repair 9.2.3 Environment-Aware Disk Reliability Design 9.2.3.1 Anti-corrosion 9.2.3.2 Cooling 9.2.3.3 Vibration-Resistant Design 9.3 Flash Media Reliability 9.3.1 Flash Media Error Sources 9.3.1.1 P/E Cycle 9.3.1.2 Retention Time 9.3.1.3 Program Interference 9.3.1.4 Read Disturb 9.3.2 Key Flash Reliability Technologies 9.3.2.1 Shadow Program Sequence 9.3.2.2 Refresh 9.3.2.3 Read-Retry 9.3.2.4 Voltage Optimization 9.4 Erasure Coding 9.4.1 Multi-copy 9.4.2 Erasure Coding 9.4.3 Typical Erasure Codes 9.4.3.1 EVENODD Code 9.4.3.2 RDP Code 9.4.3.3 X-Code 9.4.3.4 HV Code 9.4.3.5 Summary and Comparison 9.4.4 Development Trend of Erasure Code Technology 9.4.4.1 Construction of Erasure Codes with Optimized Repair Efficiency 9.4.4.2 Erasure Code Repair Algorithm 9.4.4.3 Proactive Repair 9.4.4.4 Redundancy Transitioning 9.4.4.5 Elastic Parameter Scaling 9.5 Distributed Storage System Reliability 9.5.1 Data Redundancy 9.5.2 Failure Recovery 9.5.3 Data Consistency Protocols 9.5.4 Load Balancing 9.6 Summary 9.7 Practice Questions References Chapter 10: Storage Security 10.1 Concept and Security System 10.2 System Security 10.2.1 Hardware Security 10.2.2 Container Security 10.2.3 System Resilience 10.3 Data Security 10.3.1 Data Encryption 10.3.1.1 Symmetric and Asymmetric Encryption Symmetric Encryption Asymmetric Encryption 10.3.1.2 File-Level Encryption Local Data Encryption Remote Data Encryption 10.3.1.3 Storage-Level Encryption 10.3.2 Data Integrity 10.3.2.1 Hashing MD5 SHA 10.3.2.2 Digital Signature 10.3.3 Permission Management 10.3.3.1 Access Control 10.3.3.2 Encrypted Search 10.3.4 Secure Destruction of Data 10.3.4.1 Local Data Destruction 10.3.4.2 Remote Data Destruction 10.3.5 Secure Data Computing 10.3.5.1 Trusted Execution Environment 10.3.5.2 Homomorphic Encryption 10.4 Security Management 10.4.1 System Access Control (Authentication Management) 10.4.1.1 Unified Management of Identity Authentication 10.4.1.2 Two-Factor Authentication 10.4.1.3 Account Security Management 10.4.2 User Identity and Access Management 10.4.3 Certificate and Key Management 10.4.3.1 Public Key Infrastructure Definition of Digital Certificate Why Do We Manage Digital Certificates? Storage Device Certificate Management 10.4.3.2 The Key Distribution Mechanism 10.4.4 Cybersecurity Management 10.4.4.1 Cybersecurity Framework 10.4.4.2 Host-Based Intrusion Detection 10.4.4.3 Trusted Boot and Remote Attestation 10.5 Summary 10.6 Practice Questions References Chapter 11: Data Protection 11.1 Data Protection Background 11.1.1 Data Protection Standards 11.1.2 The Features of Data Protection Technologies 11.2 Data Protection Technologies 11.2.1 Mirroring 11.2.2 Snapshots 11.2.3 Cloning 11.3 Data Protection Scenarios 11.3.1 Backup 11.3.1.1 The 3-2-1 System Architecture 11.3.1.2 Networking Architecture 11.3.1.3 Converged Architecture 11.3.2 Archiving 11.3.2.1 Application Scenarios 11.3.2.2 Technical Principles 11.3.3 DR 11.3.3.1 Active/Standby DR 11.3.3.2 Active-Active DR Active-Active SAN Architecture Active-Active NAS Architecture 11.3.3.3 Multi-data Center DR 11.4 Summary 11.5 Practice Questions References Chapter 12: Storage Maintenance 12.1 Overview 12.2 Preventive Maintenance 12.2.1 Disk Health Prediction 12.2.1.1 Data Collection 12.2.1.2 Analytics Platform 12.2.1.3 Result Evaluation 12.2.2 Capacity Trend Prediction 12.2.2.1 Data Collection 12.2.2.2 Analytics Platform 12.2.3 Performance Anomaly Detection 12.2.4 Performance Fluctuation Analysis 12.3 Corrective Maintenance 12.3.1 Proactive Troubleshooting 12.3.2 Upgrade 12.3.2.1 Online Upgrade in Batches 12.3.2.2 Simultaneous Online Upgrade 12.3.2.3 Large-Scale Node Upgrade Technology Component-Based Upgrade Gray Upgrade Parallel Upgrade in a Storage Pool 12.3.3 Capacity Expansion 12.3.3.1 Capacity Expansion Type Scale-Up Scale-Out 12.3.3.2 Capacity Expansion Process Capacity Expansion Preparation Capacity Expansion Implementation Post-expansion Check 12.4 Practice Questions References Chapter 13: Storage Solutions 13.1 e-Government Converged Storage Resource Pool Solution 13.1.1 Scenario Requirements 13.1.2 Converged Storage Resource Pool Solution 13.2 Disaster Recovery Solution for the Financial Industry 13.2.1 Scenario Requirements 13.2.2 DR Construction Requirements 13.2.3 Geo-redundant 3DC DR Solution 13.3 Storage Solution for the Healthcare Industry 13.3.1 Scenario Requirements 13.3.2 PACS Storage Solution 13.4 Storage Solution for the Education Industry 13.4.1 Scenario Requirements 13.4.2 High-Performance Data Analytics for Education and Scientific Research 13.5 Practice Questions Reference Chapter 14: Storage Technology Trends and Development 14.1 Flash Storage System 14.1.1 OC SSD 14.1.2 ZNS SSD 14.2 Near-Data Computing 14.2.1 Near-Storage Computing 14.2.1.1 Near-Disk Computing 14.2.1.2 Near-Memory Computing 14.2.2 Coupled Storage and Compute 14.3 Persistent Memory 14.3.1 File Systems 14.3.1.1 Consistency Guarantee Mechanisms 14.3.1.2 Cache Removal 14.3.1.3 User-Mode File Systems 14.3.2 Key-Value Store Systems 14.3.2.1 Persistent Index Structure 14.3.2.2 Space Management 14.3.3 Distributed Storage Systems 14.3.3.1 Software Stacks 14.3.3.2 Distributed Protocols 14.4 In-Network Storage 14.4.1 In-Network Data Coordination 14.4.2 In-Network Data Scheduling 14.4.3 In-Network Caching 14.5 Intelligent Storage 14.5.1 AI for Storage 14.5.1.1 Learned Index 14.5.1.2 Automatic Parameter Tuning 14.5.1.3 Heuristic Algorithm Optimization 14.5.2 Storage for AI 14.5.2.1 Data Loading 14.5.2.2 Data Preprocessing 14.5.2.3 Model Training 14.6 Edge Storage 14.6.1 Edge Storage Devices 14.6.2 Edge Storage I/O Stack 14.6.3 Edge Data Organization and Search 14.7 Blockchain Storage 14.7.1 Blockchain Storage System Overview 14.7.2 Blockchain Storage System Optimization 14.7.2.1 On-Blockchain Content Pruning 14.7.2.2 Blockchain Sharding 14.7.2.3 Blockchain Storage Structure Optimization 14.7.2.4 Off-Blockchain Storage Support 14.8 Disaggregated Data Center Architectures 14.8.1 Background 14.8.2 Architecture Features and Key Technologies 14.8.3 Future Trends 14.9 High-Density New Storage 14.9.1 Shingled Magnetic Recording 14.9.2 High-Density Optical Storage 14.9.3 DNA Storage 14.10 Summary 14.11 Practice Questions References