MANAGING CLOUD NATIVE DATA ON KUBERNETES : architecting cloud native data services using... open source technology

Is Kubernetes ready for stateful workloads? This open source system has become the primary platform for deploying and managing cloud native applications. But because it was originally designed for stateless workloads, working with data on Kubernetes has been challenging. If you want to avoid the inefficiencies and duplicative costs of having separate infrastructure for applications and data, this practical guide can help. Using Kubernetes as your platform, you’ll learn open source technologies that are designed and built for the cloud. Authors Jeff Carpenter and Patrick McFadin provide case studies to help you explore new use cases and avoid the pitfalls others have faced. You’ll get an insider’s view of what’s coming from innovators who are creating next-generation architectures and infrastructure. With this book, you will: Learn how to use basic Kubernetes resources to compose data infrastructure Automate the deployment and operations of data infrastructure on Kubernetes using tools like Helm and operators Evaluate and select data infrastructure technologies for use in your applications Integrate data infrastructure technologies into your overall stack Explore emerging technologies that will enhance your Kubernetes-based applications in the future Cover 1 Copyright 6 Table of Contents 7 Foreword 13 Preface 17 Why We Wrote This Book 18 Who Is This Book For? 18 How to Read This Book 19 Conventions Used in This Book 22 Using Code Examples 22 O’Reilly Online Learning 23 How to Contact Us 23 Acknowledgments 24 Chapter 1. Introduction to Cloud Native Data Infrastructure: Persistence, Streaming, and Batch Analytics 27 Infrastructure Types 28 What Is Cloud Native Data? 30 More Infrastructure, More Problems 32 Kubernetes Leading the Way 33 Managing Compute on Kubernetes 34 Managing Network on Kubernetes 35 Managing Storage on Kubernetes 35 Cloud Native Data Components 36 Looking Forward 37 Getting Ready for the Revolution 38 Adopt an SRE Mindset 38 Embrace Distributed Computing 40 Principles of Cloud Native Data Infrastructure 40 Summary 43 Chapter 2. Managing Data Storage on Kubernetes 45 Docker, Containers, and State 45 Managing State in Docker 47 Bind Mounts 47 Volumes 48 Tmpfs Mounts 49 Volume Drivers 50 Kubernetes Resources for Data Storage 52 Pods and Volumes 52 PersistentVolumes 59 PersistentVolumeClaims 63 StorageClasses 65 Kubernetes Storage Architecture 68 Flexvolume 68 Container Storage Interface 69 Container Attached Storage 71 Container Object Storage Interface 73 Summary 75 Chapter 3. Databases on Kubernetes the Hard Way 77 The Hard Way 78 Prerequisites for Running Data Infrastructure on Kubernetes 79 Running MySQL on Kubernetes 79 ReplicaSets 80 Deployments 82 Services 86 Accessing MySQL 89 Running Apache Cassandra on Kubernetes 91 StatefulSets 93 Accessing Cassandra 104 Summary 106 Chapter 4. Automating Database Deployment on Kubernetes with Helm 107 Deploying Applications with Helm Charts 108 Using Helm to Deploy MySQL 109 How Helm Works 113 Labels 115 ServiceAccounts 116 Secrets 116 ConfigMaps 117 Updating Helm Charts 119 Uninstalling Helm Charts 120 Using Helm to Deploy Apache Cassandra 120 Affinity and Anti-Affinity 122 Helm, CI/CD, and Operations 125 Summary 128 Chapter 5. Automating Database Management on Kubernetes with Operators 129 Extending the Kubernetes Control Plane 130 Extending Kubernetes Clients 131 Extending Kubernetes Control Plane Components 131 Extending Kubernetes Worker Node Components 132 The Operator Pattern 133 Controllers 133 Custom Resources 136 Operators 138 Managing MySQL in Kubernetes Using the Vitess Operator 140 Vitess Overview 140 PlanetScale Vitess Operator 143 A Growing Ecosystem of Operators 153 Choosing Operators 153 Building Operators 156 Summary 159 Chapter 6. Integrating Data Infrastructure in a Kubernetes Stack 161 K8ssandra: Production-Ready Cassandra on Kubernetes 161 K8ssandra Architecture 162 Installing the K8ssandra Operator 163 Creating a K8ssandraCluster 167 Managing Cassandra in Kubernetes with Cass Operator 169 Enabling Developer Productivity with Stargate APIs 173 Unified Monitoring Infrastructure with Prometheus and Grafana 176 Performing Repairs with Cassandra Reaper 180 Backing Up and Restoring Data with Cassandra Medusa 182 Creating a Backup 183 Restoring from Backup 184 Deploying Multicluster Applications in Kubernetes 185 Summary 191 Chapter 7. The Kubernetes Native Database 193 Why a Kubernetes Native Approach Is Needed 193 Hybrid Data Access at Scale with TiDB 195 TiDB Architecture 196 Deploying TiDB in Kubernetes 199 Serverless Cassandra with DataStax Astra DB 208 What to Look for in a Kubernetes Native Database 215 Basic Requirements 215 The Future of Kubernetes Native 217 Summary 220 Chapter 8. Streaming Data on Kubernetes 221 Introduction to Streaming 221 Types of Delivery 222 Delivery Guarantees 223 Feature Scope 224 The Role of Streaming in Kubernetes 225 Streaming on Kubernetes with Apache Pulsar 228 Preparing Your Environment 231 Securing Communications by Default with cert-manager 233 Using Helm to Deploy Apache Pulsar 237 Stream Analytics with Apache Flink 238 Deploying Apache Flink on Kubernetes 240 Summary 243 Chapter 9. Data Analytics on Kubernetes 245 Introduction to Analytics 246 Deploying Analytic Workloads in Kubernetes 247 Introduction to Apache Spark 250 Deploying Apache Spark in Kubernetes 252 Build Your Custom Container 254 Submit and Run Your Application 254 Kubernetes Operator for Apache Spark 256 Alternative Schedulers for Kubernetes 259 Apache YuniKorn 261 Volcano 263 Analytic Engines for Kubernetes 266 Dask 268 Ray 270 Summary 272 Chapter 10. Machine Learning and Other Emerging Use Cases 273 The Cloud Native AI/ML Stack 274 AI/ML Definitions 274 Defining an AI/ML Stack 276 Real-Time Model Serving with KServe 278 Full Lifecycle Feature Management with Feast 281 Vector Similarity Search with Milvus 284 Efficient Data Movement with Apache Arrow 287 Versioned Object Storage with lakeFS 290 Summary 294 Chapter 11. Migrating Data Workloads to Kubernetes 295 The Vision: Application-Aware Platforms 295 Charting Your Path to Success 297 People 298 Technology 302 Process 309 The Future of Cloud Native Data 314 Summary 318 Index 319 About the Authors 331 Colophon 331 Kubernetes has become the primary platform for deploying and managing cloud native applications. But because it was originally designed for stateless workloads, working with data on Kubernetes has been challenging. If you want to avoid the inefficiencies and duplicative costs of having separate infrastructure for applications and data, this practical guide can help. Using Kubernetes as your platform, you'll discover open source technologies that are designed and built for the cloud. Delve into case studies to avoid the pitfalls others have faced and explore new use cases. Get an insider's view of what's coming from the innovators who are creating next-generation architectures and infrastructure. And you'll learn how to: Manage different data use cases on Kubernetes Reduce costs and simplify application development Leverage data and infrastructure to create new use cases and business models Make data infrastructure choices that are cost-efficient, secure, scalable, and elastic And more