معرفی .NET برای آپاچی اسپارک: پردازش توزیع‌شده برای مجموعه‌داده‌های عظیم

Get started using Apache Spark via C# or F# and the .NET for Apache Spark bindings. This book is an introduction to both Apache Spark and the .NET bindings. Readers new to Apache Spark will get up to speed quickly using Spark for data processing tasks performed against large and very large datasets. You will learn how to combine your knowledge of .NET with Apache Spark to bring massive computing power to bear by distributed processing of extremely large datasets across multiple servers. This book covers how to get a local instance of Apache Spark running on your developer machine and shows you how to create your first .NET program that uses the Microsoft .NET bindings for Apache Spark. Techniques shown in the book allow you to use Apache Spark to distribute your data processing tasks over multiple compute nodes. You will learn to process data using both batch mode and streaming mode so you can make the right choice depending on whether you are processing an existing dataset or are working against new records in micro-batches as they arrive. The goal of the book is leave you comfortable in bringing the power of Apache Spark to your favorite .NET language. What You Will Learn Install and configure Spark .NET on Windows, Linux, and macOS Write Apache Spark programs in C# and F# using the .NET bindings Access and invoke the Apache Spark APIs from .NET with the same high performance as Python, Scala, and R Encapsulate functionality in user-defined functions Transform and aggregate large datasets Execute SQL queries against files through Apache Hive Distribute processing of large datasets across multiple servers Create your own batch, streaming, and machine learning programs Who This Book Is For .NET developers who want to perform big data processing without having to migrate to Python, Scala, or R; and Apache Spark developers who want to run natively on .NET and take advantage of the C# and F# ecosystems Table of Contents About the Author About the Technical Reviewer Introduction Part I: Getting Started Chapter 1: Understanding Apache Spark An Example The Core Use Cases Transform Your Data Analyze Your Data Machine Learning .NET for Apache Spark Feature Parity Summary Chapter 2: Setting Up Spark Choosing Your Software Versions Choosing a Version of Apache Spark Choosing a Java Version Configuring Apache Spark and .NET for Apache Spark on macOS Configuring Already Installed Java Installing Java Downloading and Configuring Apache Spark Test Overriding Default Config Configuring Apache Spark and .NET for Apache Spark on Windows Configuring Already Installed Java Installing Java Downloading and Configuring Apache Spark Winutils Test Overriding Default Config Configuring Apache Spark and .NET for Apache Spark on Linux (Ubuntu) Configuring Already Installed Java Installing Java Downloading and Configuring Apache Spark Testing the Install Overriding Default Config Dotnet Worker Configuration Troubleshooting Common Errors Unsupported Class File Major Version Exception Exiting Spark Cannot Run spark-shell Summary Chapter 3: Programming with .NET for Apache Spark First Program Microsoft.Spark NuGet Package SparkSession Executing the Program Executing on the Command Line Debugging the .NET Code in an IDE Further Use of the Debug Command Converting Existing Applications to .NET Converting a PySpark Example to .NET Converting Scala Examples to .NET Referencing Columns Using $ Datasets Summary Part II: The APIs Chapter 4: User-Defined Functions An Example Architecture Performance Pickling Apache Arrow User-Defined Aggregate Functions (UDAFs) Debugging User-Defined Functions Summary Chapter 5: The DataFrame API The RDD API vs. the DataFrame API Actions and Transformation The DataFrame API DataFrameReader CSV, Parquet, Orc vs. Load DataFrameReader Options Infer Schema vs. Manually Specified Schema Creating DataFrames CreateDataFrame Spark SQL The Range Method DataFrameWriter DataFrameWriter Mode PartitionBy Controlling Filenames Columns and Functions Summary Chapter 6: Spark SQL and Hive Tables What Is the SQL API Passing Data Between Contexts SparkSession Catalog Summary Chapter 7: Spark Machine Learning API Library Naming Implemented Objects Params Saving/Loading Objects Identifiable TF-IDF Summary Part III: Examples Chapter 8: Batch Mode Processing Imperfect Source Data The Source Data Files The Data Pipeline The C# Data Pipeline The F# Data Pipeline Summary Chapter 9: Structured Streaming Our Streaming Example Setting Up the Example The Streaming Application Shredding the JSON Document Creating a DataFrame Starting a Stream Aggregating the Data Viewing the Output Summary Chapter 10: Troubleshooting Logging Spark UI History Server The Jobs Tab The Stages Tab The SQL Tab Remaining Tabs Summary Chapter 11: Delta Lake ACID The Delta Log Reading Data Changing Data Appending Data Overwriting Data Changing Data Checkpoints History Vacuum Merge Schema Evolution Time Travel Example Delta Lake Applications Configuration CSharp FSharp Summary Appendix A: Running in the Cloud Databricks Approaches to Executing Code on Databricks Creating a Databricks Workspace Home Screen Workspace Explorer Data Clusters Jobs Models and Search Summary Appendix B: Implementing .NET for Apache Spark Code Setting Up Your Build Environment Adding a Method to an Existing Object Adding a New Object Adding a New Data Type to the Serializer/Deserializer Tips to Help Get Contributions Merged Formatting Line Length Comments Copyright Notice Extra Blank Lines Explicit Type Declarations Tests Summary Index