Data Science with . NET and Polyglot Notebooks: Programmer's Guide to Data Science Using ML. NET, OpenAI, and Semantic Kernel

Expand your skillset by learning how to perform data science, machine learning, and generative AI experiments in .NET Interactive notebooks using a variety of languages, including C#, F#, SQL, and PowerShell Key Features • Conduct a full range of data science experiments with clear explanations from start to finish • Learn key concepts in data analytics, machine learning, and AI and apply them to solve real-world problems • Access all of the code online as a notebook and interactive GitHub Codespace Book Description As the fields of data science, machine learning, and artificial intelligence rapidly evolve, .NET developers are eager to leverage their expertise to dive into these exciting domains but are often unsure of how to do so. Data Science in .NET with Polyglot Notebooks is the practical guide you need to seamlessly bring your .NET skills into the world of analytics and AI. With Microsoft’s .NET platform now robustly supporting machine learning and AI tasks, the introduction of tools such as .NET Interactive kernels and Polyglot Notebooks has opened up a world of possibilities for .NET developers. This book empowers you to harness the full potential of these cutting-edge technologies, guiding you through hands-on experiments that illustrate key concepts and principles. Through a series of interactive notebooks, you’ll not only master technical processes but also discover how to integrate these new skills into your current role or pivot to exciting opportunities in the data science field. By the end of the book, you’ll have acquired the necessary knowledge and confidence to apply cutting-edge data science techniques and deliver impactful solutions within the .NET ecosystem. Who is this book for? This book is for experienced C# or F# developers who want to transition into data science and machine learning while leveraging their .NET expertise. It’s ideal for those looking to learn ML.NET and Semantic kernel and extend their .NET skills to data science, machine learning, and Generative AI Workflows. What you will learn • Load, analyze, and transform data using DataFrames, data visualization, and descriptive statistics • Train machine learning models with ML.NET for classification and regression tasks • Customize ML.NET model training pipelines with AutoML, transforms, and model trainers • Apply best practices for deploying models and monitoring their performance • Connect to generative AI models using Polyglot Notebooks • Chain together complex AI tasks with AI orchestration, RAG, and Semantic Kernel • Create interactive online documentation with Mermaid charts and GitHub Codespaces Cover Title Page Copyright Dedication Contributors Table of Contents Preface Part 1: Data Analysis in Polyglot Notebooks Chapter 1: Data Science, Notebooks, and Kernels Exploring the field of data science The rise of big data Data analytics Machine learning Artificial intelligence Data science notebooks and Project Jupyter Extending notebooks with kernels Polyglot Notebooks and .NET Interactive Summary Further reading Chapter 2: Exploring Polyglot Notebooks Technical requirements Installing Polyglot Notebooks Creating your first notebook Executing notebook cells Adding code cells Working with variables The Variables view Markdown cells Declaring classes and methods Declaring methods Declaring classes Working with other languages Sharing variables between languages Exporting variables Troubleshooting notebook execution Resolving compiler errors Problems with notebook execution Diagnostic output for Polyglot Notebooks errors Issues and the Polyglot Notebooks repository Summary Further reading Chapter 3: Getting Data and Code into Your Notebooks Technical requirements Importing code and NuGet packages Importing code files Importing NuGet packages Importing project files Reading CSV data Understanding CSV data Reading CSV data into a DataFrame Specialized CSV loading scenarios Troubleshooting CSV loading errors Loading TSV and other delimited file formats Getting JSON data with PowerShell Building DataFrames from objects Connecting to databases with SQL Connecting to a SQL database Executing SQL from SQL kernels Sharing SQL results with other kernels Alternative ways of connecting to the Database Querying Kusto clusters with KQL Summary Further reading Chapter 4: Working with Tabular Data and DataFrames Technical requirements Understanding data cleaning and data wrangling Where unclean data comes from The impact of unclean data Data cleaning and data wrangling Working with DataFrames in C# Viewing and sampling data Rows Getting and setting cell values Iterating over rows Working with columns Columns Analyzing columns Removing columns Renaming columns Adding a new column Handling missing values Sorting, filtering, grouping, and merging data Sorting DataFrames Grouping and aggregating DataFrames Merging DataFrames Filtering DataFrames DataFrames in other languages Summary Further reading Chapter 5: Visualizing Data Technical requirements Understanding exploratory data analysis Data visualization’s role in exploratory data analysis Descriptive statistics for EDA Extracting insights with descriptive statistics Using DataFrame.Description to generate descriptive statistics Descriptive statistics with MathNet.Numerics Creating a box plot with ScottPlot Performing univariate analysis with Plotly.NET Plotly and Plotly.NET Box plots in Plotly.NET Violin plots with Plotly.NET Histograms with Plotly.NET Summary Further reading Chapter 6: Variable Correlations Technical requirements Performing multivariate analysis with Plotly.NET Loading data and dependencies Multivariate analysis with box and violin plots Plotting multiple values with scatter plots Adding color to a scatter plot 3D scatter plots with Plotly.NET Identifying variable correlations Calculating variable correlations Building feature correlation matrixes Summary Further reading Part 2: Machine Learning with Polyglot Notebooks and ML.NET Chapter 7: Classification Experiments with ML.NET AutoML Technical requirements Understanding machine learning Supervised learning Classification and regression Introducing ML.NET and AutoML Understanding AutoML AutoML and data pre-processing Creating training and testing datasets Training a classification model with ML.NET AutoML Evaluating binary classification models Evaluating our model Calculating feature importance Predicting values with binary classification models Summary Further reading Chapter 8: Regression Experiments with ML.NET AutoML Technical requirements Understanding regression Our regression task Regression as a numerical formula Our regression dataset Performing a regression experiment Understanding cross-validation Interpreting cross-validation results Evaluating regression metrics Predicting values for outliers Applying PFI to regression models Applying a regression model Summary Further reading Chapter 9: Beyond AutoML: Pipelines, Trainers, and Transforms Technical requirements Performing regression without AutoML Features and pipelines Creating an AutoML pipeline Controlling AutoML pipelines Customizing the Featurizer Customizing the model trainer selector Customizing hyperparameter tuning Understanding the search space Customizing the search space Customizing the hyperparameter tuner Scaling numeric columns Selecting regression algorithms Selecting binary classification algorithms Summary Further reading Chapter 10: Deploying Machine Learning Models Technical requirements Introducing our multi-class classification model Training our model Evaluating multi-class classification models Generating test predictions Exporting ML.NET models Hosting ML.NET models in ASP.NET web applications Configuring a PredictionEnginePool Using the PredictionEnginePool Understanding model performance, data drift, and MLOps Detecting model drift MLOps and updating models Surveying additional ML.NET capabilities ONNX and TensorFlow models in ML.NET Summary Further reading Part 3: Exploring Generative AI with Polyglot Notebooks Chapter 11: Generative AI in Polyglot Notebooks Technical requirements Understanding Generative AI Deploying generative AI models on Azure Creating an Azure OpenAI Service Deploying models on Azure OpenAI Service Getting access credentials for Azure OpenAI Connecting to an Azure OpenAI Service Chatting with a deployed model Customizing model behavior with prompt engineering Zero-shot, one-shot, and few-shot inferencing Using text embeddings Generating images with DALL-E Summary Further reading Chapter 12: AI Orchestration with Semantic Kernel Technical requirements Understanding RAG and AI orchestration Introducing Semantic Kernel Chatting with Semantic Kernel functions Building the Kernel Creating a prompt function Adding memory to Semantic Kernel Defining complex functions Creating functions from methods Accepting KernelFunction parameters Defining a memory function Calling multiple functions using plugins Examining FunctionResult objects Azure OpenAI content filtering Handling complex requests with planners Knowing where to go from here Summary Further reading Part 4: Polyglot Notebooks in the Enterprise Chapter 13: Enriching Documentation with Mermaid Diagrams Technical requirements Introducing Mermaid diagrams Communicating logic with flowcharts Communicating structure with class diagrams Communicating data with Entity Relationship Diagrams Communicating behavior with state diagrams Communicating flow with sequence diagrams Communicating workflow with Git graphs Summary Further reading Chapter 14: Extending Polyglot Notebooks Technical requirements Understanding default formatting behavior Default object formatting Default collection formatting Styling output with custom formatters Exploring magic commands Creating a Polyglot Notebook extension Working with parameters Invoking code on kernels Summary Further reading Chapter 15: Adopting and Deploying Polyglot Notebooks Technical requirements Integrating Polyglot Notebooks into your day job Enabling rapid experimentation Supporting AI and analytics workloads Assisting testing workloads Training new team members with Polyglot Notebooks Sharing Polyglot Notebooks with your team Integrating Polyglot Notebooks into Jupyter or JupyterLab Storing Notebooks in source control Deploying Polyglot Notebooks to GitHub Codespaces Configuring GitHub codespaces Creating a codespace on GitHub Advancing into machine learning and AI Adding data science to your day job Getting into data science Succeeding in data science Summary Further reading Index About PACKT Other Books You May Enjoy