Machine Learning with PySpark : With Natural Language Processing and Recommender Systems

Master the new features in PySpark 3.1 to develop data-driven, intelligent applications. This updated edition covers topics ranging from building scalable machine learning models, to natural language processing, to recommender systems. Machine Learning with PySpark, Second Edition begins with the fundamentals of Apache Spark, including the latest updates to the framework. Next, you will learn the full spectrum of traditional machine learning algorithm implementations, along with natural language processing and recommender systems. You’ll gain familiarity with the critical process of selecting machine learning algorithms, data ingestion, and data processing to solve business problems. You’ll see a demonstration of how to build supervised machine learning models such as linear regression, logistic regression, decision trees, and random forests. You’ll also learn how to automate the steps using Spark pipelines, followed by unsupervised models such as K-means and hierarchical clustering. A section on Natural Language Processing (NLP) covers text processing, text mining, and embeddings for classification. This new edition also introduces Koalas in Spark and how to automate data workflow using Airflow and PySpark’s latest ML library. After completing this book, you will understand how to use PySpark’s machine learning library to build and train various machine learning models, along with related components such as data ingestion, processing and visualization to develop data-driven intelligent applications What you will learn: Build a spectrum of supervised and unsupervised machine learning algorithms Use PySpark's machine learning library to implement machine learning and recommender systems Leverage the new features in PySpark’s machine learning library Understand data processing using Koalas in Spark Handle issues around feature engineering, class balance, bias andvariance, and cross validation to build optimally fit models Who This Book Is For Data science and machine learning professionals. Table of Contents About the Author About the Technical Reviewer Acknowledgments Foreword Introduction Chapter 1: Introduction to Spark Data Generation Before the 1990s The Internet and Social Media Era The Machine Data Era Spark Setting Up the Environment Downloading Spark Installing Spark Docker Databricks Spin a New Cluster Create a Notebook Conclusion Chapter 2: Manage Data with PySpark Load and Read Data Data Filtering Using filter Data Filtering Using where Pandas UDF Drop Duplicate Values Writing Data CSV Parquet Data Handling Using Koalas Conclusion Chapter 3: Introduction to Machine Learning Rise in Data Increased Computational Efficiency Improved ML Algorithms Availability of Data Scientists Supervised Machine Learning Unsupervised Machine Learning Semi-supervised Learning Reinforcement Learning Industrial Application and Challenges Retail Healthcare Finance Travel and Hospitality Media and Marketing Manufacturing and Automobile Social Media Others Conclusion Chapter 4: Linear Regression Variables Theory Interpretation Evaluation Code Conclusion Chapter 5: Logistic Regression Probability Using Linear Regression Using Logit Interpretation (Coefficients) Dummy Variables Model Evaluation True Positives True Negatives False Positives False Negatives Accuracy Recall Precision F1 Score Probability Cut-Off/Threshold ROC Curve Logistic Regression Code Data Info Confusion Matrix Accuracy Recall Precision Conclusion Chapter 6: Random Forests Using PySpark Decision Tree Entropy Information Gain Random Forests Code Conclusion Chapter 7: Clustering in PySpark Applications K-Means Deciding on the Number of Clusters (K) Elbow Method Hierarchical Clustering Agglomerative Clustering Code Data Info Conclusion Chapter 8: Recommender Systems Recommendations Popularity-Based RS Content-Based RS User Profile Euclidean Distance Cosine Similarity Collaborative Filtering–Based RS User Item Matrix Explicit Feedback Implicit Feedback Nearest Neighbors–Based CF Missing Values Latent Factor–Based CF Hybrid Recommender Systems Code Data Info Conclusion Chapter 9: Natural Language Processing Steps Involved in NLP Corpus Tokenize Stopword Removal Bag of Words CountVectorizer TF-IDF Text Classification Using Machine Learning Sequence Embeddings Embeddings Conclusion Index Master The New Features In Pyspark 3.1 To Develop Data-driven, Intelligent Applications. This Updated Edition Covers Topics Ranging From Building Scalable Machine Learning Models, To Natural Language Processing, To Recommender Systems. Machine Learning With Pyspark, Second Edition Begins With The Fundamentals Of Apache Spark, Including The Latest Updates To The Framework. Next, You Will Learn The Full Spectrum Of Traditional Machine Learning Algorithm Implementations, Along With Natural Language Processing And Recommender Systems. You’ll Gain Familiarity With The Critical Process Of Selecting Machine Learning Algorithms, Data Ingestion, And Data Processing To Solve Business Problems. You’ll See A Demonstration Of How To Build Supervised Machine Learning Models Such As Linear Regression, Logistic Regression, Decision Trees, And Random Forests. You’ll Also Learn How To Automate The Steps Using Spark Pipelines, Followed By Unsupervised Models Such As K-means And Hierarchical Clustering. A Section On Natural Language Processing (nlp) Covers Text Processing, Text Mining, And Embeddings For Classification. This New Edition Also Introduces Koalas In Spark And How To Automate Data Workflow Using Airflow And Pyspark’s Latest Ml Library. After Completing This Book, You Will Understand How To Use Pyspark’s Machine Learning Library To Build And Train Various Machine Learning Models, Along With Related Components Such As Data Ingestion, Processing And Visualization To Develop Data-driven Intelligent Applications What You Will Learn: Build A Spectrum Of Supervised And Unsupervised Machine Learning Algorithms Use Of Pyspark Machine Learning Library For Implementation Of Machine Learning And Recommender Systems Leverage The New Features In Pyspark’s Machine Learning Library Understand Data Processing Using Koalas In Spark Handle Issues Around Feature Engineering, Class Balance, Bias And Variance, Cross Validation For Building Optimal Fit Model Who This Book Is For Data Science And Machine Learning Professionals. "Master the new features in PySpark 3.1 to develop data-driven, intelligent applications. This updated edition covers topics ranging from building scalable machine learning models, to natural language processing, to recommender systems. Machine Learning with PySpark, Second Edition begins with the fundamentals of Apache Spark, including the latest updates to the framework. Next, you will learn the full spectrum of traditional machine learning algorithm implementations, along with natural language processing and recommender systems. You'll gain familiarity with the critical process of selecting machine learning algorithms, data ingestion, and data processing to solve business problems. You'll see a demonstration of how to build supervised machine learning models such as linear regression, logistic regression, decision trees, and random forests. You'll also learn how to automate the steps using Spark pipelines, followed by unsupervised models such as K-means and hierarchical clustering. A section on Natural Language Processing (NLP) covers text processing, text mining, and embeddings for classification. This new edition also introduces Koalas in Spark and how to automate data workflow using Airflow and PySpark's latest ML library. After completing this book, you will understand how to use PySpark's machine learning library to build and train various machine learning models, along with related components such as data ingestion, processing and visualization to develop data-driven intelligent applications What you will learn: Build a spectrum of supervised and unsupervised machine learning algorithms Use PySpark's machine learning library to implement machine learning and recommender systems Leverage the new features in PySpark's machine learning library Understand data processing using Koalas in Spark Handle issues around feature engineering, class balance, bias and variance, and cross validation to build optimally fit models." --Amazon.com