Key Features* Set up real-time streaming and batch data intensive infrastructure using Spark and Python * Deliver insightful visualizations in a web app using Spark (PySpark) * Inject live data using Spark Streaming with real-time events Book DescriptionLooking for a cluster computing system that provides high-level APIs? Apache Spark is your answer―an open source, fast, and general purpose cluster computing system. Spark's multi-stage memory primitives provide performance up to 100 times faster than Hadoop, and it is also well-suited for machine learning algorithms. Are you a Python developer inclined to work with Spark engine? If so, this book will be your companion as you create data-intensive app using Spark as a processing engine, Python visualization libraries, and web frameworks such as Flask. To begin with, you will learn the most effective way to install the Python development environment powered by Spark, Blaze, and Bookeh. You will then find out how to connect with data stores such as MySQL, MongoDB, Cassandra, and Hadoop. You'll expand your skills throughout, getting familiarized with the various data sources (Github, Twitter, Meetup, and Blogs), their data structures, and solutions to effectively tackle complexities. You'll explore datasets using iPython Notebook and will discover how to optimize the data models and pipeline. Finally, you'll get to know how to create training datasets and train the machine learning models. By the end of the book, you will have created a real-time and insightful trend tracker data-intensive app with Spark. What you will learn* Create a Python development environment powered by Spark (PySpark), Blaze, and Bookeh * Build a real-time trend tracker data intensive app * Visualize the trends and insights gained from data using Bookeh * Generate insights from data using machine learning through Spark MLLIB * Juggle with data using Blaze * Create training data sets and train the Machine Learning models * Test the machine learning models on test datasets * Deploy the machine learning algorithms and models and scale it for real-time events About the Author**Amit Nandi** studied physics at the Free University of Brussels in Belgium, where he did his research on computer generated holograms. Computer generated holograms are the key components of an optical computer, which is powered by photons running at the speed of light. He then worked with the university Cray supercomputer, sending batch jobs of programs written in Fortran. This gave him a taste for computing, which kept growing. He has worked extensively on large business reengineering initiatives, using SAP as the main enabler. He focused for the last 15 years on start-ups in the data space, pioneering new areas of the information technology landscape. He is currently focusing on large-scale data-intensive applications as an enterprise architect, data engineer, and software developer. He understands and speaks seven human languages. Although Python is his computer language of choice, he aims to be able to write fluently in seven computer languages too. Table of Contents1. Setting Up a Spark Virtual Environment 2. Building Batch and Streaming Apps with Spark 3. Juggling Data with Spark 4. Learning from Data Using Spark 5. Streaming Live Data with Spark 6. Visualizing Insights and Trends Key Features Set up real-time streaming and batch data intensive infrastructure using Spark and Python Deliver insightful visualizations in a web app using Spark (PySpark) Inject live data using Spark Streaming with real-time events Book Description Looking for a cluster computing system that provides high-level APIs? Apache Spark is your answer―an open source, fast, and general purpose cluster computing system. Spark's multi-stage memory primitives provide performance up to 100 times faster than Hadoop, and it is also well-suited for machine learning algorithms. Are you a Python developer inclined to work with Spark engine? If so, this book will be your companion as you create data-intensive app using Spark as a processing engine, Python visualization libraries, and web frameworks such as Flask. To begin with, you will learn the most effective way to install the Python development environment powered by Spark, Blaze, and Bookeh. You will then find out how to connect with data stores such as MySQL, MongoDB, Cassandra, and Hadoop. You'll expand your skills throughout, getting familiarized with the various data sources (Github, Twitter, Meetup, and Blogs), their data structures, and solutions to effectively tackle complexities. You'll explore datasets using iPython Notebook and will discover how to optimize the data models and pipeline. Finally, you'll get to know how to create training datasets and train the machine learning models. By the end of the book, you will have created a real-time and insightful trend tracker data-intensive app with Spark. What you will learn Create a Python development environment powered by Spark (PySpark), Blaze, and Bookeh Build a real-time trend tracker data intensive app Visualize the trends and insights gained from data using Bookeh Generate insights from data using machine learning through Spark MLLIB Juggle with data using Blaze Create training data sets and train the Machine Learning models Test the machine learning models on test datasets Deploy the machine learning algorithms and models and scale it for real-time events About the Author Amit Nandi studied physics at the Free University of Brussels in Belgium, where he did his research on computer generated holograms. Computer generated holograms are the key components of an optical computer, which is powered by photons running at the speed of light. He then worked with the university Cray supercomputer, sending batch jobs of programs written in Fortran. This gave him a taste for computing, which kept growing. He has worked extensively on large business reengineering initiatives, using SAP as the main enabler. He focused for the last 15 years on start-ups in the data space, pioneering new areas of the information technology landscape. He is currently focusing on large-scale data-intensive applications as an enterprise architect, data engineer, and software developer. He understands and speaks seven human languages. Although Python is his computer language of choice, he aims to be able to write fluently in seven computer languages too. Table of Contents Setting Up a Spark Virtual Environment Building Batch and Streaming Apps with Spark Juggling Data with Spark Learning from Data Using Spark Streaming Live Data with Spark Visualizing Insights and Trends
A concise guide to implementing Spark Big Data analytics for Python developers, and building a real-time and insightful trend tracker data intensive app
About This Book
- Set up real-time streaming and batch data intensive infrastructure using Spark and Python
- Deliver insightful visualizations in a web app using Spark (PySpark)
- Inject live data using Spark Streaming with real-time events
Who This Book Is For
This book is for data scientists and software developers with a focus on Python who want to work with the Spark engine, and it will also benefit Enterprise Architects. All you need to have is a good background of Python and an inclination to work with Spark.
What You Will Learn
- Create a Python development environment powered by Spark (PySpark), Blaze, and Bookeh
- Build a real-time trend tracker data intensive app
- Visualize the trends and insights gained from data using Bookeh
- Generate insights from data using machine learning through Spark MLLIB
- Juggle with data using Blaze
- Create training data sets and train the Machine Learning models
- Test the machine learning models on test datasets
- Deploy the machine learning algorithms and models and scale it for real-time events
In Detail
Looking for a cluster computing system that provides high-level APIs? Apache Spark is your answer—an open source, fast, and general purpose cluster computing system. Spark's multi-stage memory primitives provide performance up to 100 times faster than Hadoop, and it is also well-suited for machine learning algorithms.
Are you a Python developer inclined to work with Spark engine? If so, this book will be your companion as you create data-intensive app using Spark as a processing engine, Python visualization libraries, and web frameworks such as Flask.
To begin with, you will learn the most effective way to install the Python development environment powered by Spark, Blaze, and Bookeh. You will then find out how to connect with data stores such as MySQL, MongoDB, Cassandra, and Hadoop.
You'll expand your skills throughout, getting familiarized with the various data sources (Github, Twitter, Meetup, and Blogs), their data structures, and solutions to effectively tackle complexities. You'll explore datasets using iPython Notebook and will discover how to optimize the data models and pipeline. Finally, you'll get to know how to create training datasets and train the machine learning models.
By the end of the book, you will have created a real-time and insightful trend tracker data-intensive app with Spark.
Style and approach
This is a comprehensive guide packed with easy-to-follow examples that will take your skills to the next level and will get you up and running with Spark.
Annotation A concise guide to implementing Spark Big Data analytics for Python developers, and building a real-time and insightful trend tracker data intensive appAbout This Book Set up real-time streaming and batch data intensive infrastructure using Spark and Python Deliver insightful visualizations in a web app using Spark (PySpark) Inject live data using Spark Streaming with real-time eventsWho This Book Is ForThis book is for data scientists and software developers with a focus on Python who want to work with the Spark engine, and it will also benefit Enterprise Architects. All you need to have is a good background of Python and an inclination to work with Spark. What You Will Learn Create a Python development environment powered by Spark (PySpark), Blaze, and Bookeh Build a real-time trend tracker data intensive app Visualize the trends and insights gained from data using Bookeh Generate insights from data using machine learning through Spark MLLIB Juggle with data using Blaze Create training data sets and train the Machine Learning models Test the machine learning models on test datasets Deploy the machine learning algorithms and models and scale it for real-time eventsIn DetailLooking for a cluster computing system that provides high-level APIs? Apache Spark is your answeran open source, fast, and general purpose cluster computing system. Spark's multi-stage memory primitives provide performance up to 100 times faster than Hadoop, and it is also well-suited for machine learning algorithms. Are you a Python developer inclined to work with Spark engine? If so, this book will be your companion as you create data-intensive app using Spark as a processing engine, Python visualization libraries, and web frameworks such as Flask. To begin with, you will learn the most effective way to install the Python development environment powered by Spark, Blaze, and Bookeh. You will then find out how to connect with data stores such as MySQL, MongoDB, Cassandra, and Hadoop. You'll expand your skills throughout, getting familiarized with the various data sources (Github, Twitter, Meetup, and Blogs), their data structures, and solutions to effectively tackle complexities. You'll explore datasets using iPython Notebook and will discover how to optimize the data models and pipeline. Finally, you'll get to know how to create training datasets and train the machine learning models. By the end of the book, you will have created a real-time and insightful trend tracker data-intensive app with Spark. Style and approach This is a comprehensive guide packed with easy-to-follow examples that will take your skills to the next level and will get you up and running with Spark About This BookSet up real-time streaming and batch data intensive infrastructure using Spark and PythonDeliver insightful visualizations in a web app using Spark (PySpark)Inject live data using Spark Streaming with real-time eventsWho This Book Is ForThis book is for data scientists and software developers with a focus on Python who want to work with the Spark engine, and it will also benefit Enterprise architects. All you need to have is a good background in Python and an inclination to work with Spark. What You Will LearnCreate a Python development environment powered by Spark (PySpark), Blaze, and BokehBuild a real-time trend tracker data-intensive appVisualize the trends and insights gained from data using BokehGenerate insights from data using machine learning through Spark MLLIBJuggle data using BlazeCreate training data sets and train Machine Learning modelsTest machine learning models on test datasetsDeploy machine learning algorithms and models and scale them for real-time eventsIn DetailLooking for a cluster computing system that provides high-level APIs? Apache Spark is your answer―an open source, fast, and general purpose cluster computing system. Are you a Python developer inclined to work with the Spark engine? If so, this book will be your companion as you create a data-intensive app using Spark as a processing engine, Python visualization libraries, and web frameworks such as Flask. To begin with, you will learn the most effective way to install the Python development environment powered by Spark, Blaze, and Bokeh. You will then find out how to connect with data stores such as MySQL, MongoDB, Cassandra, and Hadoop. You'll expand your skills throughout, becoming familiar with the various data sources (Github, Twitter, Meetup, and blogs), their data structures, and solutions to effectively tackle complexities. By the end of the book, you will have created a real-time and insightful trend tracker data-intensive app with Spark.
