Practical deep learning for cloud, mobile, and edge : real-world AI and computer-vision projects using Python, Keras, and TensorFlow

Whether you're a software engineer aspiring to enter the world of deep learning, a veteran data scientist, or a hobbyist with a simple dream of making the next viral AI app, you might have wondered where to begin. This step-by-step guide teaches you how to build practical deep learning applications for the cloud, mobile, browsers, and edge devices using a hands-on approach. Relying on years of industry experience transforming deep learning research into award-winning applications, Anirudh Koul, Siddha Ganju, and Meher Kasam guide you through the process of converting an idea into something that people in the real world can use. * Train, tune, and deploy computer vision models with Keras, TensorFlow, Core ML, and TensorFlow Lite. * Develop AI for a range of devices including Raspberry Pi, Jetson Nano, and Google Coral. * Explore fun projects, from Silicon Valley's Not Hotdog app to 40+ industry case studies. * Simulate an autonomous car in a video game environment and build a miniature version with reinforcement learning. * Use transfer learning to train models in minutes. * Discover 50+ practical tips for maximizing model accuracy and speed, debugging, and scaling to millions of users. **List of Chapters**1. Exploring the Landscape of Artificial Intelligence 2. What's in the Picture: Image Classification with Keras 3. Cats Versus Dogs: Transfer Learning in 30 Lines with Keras 4. Building a Reverse Image Search Engine: Understanding Embeddings 5. From Novice to Master Predictor: Maximizing Convolutional Neural Network Accuracy 6. Maximizing Speed and Performance of TensorFlow: A Handy Checklist 7. Practical Tools, Tips, and Tricks 8. Cloud APIs for Computer Vision: Up and Running in 15 Minutes 9. Scalable Inference Serving on Cloud with TensorFlow Serving and KubeFlow 10. AI in the Browser with TensorFlow.js and ml5.js 11. Real-Time Object Classification on iOS with Core ML 12. Not Hotdog on iOS with Core ML and Create ML 13. Shazam for Food: Developing Android Apps with TensorFlow Lite and ML Kit 14. Building the Purrfect Cat Locator App with TensorFlow Object Detection API 15. Becoming a Maker: Exploring Embedded AI at the Edge 16. Simulating a Self-Driving Car Using End-to-End Deep Learning with Keras 17. Building an Autonomous Car in Under an Hour: Reinforcement Learning with AWS DeepRacer **Guest-contributed Content**The book features chapters from the following industry experts:* Sunil Mallya (Amazon **AWS DeepRacer**) * Aditya Sharma and Mitchell Spryn (**Microsoft Autonomous Driving Cookbook**) * Sam Sterckval (**Edgise**) * Zaid Alyafeai (**TensorFlow.js**) The book also features content contributed by several industry veterans including François Chollet (**Keras**, **Google**), Jeremy Howard (**Fast.ai**), Pete Warden (**TensorFlow Mobile**), Anima Anandkumar (**NVIDIA**), Chris Anderson (**3D Robotics**), Shanqing Cai (**TensorFlow.js**), Daniel Smilkov (**TensorFlow.js**), Cristobal Valenzuela (**ml5.js**), Daniel Shiffman (**ml5.js**), Hart Woolery (**CV 2020**), Dan Abdinoor (**Fritz**), Chitoku Yato (**NVIDIA** Jetson Nano), John Welsh (**NVIDIA** Jetson Nano), and Danny Atsmon (**Cognata**). Preface To the Backend/Frontend/Mobile Software Developer To the Data Scientist To the Student To the Teacher To the Robotics Enthusiast What to Expect in Each Chapter Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Group Acknowledgments Personal Acknowledgments 1. Exploring the Landscape of Artificial Intelligence An Apology The Real Introduction What Is AI? Motivating Examples A Brief History of AI Exciting Beginnings The Cold and Dark Days A Glimmer of Hope How Deep Learning Became a Thing Recipe for the Perfect Deep Learning Solution Datasets Model Architecture Frameworks TensorFlow Keras PyTorch A continuously evolving landscape Hardware Responsible AI Bias Accountability and Explainability Reproducibility Robustness Privacy Summary Frequently Asked Questions 2. What’s in the Picture: Image Classification with Keras Introducing Keras Predicting an Image’s Category Investigating the Model ImageNet Dataset Model Zoos Class Activation Maps Summary 3. Cats Versus Dogs: Transfer Learning in 30 Lines with Keras Adapting Pretrained Models to New Tasks A Shallow Dive into Convolutional Neural Networks Transfer Learning Fine Tuning How Much to Fine Tune Building a Custom Classifier in Keras with Transfer Learning Organize the Data Build the Data Pipeline Number of Classes Binary classification Multiclass classification Batch Size Data Augmentation Model Definition Train the Model Set Training Parameters Start Training Test the Model Analyzing the Results Further Reading Summary 4. Building a Reverse Image Search Engine: Understanding Embeddings Image Similarity Feature Extraction Similarity Search Visualizing Image Clusters with t-SNE Improving the Speed of Similarity Search Length of Feature Vectors Reducing Feature-Length with PCA Scaling Similarity Search with Approximate Nearest Neighbors Approximate Nearest-Neighbor Benchmark Which Library Should I Use? Creating a Synthetic Dataset Brute Force Annoy NGT Faiss Improving Accuracy with Fine Tuning Fine Tuning Without Fully Connected Layers Siamese Networks for One-Shot Face Verification Case Studies Flickr Pinterest Celebrity Doppelgangers Spotify Image Captioning Summary 5. From Novice to Master Predictor: Maximizing Convolutional Neural Network Accuracy Tools of the Trade TensorFlow Datasets TensorBoard What-If Tool tf-explain Common Techniques for Machine Learning Experimentation Data Inspection Breaking the Data: Train, Validation, Test Early Stopping Reproducible Experiments End-to-End Deep Learning Example Pipeline Basic Transfer Learning Pipeline Basic Custom Network Pipeline How Hyperparameters Affect Accuracy Transfer Learning Versus Training from Scratch Effect of Number of Layers Fine-Tuned in Transfer Learning Effect of Data Size on Transfer Learning Effect of Learning Rate Effect of Optimizers Effect of Batch Size Effect of Resizing Effect of Change in Aspect Ratio on Transfer Learning Tools to Automate Tuning for Maximum Accuracy Keras Tuner AutoAugment AutoKeras Summary 6. Maximizing Speed and Performance of TensorFlow: A Handy Checklist GPU Starvation nvidia-smi TensorFlow Profiler + TensorBoard How to Use This Checklist Performance Checklist Data Preparation Data Reading Data Augmentation Training Inference Data Preparation Store as TFRecords Reduce Size of Input Data Use TensorFlow Datasets Data Reading Use tf.data Prefetch Data Parallelize CPU Processing Parallelize I/O and Processing Enable Nondeterministic Ordering Cache Data Turn on Experimental Optimizations Filter fusion Map and filter fusion Map fusion Autotune Parameter Values Data Augmentation Use GPU for Augmentation tf.image built-in augmentations NVIDIA DALI Training Use Automatic Mixed Precision Use Larger Batch Size Use Multiples of Eight Find the Optimal Learning Rate Use tf.function Overtrain, and Then Generalize Use progressive sampling Use progressive augmentation Use progressive resizing Install an Optimized Stack for the Hardware Optimize the Number of Parallel CPU Threads Use Better Hardware Distribute Training Examine Industry Benchmarks Inference Use an Efficient Model Quantize the Model Prune the Model Use Fused Operations Enable GPU Persistence Summary 7. Practical Tools, Tips, and Tricks Installation Training Model Data Privacy Education and Exploration One Last Question 8. Cloud APIs for Computer Vision: Up and Running in 15 Minutes The Landscape of Visual Recognition APIs Clarifai What’s unique about this API? Microsoft Cognitive Services What’s unique about this API? Google Cloud Vision What’s unique about this API? Amazon Rekognition What’s unique about this API? IBM Watson Visual Recognition Algorithmia What’s unique about this API? Comparing Visual Recognition APIs Service Offerings Cost Accuracy Bias Getting Up and Running with Cloud APIs Training Our Own Custom Classifier Top Reasons Why Our Classifier Does Not Work Satisfactorily Comparing Custom Classification APIs Performance Tuning for Cloud APIs Effect of Resizing on Image Labeling APIs Effect of Compression on Image Labeling APIs Effect of Compression on OCR APIs Effect of Resizing on OCR APIs Case Studies The New York Times Uber Giphy OmniEarth Photobucket Staples InDro Robotics Summary 9. Scalable Inference Serving on Cloud with TensorFlow Serving and KubeFlow Landscape of Serving AI Predictions Flask: Build Your Own Server Making a REST API with Flask Deploying a Keras Model to Flask Pros of Using Flask Cons of Using Flask Desirable Qualities in a Production-Level Serving System High Availability Scalability Low Latency Geographic Availability Failure Handling Monitoring Model Versioning A/B Testing Support for Multiple Machine Learning Libraries Google Cloud ML Engine: A Managed Cloud AI Serving Stack Pros of Using Cloud ML Engine Cons of Using Cloud ML Engine Building a Classification API TensorFlow Serving Installation KubeFlow Pipelines Fairing Installation Price Versus Performance Considerations Cost Analysis of Inference-as-a-Service Cost Analysis of Building Your Own Stack Summary 10. AI in the Browser with TensorFlow.js and ml5.js JavaScript-Based Machine Learning Libraries: A Brief History ConvNetJS Keras.js ONNX.js TensorFlow.js TensorFlow.js Architecture Running Pretrained Models Using TensorFlow.js Model Conversion for the Browser Training in the Browser Feature Extraction Data Collection Training GPU Utilization ml5.js PoseNet pix2pix Benchmarking and Practical Considerations Model Size Inference Time Case Studies Semi-Conductor TensorSpace Metacar Airbnb’s Photo Classification GAN Lab Summary 11. Real-Time Object Classification on iOS with Core ML The Development Life Cycle for Artificial Intelligence on Mobile A Brief History of Core ML Alternatives to Core ML TensorFlow Lite ML Kit Fritz Apple’s Machine Learning Architecture Domain-Based Frameworks ML Framework ML Performance Primitives Building a Real-Time Object Recognition App Conversion to Core ML Conversion from Keras Conversion from TensorFlow Dynamic Model Deployment On-Device Training Federated Learning Performance Analysis Benchmarking Models on iPhones Measuring Energy Impact Benchmarking Load Reducing App Size Avoid Bundling the Model Use Quantization Use Create ML Case Studies Magic Sudoku Seeing AI HomeCourt InstaSaber + YoPuppet Summary 12. Not Hotdog on iOS with Core ML and Create ML Collecting Data Approach 1: Find or Collect a Dataset Approach 2: Fatkun Chrome Browser Extension Approach 3: Web Scraper Using Bing Image Search API Training Our Model Approach 1: Use Web UI-based Tools Approach 2: Use Create ML Approach 3: Fine Tuning Using Keras Model Conversion Using Core ML Tools Building the iOS App Further Exploration Summary 13. Shazam for Food: Developing Android Apps with TensorFlow Lite and ML Kit The Life Cycle of a Food Classifier App An Overview of TensorFlow Lite TensorFlow Lite Architecture Model Conversion to TensorFlow Lite Building a Real-Time Object Recognition App ML Kit + Firebase Object Classification in ML Kit Custom Models in ML Kit Hosted Models Accessing a hosted model Uploading a hosted model A/B Testing Hosted Models Measuring an experiment Using the Experiment in Code TensorFlow Lite on iOS Performance Optimizations Quantizing with TensorFlow Lite Converter TensorFlow Model Optimization Toolkit Fritz A Holistic Look at the Mobile AI App Development Cycle How Do I Collect Initial Data? How Do I Label My Data? How Do I Train My Model? How Do I Convert the Model to a Mobile-Friendly Format? How Do I Make my Model Performant? How Do I Build a Great UX for My Users? How Do I Make the Model Available to My Users? How Do I Measure the Success of My Model? How Do I Improve My Model? How Do I Update the Model on My Users’ Phones? The Self-Evolving Model Case Studies Lose It! Portrait Mode on Pixel 3 Phones Speaker Recognition by Alibaba Face Contours in ML Kit Real-Time Video Segmentation in YouTube Stories Summary 14. Building the Purrfect Cat Locator App with TensorFlow Object Detection API Types of Computer-Vision Tasks Classification Localization Detection Segmentation Semantic segmentation Instance-level segmentation Approaches to Object Detection Invoking Prebuilt Cloud-Based Object Detection APIs Reusing a Pretrained Model Obtaining the Model Test Driving Our Model Deploying to a Device Building a Custom Detector Without Any Code The Evolution of Object Detection Performance Considerations Key Terms in Object Detection Intersection over Union Mean Average Precision Non-Maximum Suppression Using the TensorFlow Object Detection API to Build Custom Models Data Collection Labeling the Data Preprocessing the Data Inspecting the Model Training Model Conversion Image Segmentation Case Studies Smart Refrigerator Crowd Counting Wildlife conservation Kumbh Mela Face Detection in Seeing AI Autonomous Cars Summary 15. Becoming a Maker: Exploring Embedded AI at the Edge Exploring the Landscape of Embedded AI Devices Raspberry Pi Intel Movidius Neural Compute Stick Google Coral USB Accelerator NVIDIA Jetson Nano FPGA + PYNQ FPGAs PYNQ platform Arduino A Qualitative Comparison of Embedded AI Devices Hands-On with the Raspberry Pi Speeding Up with the Google Coral USB Accelerator Port to NVIDIA Jetson Nano Comparing the Performance of Edge Devices Case Studies JetBot Squatting for Metro Tickets Cucumber Sorter Further Exploration Summary 16. Simulating a Self-Driving Car Using End-to-End Deep Learning with Keras A Brief History of Autonomous Driving Deep Learning, Autonomous Driving, and the Data Problem The “Hello, World!” of Autonomous Driving: Steering Through a Simulated Environment Setup and Requirements Data Exploration and Preparation Identifying the Region of Interest Data Augmentation Dataset Imbalance and Driving Strategies Training Our Autonomous Driving Model Drive Data Generator Model Definition Callbacks Deploying Our Autonomous Driving Model Further Exploration Expanding Our Dataset Training on Sequential Data Reinforcement Learning Summary 17. Building an Autonomous Car in Under an Hour: Reinforcement Learning with AWS DeepRacer A Brief Introduction to Reinforcement Learning Why Learn Reinforcement Learning with an Autonomous Car? Practical Deep Reinforcement Learning with DeepRacer Building Our First Reinforcement Learning Step 1: Create Model Step 2: Configure Training Configure the simulation environment Configure the action space Configure reward function Configure stop conditions Step 3: Model Training Step 4: Evaluating the Performance of the Model Reinforcement Learning in Action How Does a Reinforcement Learning System Learn? Reinforcement Learning Theory The Markov decision process Model free versus model based Value based Policy based Policy based or value based—why not both? Delayed rewards and discount factor (γ) Reinforcement Learning Algorithm in AWS DeepRacer Deep Reinforcement Learning Summary with DeepRacer as an Example Step 5: Improving Reinforcement Learning Models Algorithm settings Hyperparameters for the neural network Insights into model training Heatmap visualization Improving the speed of our model Racing the AWS DeepRacer Car Building the Track AWS DeepRacer Single-Turn Track Template Running the Model on AWS DeepRacer Driving the AWS DeepRacer Vehicle Autonomously Sim2Real transfer Further Exploration DeepRacer League Advanced AWS DeepRacer AI Driving Olympics DIY Robocars Roborace Summary A. A Crash Course in Convolutional Neural Networks Machine Learning Perceptron Activation Functions Neural Networks Backpropagation Shortcoming of Neural Networks Desired Properties of an Image Classifier Convolution Pooling Structure of a CNN Further Exploration Index Whether you're a software engineer aspiring to enter the world of deep learning, a veteran data scientist, or a hobbyist with a simple dream of making the next viral AI app, you might have wondered where to begin. This step-by-step guide teaches you how to build practical deep learning applications for the cloud, mobile, browsers, and edge devices using a hands-on approach. Relying on years of industry experience transforming deep learning research into award-winning applications, Anirudh Koul, Siddha Ganju, and Meher Kasam guide you through the process of converting an idea into something that people in the real world can use. Train, tune, and deploy computer vision models with Keras, TensorFlow, Core ML, and TensorFlow Lite. Develop AI for a range of devices including Raspberry Pi, Jetson Nano, and Google Coral. Explore fun projects, from Silicon Valley's Not Hotdog app to 40+ industry case studies. Simulate an autonomous car in a video game environment and build a miniature version with reinforcement learning. Use transfer learning to train models in minutes. Discover 50+ practical tips for maximizing model accuracy and speed, debugging, and scaling to millions of users. List of Chapters Exploring the Landscape of Artificial Intelligence What's in the Picture: Image Classification with Keras Cats Versus Dogs: Transfer Learning in 30 Lines with Keras Building a Reverse Image Search Engine: Understanding Embeddings From Novice to Master Predictor: Maximizing Convolutional Neural Network Accuracy Maximizing Speed and Performance of TensorFlow: A Handy Checklist Practical Tools, Tips, and Tricks Cloud APIs for Computer Vision: Up and Running in 15 Minutes Scalable Inference Serving on Cloud with TensorFlow Serving and KubeFlow AI in the Browser with TensorFlow.js and ml5.js Real-Time Object Classification on iOS with Core ML Not Hotdog on iOS with Core ML and Create ML Shazam for Food: Developing Android Apps with TensorFlow Lite and ML Kit Building the Purrfect Cat Locator App with TensorFlow Object Detection API Becoming a Maker: Exploring Embedded AI at the Edge Simulating a Self-Driving Car Using End-to-End Deep Learning with Keras Building an Autonomous Car in Under an Hour: Reinforcement Learning with AWS DeepRacer Guest-contributed Content The book features chapters from the following industry experts: Sunil Mallya (Amazon AWS DeepRacer ) Aditya Sharma and Mitchell Spryn ( Microsoft Autonomous Driving Cookbook ) Sam Sterckval ( Edgise ) Zaid Alyafeai ( TensorFlow.js ) The book also features content contributed by several industry veterans including François Chollet ( Keras , Google ), Jeremy Howard ( Fast.ai ), Pete Warden ( TensorFlow Mobile ), Anima Anandkumar ( NVIDIA ), Chris Anderson ( 3D Robotics ), Shanqing Cai ( TensorFlow.js ), Daniel Smilkov ( TensorFlow.js ), Cristobal Valenzuela ( ml5.js ), Daniel Shiffman ( ml5.js ), Hart Woolery ( CV 2020 ), Dan Abdinoor ( Fritz ), Chitoku Yato ( NVIDIA Jetson Nano), John Welsh ( NVIDIA Jetson Nano), and Danny Atsmon ( Cognata ). Whether you're a software engineer aspiring to enter the world of artificial intelligence, a veteran data scientist, or a hobbyist with a simple dream of making the next viral AI app, you might have wondered where do I begin? This step-by-step guide teaches you how to build practical applications using deep neural networks for the cloud and mobile using a hands-on approach. Relying on years of industry experience transforming deep learning research into award-winning applications, Anirudh Koul, Siddha Ganju, and Meher Kasam guide you through the process of converting an idea into something that people can use in the real world. Train, optimize, and deploy computer vision models with Keras, TensorFlow, CoreML, TensorFlow Lite, and MLKit, rapidly taking your system from zero to production quality. Develop AI applications for the desktop, cloud, smartphones, browser, and smart robots using Raspberry Pi, Jetson Nano, and Google Coral Perform Object Classification, Detection, Segmentation in real-time Learn by building examples such as Silicon Valley's "Not Hotdog" app, image search engines, and Snapchat filters Train an autonomous car in a video game environment and then build a real mini version Use transfer learning to train models in minutes Generate photos from sketches in your browser with Generative Adversarial Networks (GANs with pix2pix), and Body Pose Estimation (PoseNet) Discover 50+ practical tips for data collection, model interoperability, debugging, avoiding bias, and scaling to millions of users ** Featured as a learning resource on the official Keras website ** Whether you're a software engineer aspiring to enter the world of deep learning, a veteran data scientist, or a hobbyist with a simple dream of making the next viral AI app, you might have wondered where to begin. This step-by-step guide teaches you how to build practical deep learning applications for the cloud, mobile, browsers, and edge devices using a hands-on approach. If your goal is to build something creative, useful, scalable, or just plain cool, this book is for you. Relying on decades of combined industry experience transforming deep learning research into award-winning applications, Anirudh Koul, Siddha Ganju, and Meher Kasam guide you through the process of converting an idea into something that people in the real world can use. List of Chapters Guest-contributed Content The book features chapters from the following industry The book also features content contributed by several industry veterans including Franois Chollet ( Keras , Google ), Jeremy Howard ( Fast.ai ), Pete Warden ( TensorFlow Mobile ), Anima Anandkumar ( NVIDIA ), Chris Anderson ( 3D Robotics ), Shanqing Cai ( TensorFlow.js ), Daniel Smilkov ( TensorFlow.js ), Cristobal Valenzuela ( ml5.js ), Daniel Shiffman ( ml5.js ), Hart Woolery ( CV 2020 ), Dan Abdinoor ( Fritz ), Chitoku Yato ( NVIDIA Jetson Nano), John Welsh ( NVIDIA Jetson Nano), and Danny Atsmon ( Cognata ).