Introducing Microsoft Azure HDInsight

Microsoft Azure Hdinsight Is Microsoft's 100 Percent Compliant Distribution Of Apache Hadoop On Microsoft Azure. This Means That Standard Hadoop Concepts And Technologies Apply, So Learning The Hadoop Stack Helps You Learn The Hdinsight Service. At The Time Of This Writing, Hdinsight (version 3.0) Uses Hadoop Version 2.2 And Hortonworks Data Platform 2.0. In Introducing Microsoft Azure Hdinsight, We Cover What Big Data Really Means, How You Can Use It To Your Advantage In Your Company Or Organization, And One Of The Services You Can Use To Do That Quickly-specifically, Microsoft's Hdinsight Service. We Start With An Overview Of Big Data And Hadoop, But We Don't Emphasize Only Concepts In This Book-we Want You To Jump In And Get Your Hands Dirty Working With Hdinsight In A Practical Way. To Help You Learn And Even Implement Hdinsight Right Away, We Focus On A Specific Use Case That Applies To Almost Any Organization And Demonstrate A Process That You Can Follow Along With. We Also Help You Learn More. In The Last Chapter, We Look Ahead At The Future Of Hdinsight And Give You Recommendations For Self-learning So That You Can Dive Deeper Into Important Concepts And Round Out Your Education On Working With Big Data. Cover 1 Microsoft Press Store 2 Newsletters 3 Guided Tours app 4 Copyright page 5 Table of Contents 6 Foreword 9 Introduction 11 Who should read this book 11 Assumptions 11 Who should not read this book 12 Organization of this book 12 Finding your best starting point in this book 12 Book scenario 13 Conventions and features in this book 14 System requirements 15 Sample data and code samples 15 Working with sample data 16 Using the code samples 17 Acknowledgments 17 Errata & book support 18 We want to hear from you 18 Stay in touch 19 Chapter 1: Big data, quick intro 20 A quick (and not so quick) definition of terms 20 Use cases, when and why 21 Tools and approaches—scale up and scale out 22 Hadoop 23 HDFS 24 MapReduce 24 HDInsight 25 Microsoft Azure 25 Services 27 Storage 29 HDInsight service 30 Interfaces 31 Pig 31 Hive 31 Other interfaces and tools 31 HDInsight Emulator 32 Summary 32 Chapter 2: Getting started with HDInsight 33 HDInsight as cloud service 33 Microsoft Azure subscription 34 Open the Azure Management Portal 34 Add storage to your Azure subscription 35 Create an HDInsight cluster 38 Manage a cluster from the Azure Management Portal 41 The cluster dashboard 41 Monitor a cluster 43 Configure a cluster 43 Accessing the HDInsight name node using Remote Desktop 47 Hadoop name node status 48 Hadoop MapReduce status 51 Hadoop command line 58 Setting up the HDInsight Emulator 61 HDInsight Emulator and Windows PowerShell 61 Installing the HDInsight Emulator 62 Using the HDInsight Emulator 63 Name node status 67 MapReduce job status 69 Running the WordCount MapReduce job in the HDInsight Emulator 70 Summary 74 Chapter 3: Programming HDInsight 75 Getting started 75 MapReduce jobs and Windows PowerShell 76 Hadoop streaming 81 Write a Hadoop streaming mapper and reducer using C# 82 Run the HDInsight streaming job 84 Using the HDInsight .NET SDK 87 Summary 94 Chapter 4: Working with HDInsight data 95 Using Apache Hive with HDInsight 95 Upload the data to Azure Storage 96 Use PowerShell to create tables in Hive 97 Run HiveQL queries against the Hive table 100 Using Apache Pig with HDInsight 101 Using Microsoft Excel and Power Query to work with HDInsight data 104 Using Sqoop with HDInsight 110 Summary 115 Chapter 5: What next? 116 Integrating your HDInsight clusters into your organization 116 Data management layer 117 Data enrichment layer 117 Analytics layer 118 Hadoop deployment options on Windows 119 Latest product releases and the future of HDInsight 121 Latest HDInsight improvements 121 HDInsight and the Microsoft Analytics Platform System 122 Data refinery or data lakes use case 125 Data exploration use case 125 Hadoop as a store for cold data 126 Study guide: Your next steps 126 Getting started with HDInsight 127 Running HDInsight samples 127 Connecting HDInsight to Excel with Power Query 128 Using Hive with HDInsight 128 Hadoop 2.0 and Hortonworks Data Platform 128 PolyBase in the Parallel Data Warehouse appliance 128 Recommended books 129 Summary 129 About the authors 130 Free ebooks 132 Tell us what you think! 133 Microsoft Press, 2014. — 130 p. Book is intended to help database and business intelligence (BI) professionals, programmers, Hadoop administrators, researchers, technical architects, operations engineers, data analysts, and data scientists understand the core concepts of HDInsight and related technologies. Topics included: - Inreoduction - Big data, quick overview - Getting started with HDInsight - Programming HDInsight - Working with HDInsight data - What next?