Pro Hadoop (Expert's Voice in Open Source)

You've heard the hype about Hadoop: it runs petabyte–scale data mining tasks insanely fast, it runs gigantic tasks on clouds for absurdly cheap, it's been heavily committed to by tech giants like IBM, Yahoo!, and the Apache Project, and it's completely open-source (thus __free__). But what exactly is it, and more importantly, how do you even get a Hadoop cluster up and running? From Apress, the name you've come to trust for hands–on technical knowledge, __Pro Hadoop__ brings you up to speed on Hadoop. You learn the ins and outs of MapReduce; how to structure a cluster, design, and implement the Hadoop file system; and how to build your first cloud–computing tasks using Hadoop. Learn how to let Hadoop take care of distributing and parallelizing your software—you just focus on the code, Hadoop takes care of the rest. Best of all, you'll learn from a tech professional who's been in the Hadoop scene since day one. Written from the perspective of a principal engineer with down–in–the–trenches knowledge of what to do __wrong__ with Hadoop, you learn how to avoid the common, expensive first errors that everyone makes with creating their own Hadoop system or inheriting someone else's. Skip the novice stage and the expensive, hard–to–fix mistakes...go straight to seasoned pro on the hottest cloud–computing framework with __Pro Hadoop__. Your productivity will blow your managers away. Prelims ......Page 1 Contents at a Glance......Page 7 Contents......Page 9 About the Author......Page 21 About the Technical Reviewer......Page 23 Acknowledgments......Page 25 Introduction......Page 27 Getting Started with hadoop Core......Page 31 Unknown......Page 0 Hadoop Core MapReduce......Page 35 The Hadoop Distributed File System......Page 36 hadoop on a Linux System......Page 37 hadoop on a Windows System: how to and Common problems......Page 41 Checking Your Environment......Page 43 running the pi estimator......Page 48 examining the Output: Input Splits, Shuffles, Spills, and Sorts......Page 49 Hadoop Tests......Page 53 the Basics of a Mapreduce Job......Page 57 A Simple Map Function: IdentityMapper......Page 61 A Simple Reduce Function: IdentityReducer......Page 64 Specifying Input Formats......Page 75 Setting the Output Parameters......Page 77 Configuring the Reduce Phase......Page 81 Setting Up a Custom Mapper......Page 86 the reporter Object......Page 87 the Counters and exceptions......Page 89 After the Job Finishes......Page 91 Was this Job really Successful?......Page 92 Creating a Custom Reducer......Page 93 the configure Method......Page 96 Using a Custom Partitioner......Page 97 the Basics of Multimachine Clusters......Page 101 Hadoop Configuration Files......Page 105 Hadoop Core Server Configuration......Page 106 per-Machine Data......Page 107 Maximum Concurrent Map tasks per tasktracker......Page 108 JVM Options for the task Virtual Machines......Page 109 Network requirements......Page 110 the hadoop-site.xml File......Page 112 the hadoop-metrics.properties File......Page 115 Distributing the Configuration......Page 116 Verifying the Cluster Configuration......Page 117 Formatting HDFS......Page 118 Starting HDFS......Page 119 Correcting Errors......Page 121 Starting MapReduce......Page 122 Running a Test Job on the Cluster......Page 124 hDFS Details for Multimachine Clusters......Page 127 Building the HDFS Configuration......Page 128 Customizing the conf/hadoop-env.sh File......Page 129 Distributing Your Installation Data......Page 131 Formatting Your HDFS......Page 132 Starting Your HDFS Installation......Page 134 Verifying HDFS Is Running......Page 135 Checking the NameNodes......Page 136 Checking the DataNodes......Page 138 File Descriptors......Page 141 Block Service Threads......Page 142 NameNode Threads......Page 143 Reserved Disk Space......Page 144 Disk I/O......Page 145 Secondary NameNode Disk I/O tuning......Page 146 DataNode Disk I/O tuning......Page 147 Network I/O Tuning......Page 149 DataNode Recovery and Addition......Page 150 DataNode Decommissioning......Page 151 Deleted File Recovery......Page 152 Data Loss or Corruption......Page 153 No Live Node Contains Block errors......Page 154 DataNode or NameNode Pauses......Page 155 Mapreduce Details for Multimachine Clusters......Page 157 Adding Resources to the Task Classpath......Page 162 Looking Up Names......Page 163 Finding a File or archive in the Localized Cache......Page 164 Setting the Default File System URI......Page 165 Setting the JobTracker Location......Page 166 the configure() Method......Page 168 the close() Method......Page 170 Mapper Class Declaration and Member Fields......Page 172 Initializing the Mapper with Spring......Page 173 Creating the Spring application Context......Page 174 Using Spring to autowire the Mapper Class......Page 175 The TotalOrderPartitioner Class......Page 179 Building a range table......Page 180 The KeyFieldBasedPartitioner Class......Page 181 A Simple Transforming Reducer......Page 184 A Reducer That Uses Three Partitions......Page 189 Text Files......Page 196 Sequence Files......Page 198 Map Files......Page 199 Codec Specification......Page 201 Map Task Output......Page 202 JAR, Zip, and Tar Files......Page 204 tuning Your Mapreduce Jobs......Page 207 On Job Submission......Page 208 Map task Submission and execution......Page 209 Merge-Sorting......Page 210 Writing to hDFS......Page 211 Server-Level parameters......Page 212 hDFS tunable parameters......Page 213 Jobtracker and tasktracker tunable parameters......Page 215 Per-Job Tunable Parameters......Page 218 Nagios: A Monitoring and Alert Generation Framework......Page 222 Ganglia: A Visual Monitoring Tool with History......Page 223 Speeding Up the Job and Task Start......Page 226 Optimizing a Job’s Map Phase......Page 228 Tuning the Reduce Task Setup......Page 231 Dealing with the Job tail......Page 235 Unit testing and Debugging......Page 237 Requirements for Using ClusterMapReduceTestCase......Page 238 troubles with Jetty, the http Server for the Web UI......Page 239 the hadoop Core Jar Is Missing or Malformed......Page 241 the Virtual Cluster Failed to Start......Page 242 Core Methods of ClusterMapreduceDelegate......Page 244 Configuration parameters for Interacting with Virtual Clusters......Page 245 the testCase Class Declaration......Page 246 the Cluster Stop Method......Page 247 the actual test......Page 248 a test Case that Launches a Mapreduce Job......Page 250 Running an Entire MapReduce Job in a Single JVM......Page 253 Debugging a Task Running on a Cluster......Page 260 Configuring the Job or Cluster to Save the task Local Working Directory......Page 264 running a Job with a Keep pattern and Debugging via the Isolationrunner......Page 265 advanced and alternate Mapreduce techniques......Page 269 Streaming Command-Line Arguments......Page 273 Using -inputreader org.apache.hadoop.streaming.StreamXmlrecordreader......Page 276 Using Counters in Streaming and Pipes Jobs......Page 278 libhdfs......Page 279 fuse-dfs......Page 281 Mounting an HDFS File System Using fuse_dfs......Page 282 Chaining: Efficiently Connecting Multiple Map and/or Reduce Steps......Page 287 type Checking for Chained Keys and Values......Page 288 Configuring Mapper tasks to be a Chain......Page 289 Configuring the reducer tasks to Be Chains......Page 291 Map-side Join: Sequentially Reading Data from Multiple Sorted Inputs......Page 295 examining Join Datasets......Page 296 Under the Covers: how a Join Works......Page 297 types of Joins Supported......Page 298 Composing a Join Specification......Page 299 Building and running a Join......Page 300 the Magic of the tupleWritable in the Mapper.map() Method......Page 301 Aggregation Using Streaming......Page 305 Aggregation Using Java Classes......Page 307 Specifying the ValueAggregatorDescriptor Class via Configuration Parameters......Page 308 Side Effect Files: Map and Reduce Tasks Can Write Additional Output Files......Page 309 Skipping Bad Records......Page 310 Enabling the Capacity Scheduler......Page 311 Solving problems with hadoop......Page 315 A Single Reduce Task......Page 317 Key Contents and Comparators......Page 318 A Helper Class for Keys......Page 321 The Mapper......Page 324 The Reducer......Page 328 The Driver......Page 331 The Simple IP Range Partitioner......Page 332 Search Space Keys for Each Reduce Task That May Contain Matching Keys......Page 335 Helper Class for Keys Modifications......Page 341 HBase: HDFS-Based Column-Oriented Table......Page 359 Setting Up and running hive......Page 360 Mahout: Machine Learning Algorithms......Page 362 Katta: a Distributed Lucene Index Server......Page 363 CloudBase: Data Warehousing......Page 364 training......Page 365 Scale Unlimited......Page 366 Zero-Configuration, Two-Node Virtual Cluster for Testing......Page 367 Eclipse Project for the Example Code......Page 368 The JobConf Object in detail......Page 369 public JobConf(Configuration conf, Class exampleClass)......Page 377 public JobConf(boolean loadDefaults)......Page 378 public void addResource(String name)......Page 379 public String get(String name)......Page 380 public float getFloat(String name, float defaultValue)......Page 381 public Configuration.IntegerRanges getRange(String name, String defaultValue)......Page 382 public String[] getStrings(String name)......Page 383 public void setStrings(String name, String... values)......Page 384 public Class getClass(String name, Class defaultValue)......Page 385 public void setClass(String name, Class theClass, Class xface)......Page 386 public String[] getLocalDirs() throws IOException......Page 387 public String getJobLocalDir()......Page 388 public Reader getConfResourceAsReader(String name)......Page 389 public String getUser()......Page 390 public void setWorkingDirectory(Path dir)......Page 391 public int getNumTasksToExecutePerJvm()......Page 392 public OutputCommitter getOutputCommitter()......Page 393 public boolean getCompressMapOutput()......Page 394 public Class getMapOutp utCompressorClass(Class defaultValue)......Page 395 public void setMapOutputValueClass(Class theClass)......Page 396 public void setOutputValueClass(Class theClass)......Page 397 public void setKeyFieldComparatorOptions(String keySpec)......Page 398 public String getKeyFieldComparatorOption()......Page 399 public void setKeyFieldPartitionerOptions(String keySpec)......Page 400 public void setOutputValueGroupingComparator(Class theClass)......Page 401 public void setMapRunnerClass(Class theClass)......Page 403 public void setCombinerClass(Class theClass)......Page 404 public void setMapSpeculativeExecution (boolean speculativeExecution)......Page 405 public void setNumReduceTasks(int n)......Page 406 public int getMaxTaskFailuresPerTracker()......Page 407 public void setMaxReduceTaskFailuresPercent(int percent)......Page 408 public String getSessionId()......Page 409 public boolean getProfileEnabled()......Page 410 public Configuration.IntegerRanges getProfileTaskRange (boolean isMap)......Page 411 public String getMapDebugScript()......Page 412 public void setReduceDebugScript(String rDbgScript)......Page 413 public void setQueueName(String queueName)......Page 414 public Iterator iterator()......Page 415 public void write(DataOutput out) throws IOException......Page 416 Index......Page 417

You’ve heard the hype about Hadoop: it runs petabyte–scale data mining tasks insanely fast, it runs gigantic tasks on clouds for absurdly cheap, it’s been heavily committed to by tech giants like IBM, Yahoo!, and the Apache Project, and it’s completely open-source (thus free). But what exactly is it, and more importantly, how do you even get a Hadoop cluster up and running?

From Apress, the name you’ve come to trust for hands–on technical knowledge, Pro Hadoop brings you up to speed on Hadoop. You learn the ins and outs of MapReduce; how to structure a cluster, design, and implement the Hadoop file system; and how to build your first cloud–computing tasks using Hadoop. Learn how to let Hadoop take care of distributing and parallelizing your software—you just focus on the code, Hadoop takes care of the rest.

Best of all, you’ll learn from a tech professional who’s been in the Hadoop scene since day one. Written from the perspective of a principal engineer with down–in–the–trenches knowledge of what to do wrong with Hadoop, you learn how to avoid the common, expensive first errors that everyone makes with creating their own Hadoop system or inheriting someone else’s.

Skip the novice stage and the expensive, hard–to–fix mistakes...go straight to seasoned pro on the hottest cloud–computing framework with Pro Hadoop. Your productivity will blow your managers away.

What you’ll learn

• Set up a stand–alone Hadoop cluster the smart way, laid out simply and step by step so you can get up and running quickly to build your next data center, collaborative, data–intensive Internet services application, Software as a Service (SaaS), and more.
• Optimize your Hadoop production tasks like an experienced pro.
• Work with time–proven, bulletproof standard patterns that have been tested and debugged in high–volume production.
• Understand just enough theoretical knowledge to know why something works in Hadoop, without getting bogged down in abstruse walls of theory.
• Get detailed explanations of not only how to do something with Hadoop, but also why, from a front–line coder with years in the Hadoop game.
• Turn someone else’s expensive cluster–wide “wrong” into an orderly, productive "right" with professional–level debugging and testing.

Who this book is for

IT professionals interested in investigating Hadoop and implementing it in their organizations, and existing Hadoop users who want to deepen their professional toolkits.

Table of Contents

1. Getting Started with Hadoop Core
2. The Basics of a MapReduce Job
3. The Basics of Multimachine Clusters
4. HDFS Details for Multimachine Clusters
5. MapReduce Details for Multimachine Clusters
6. Tuning Your MapReduce Jobs
7. Unit Testing and Debugging
8. Advanced and Alternate MapReduce Techniques
9. Solving Problems with Hadoop
10. Projects Based On Hadoop and Future Directions

Annotation You've heard the hype about Hadoop: it runs petabyte-scale data mining tasks insanely fast, it runs gigantic tasks on clouds for absurdly cheap, it's been heavily committed to by tech giants like IBM, Yahoo!, and the Apache Project, and it's completely open-source (thus free ). But what exactly is it, and more importantly, how do you even get a Hadoop cluster up and running? From Apress, the name you've come to trust for hands-on technical knowledge, Pro Hadoop brings you up to speed on Hadoop. You learn the ins and outs of MapReduce; how to structure a cluster, design, and implement the Hadoop file system; and how to build your first cloud-computing tasks using Hadoop. Learn how to let Hadoop take care of distributing and parallelizing your software--you just focus on the code, Hadoop takes care of the rest. Best of all, you'll learn from a tech professional who's been in the Hadoop scene since day one. Written from the perspective of a principal engineer with down-in-the-trenches knowledge of what to do wrong with Hadoop, you learn how to avoid the common, expensive first errors that everyone makes with creating their own Hadoop system or inheriting someone else's. Skip the novice stage and the expensive, hard-to-fix mistakes ... go straight to seasoned pro on the hottest cloud-computing framework with Pro Hadoop . Your productivity will blow your managers away. What you'll learn Set up a stand-alone Hadoop cluster the smart way, laid out simply and step by step so you can get up and running quickly to build your next data center, collaborative, data-intensive Internet services application, Software as a Service (SaaS), and more. Optimize your Hadoop production tasks like an experienced pro. Work with time-proven, bulletproof standard patterns that have been tested and debugged in high-volume production. Understand just enough theoretical knowledge to know why something works in Hadoop, without getting bogged down in abstruse walls of theory. Get detailed explanations of not only how to do something with Hadoop, but also why , from a front-line coder with years in the Hadoop game. Turn someone else's expensive cluster-wide "wrong" into an orderly, productive "right" with professional-level debugging and testing. Who this book is for IT professionals interested in investigating Hadoop and implementing it in their organizations, and existing Hadoop users who want to deepen their professional toolkits. Table of Contents Getting Started with Hadoop Core The Basics of a MapReduce Job The Basics of Multimachine Clusters HDFS Details for Multimachine Clusters MapReduce Details for Multimachine Clusters Tuning Your MapReduce Jobs Unit Testing and Debugging Advanced and Alternate MapReduce Techniques Solving Problems with Hadoop Projects Based On Hadoop and Future Directions "You learn the ins and outs of MapReduce; how to structure a cluster, design, and implement the Hadoop file system; and how to structure your first cloud--computing tasks using Hadoop. Learn how to let Hadoop take care of distributing and parallelizing your software--you just focus on the code, Hadoop takes care of the rest"--Resource description page