Parallel R

It's tough to argue with R as a high-quality, cross-platform, open source statistical software product—unless you're in the business of crunching Big Data. This concise book introduces you to several strategies for using R to analyze large datasets, including three chapters on using R and Hadoop together. You'll learn the basics of Snow, Multicore, Parallel, Segue, RHIPE, and Hadoop Streaming, including how to find them, how to use them, when they work well, and when they don't. With these packages, you can overcome R's single-threaded nature by spreading work across multiple CPUs, or offloading work to multiple machines to address R's memory barrier. Snow: works well in a traditional cluster environment Multicore: popular for multiprocessor and multicore computers Parallel: part of the upcoming R 2.14.0 release R+Hadoop: provides low-level access to a popular form of cluster computing RHIPE: uses Hadoop's power with R's language and interactive shell Segue: lets you use Elastic MapReduce as a backend for lapply-style operations Table of Contents 5 Preface 9 Conventions Used in This Book 9 Using Code Examples 9 Safari® Books Online 10 How to Contact Us 10 Acknowledgments 11 Q. Ethan McCallum 11 Stephen Weston 12 Chapter 1. Getting Started 13 Why R? 13 Why Not R? 13 The Solution: Parallel Execution 14 A Road Map for This Book 14 What We’ll Cover 15 Looking Forward... 15 What We’ll Assume You Already Know 15 In a Hurry? 16 snow 16 multicore 16 parallel 16 R+Hadoop 16 RHIPE 17 Segue 17 Summary 17 Chapter 2. snow 19 Quick Look 19 How It Works 19 Setting Up 20 Working with It 21 Creating Clusters with makeCluster 21 Parallel K-Means 22 Initializing Workers 24 Load Balancing with clusterApplyLB 25 Task Chunking with parLapply 27 Vectorizing with clusterSplit 30 Load Balancing Redux 32 Functions and Environments 35 Random Number Generation 37 snow Configuration 38 Installing Rmpi 41 Executing snow Programs on a Cluster with Rmpi 42 Executing snow Programs with a Batch Queueing System 44 Troubleshooting snow Programs 45 When It Works... 47 ...And When It Doesn’t 48 The Wrap-up 48 Chapter 3. multicore 49 Quick Look 49 How It Works 50 Setting Up 50 Working with It 51 The mclapply Function 51 The mc.cores Option 51 The mc.set.seed Option 52 Load Balancing with mclapply 54 The pvec Function 54 The parallel and collect Functions 55 Using collect Options 56 Parallel Random Number Generation 58 The Low-Level API 59 When It Works... 61 ...And When It Doesn’t 61 The Wrap-up 61 Chapter 4. parallel 63 Quick Look 64 How It Works 64 Setting Up 64 Working with It 65 Getting Started 65 Creating Clusters with makeCluster 66 Parallel Random Number Generation 67 Summary of Differences 69 When It Works... 70 ...And When It Doesn’t 70 The Wrap-up 70 Chapter 5. A Primer on MapReduce and Hadoop 71 Hadoop at Cruising Altitude 71 A MapReduce Primer 72 Thinking in MapReduce: Some Pseudocode Examples 73 Calculate Average Call Length for Each Date 74 Number of Calls by Each User, on Each Date 74 Run a Special Algorithm on Each Record 75 Binary and Whole-File Data: SequenceFiles 75 No Cluster? No Problem! Look to the Clouds... 76 The Wrap-up 78 Chapter 6. R+Hadoop 79 Quick Look 79 How It Works 79 Setting Up 80 Working with It 80 Simple Hadoop Streaming (All Text) 81 Streaming, Redux: Indirectly Working with Binary Data 84 The Java API: Binary Input and Output 86 Processing Related Groups (the Full Map and Reduce Phases) 91 When It Works... 95 ...And When It Doesn’t 95 The Wrap-up 96 Chapter 7. RHIPE 97 Quick Look 97 How It Works 97 Setting Up 98 Working with It 99 Phone Call Records, Redux 99 Tweet Brevity 103 More Complex Tweet Analysis 108 When It Works... 110 ...And When It Doesn’t 111 The Wrap-up 112 Chapter 8. Segue 113 Quick Look 113 How It Works 114 Setting Up 114 Working with It 114 Model Testing: Parameter Sweep 114 When It Works... 117 ...And When It Doesn’t 117 The Wrap-up 118 Chapter 9. New and Upcoming 119 doRedis 119 RevoScale R and RevoConnectR (RHadoop) 120 cloudNumbers.com 120 It’s tough to argue with R as a high-quality, cross-platform, open source statistical software product—unless you’re in the business of crunching Big Data. This concise book introduces you to several strategies for using R to analyze large datasets, including three chapters on using R and Hadoop together. You’ll learn the basics of Snow, Multicore, Parallel, Segue, RHIPE, and Hadoop Streaming, including how to find them, how to use them, when they work well, and when they don’t. With these packages, you can overcome R’s single-threaded nature by spreading work across multiple CPUs, or offloading work to multiple machines to address R’s memory barrier. Snow: works well in a traditional cluster environment Multicore: popular for multiprocessor and multicore computers Parallel: part of the upcoming R 2.14.0 release R Hadoop: provides low-level access to a popular form of cluster computing RHIPE: uses Hadoop’s power with R’s language and interactive shell Segue: lets you use Elastic MapReduce as a backend for lapply-style operations It's tough to argue with R as a high-quality, cross-platform, open source statistical software product--unless you're in the business of crunching Big Data. This concise book introduces you to several strategies for using R to analyze large datasets. You'll learn the basics of Snow, Multicore, Parallel, and some Hadoop-related tools, including how to find them, how to use them, when they work well, and when they don't. With these packages, you can overcome R's single-threaded nature by spreading work across multiple CPUs, or offloading work to multiple machines to address R's memory barrier. Snow Annotation R is a wonderful thing: in recent years this free, open-source product has become a popular toolkit for statistical analysis and programming. Two of R's limitations become especially troublesome in the current era of large-scale data analysis. It's possible to break past these boundaries by putting R on the parallel path