Big Data 2.0 Processing Systems: A Survey (SpringerBriefs in Computer Science)

This book provides readers the ℓ́ℓbig pictureℓ́ℓ and a comprehensive survey of the domain of big data processing systems. For the past decade, the Hadoop framework has dominated the world of big data processing, yet recently academia and industry have started to recognize its limitations in several application domains and big data processing scenarios such as the large-scale processing of structured data, graph data and streaming data. Thus, it is now gradually being replaced by a collection of engines that are dedicated to specific verticals (e.g. structured data, graph data, and streaming data). The book explores this new wave of systems, which it refers to as Big Data 2.0 processing systems. After Chapter 1 presents the general background of the big data phenomena, Chapter 2 provides an overview of various general-purpose big data processing systems that allow their users to develop various big data processing jobs for different application domains. In turn, Chapter 3 examines various systems that have been introduced to support the SQL flavor on top of the Hadoop infrastructure and provide competing and scalable performance in the processing of large-scale structured data. Chapter 4 discusses several systems that have been designed to tackle the problem of large-scale graph processing, while the main focus of Chapter 5 is on several systems that have been designed to provide scalable solutions for processing big data streams, and on other sets of systems that have been introduced to support the development of data pipelines between various types of big data processing jobs and systems. Lastly, Chapter 6 shares conclusions and an outlook on future research challenges. Overall, the book offers a valuable reference guide for students, researchers and professionals in the domain of big data processing systems. Further, its comprehensive content will hopefully encourage readers to pursue further research on the subject This book provides readers the ĺlbig pictureĺl and a comprehensive survey of the domain of big data processing systems. For the past decade, the Hadoop framework has dominated the world of big data processing, yet recently academia and industry have started to recognize its limitations in several application domains and big data processing scenarios such as the large-scale processing of structured data, graph data and streaming data. Thus, it is now gradually being replaced by a collection of engines that are dedicated to specific verticals (e.g. structured data, graph data, and streaming data). The book explores this new wave of systems, which it refers to as Big Data 2.0 processing systems. After Chapter 1 presents the general background of the big data phenomena, Chapter 2 provides an overview of various general-purpose big data processing systems that allow their users to develop various big data processing jobs for different application domains. In turn, Chapter 3 examines various systems that have been introduced to support the SQL flavor on top of the Hadoop infrastructure and provide competing and scalable performance in the processing of large-scale structured data. Chapter 4 discusses several systems that have been designed to tackle the problem of large-scale graph processing, while the main focus of Chapter 5 is on several systems that have been designed to provide scalable solutions for processing big data streams, and on other sets of systems that have been introduced to support the development of data pipelines between various types of big data processing jobs and systems. Lastly, Chapter 6 shares conclusions and an outlook on future research challenges. Overall, the book offers a valuable reference guide for students, researchers and professionals in the domain of big data processing systems. Further, its comprehensive content will hopefully encourage readers to pursue further research on the subject This book provides readers the zbig picturey and a comprehensive survey of the domain of big data processing systems. For the past decade, the Hadoop framework has dominated the world of big data processing, yet recently academia and industry have started to recognize its limitations in several application domains and big data processing scenarios such as the large-scale processing of structured data, graph data and streaming data. Thus, it is now gradually being replaced by a collection of engines that are dedicated to specific verticals (e.g. structured data, graph data, and streaming data). The book explores this new wave of systems, which it refers to as Big Data 2.0 processing systems. After Chapter 1 presents the general background of the big data phenomena, Chapter 2 provides an overview of various general-purpose big data processing systems that allow their users to develop various big data processing jobs for different application domains. In turn, Chapter 3 examines various systems that have been introduced to support the SQL flavor on top of the Hadoop infrastructure and provide competing and scalable performance in the processing of large-scale structured data. Chapter 4 discusses several systems that have been designed to tackle the problem of large-scale graph processing, while the main focus of Chapter 5 is on several systems that have been designed to provide scalable solutions for processing big data streams, and on other sets of systems that have been introduced to support the development of data pipelines between various types of big data processing jobs and systems. Lastly, Chapter 6 shares conclusions and an outlook on future research challenges. Overall, the book offers a valuable reference guide for students, researchers and professionals in the domain of big data processing systems. Further, its comprehensive content will hopefully encourage readers to pursue further research on the subject Foreword 7 Preface 9 Acknowledgements 11 Contents 12 About the Author 14 1 Introduction 15 1.1 The Big Data Phenomenon 15 1.2 Big Data and Cloud Computing 17 1.3 Big Data Storage Systems 19 1.4 Big Data Processing and Analytics Systems 22 1.5 Book Roadmap 25 2 General-Purpose Big Data Processing Systems 28 2.1 The Big Data Star: The Hadoop Framework 28 2.1.1 The Original Architecture 28 2.1.2 Enhancements of the MapReduce Framework 32 2.1.3 Hadoop's Ecosystem 40 2.2 Spark 41 2.3 Flink 46 2.4 Hyracks/ASTERIX 49 3 Large-Scale Processing Systems of Structured Data 53 3.1 Why SQL-On-Hadoop? 53 3.2 Hive 54 3.3 Impala 56 3.4 IBM Big SQL 57 3.5 SPARK SQL 58 3.6 HadoopDB 59 3.7 Presto 60 3.8 Tajo 62 3.9 Google Big Query 62 3.10 Phoenix 63 3.11 Polybase 63 4 Large-Scale Graph Processing Systems 65 4.1 The Challenges of Big Graphs 65 4.2 Does Hadoop Work Well for Big Graphs? 66 4.3 Pregel Family of Systems 70 4.3.1 The Original Architecture 70 4.3.2 Giraph: BSP + Hadoop for Graph Processing 73 4.3.3 Pregel Extensions 75 4.4 GraphLab Family of Systems 78 4.4.1 GraphLab 78 4.4.2 PowerGraph 78 4.4.3 GraphChi 80 4.5 Other Systems 80 4.6 Large-Scale RDF Processing Systems 83 5 Large-Scale Stream Processing Systems 86 5.1 The Big Data Streaming Problem 86 5.2 Hadoop for Big Streams?! 87 5.3 Storm 90 5.4 Infosphere Streams 92 5.5 Other Big Stream Processing Systems 93 5.6 Big Data Pipelining Frameworks 95 5.6.1 Pig Latin 95 5.6.2 Tez 97 5.6.3 Other Pipelining Systems 99 6 Conclusions and Outlook 101 References 106 Front Matter....Pages i-xv Introduction....Pages 1-13 General-Purpose Big Data Processing Systems....Pages 15-39 Large-Scale Processing Systems of Structured Data....Pages 41-52 Large-Scale Graph Processing Systems....Pages 53-73 Large-Scale Stream Processing Systems....Pages 75-89 Conclusions and Outlook....Pages 91-95 Back Matter....Pages 97-102 Chapter 1: Introduction.- Chapter 2: General Purpose Big Data Processing Systems.- Chapter 3: Large Scale Processing of Structured Databases.- Chapter 4: Large Scale Graph Processing Systems.- Chapter 5: Large Scale Stream Processing Systems.- Chapter 6: Conclusions and Outlook.