Building Big Data Applications

Building Big Data Applications helps data managers and their organizations make the most of unstructured data with an existing data warehouse. It provides readers with what they need to know to make sense of how Big Data fits into the world of Data Warehousing. Readers will learn about infrastructure options and integration and come away with a solid understanding on how to leverage various architectures for integration. The book includes a wide range of use cases that will help data managers visualize reference architectures in the context of specific industries (healthcare, big oil, transportation, software, etc.). Explores various ways to leverage Big Data by effectively integrating it into the data warehouse Includes real-world case studies which clearly demonstrate Big Data technologies Provides insights on how to optimize current data warehouse infrastructure and integrate newer infrastructure matching data processing workloads and requirements Cover Building Big Data Applications Copyright Dedication Preface 1. Big Data introduction Big Data delivers business value Healthcare example Data-rich and information-poor Big Data and healthcare Potential solutions Building Big Data applications Big Data applications—processing data Critical factors for success Risks and pitfalls Additional reading 2. Infrastructure and technology Introduction Distributed data processing Big data processing requirements Technologies for big data processing MapReduce MapReduce programming model MapReduce Google architecture Hadoop History Hadoop core components HDFS HDFS architecture NameNode DataNode Image Journal Checkpoint HDFS startup Block allocation and storage HDFS client Replication and recovery NameNode and DataNode—communication and management Heartbeats CheckPointNode and BackupNode CheckPointNode BackupNode Filesystem snapshots MapReduce YARN—yet another resource negotiator YARN scalability YARN execution flow Comparison between MapReduce v1 and v2 SQL/MapReduce Zookeeper Zookeeper features Locks and processing Failure and recovery Pig Programming with Pig Latin Pig data types Running Pig programs Pig program flow Common Pig command HBASE HBASE architecture HBASE architecture implementation Hive Hive architecture Infrastructure Execution—how does Hive process queries? Hive data types Hive query language (HiveQL) Hive examples Chukwa Flume Oozie HCatalog Sqoop NoSQL CAP theorem Key–value pair—Voldemort Cassandra Data model A keyspace has configurable properties that are critical to understand Data partitioning Data sorting Consistency management Write consistency Read consistency Specifying client consistency levels Built-in consistency repair features Cassandra ring architecture Data placement Data partitioning Peer to peer—simple scalability Gossip protocol—node management Basho Riak The design features of document-oriented databases include the following: Graph databases Additional reading 3. Building big data applications Data storyboard 4. Scientific research applications and usage Accelerators Big data platform and application XRootD filesystem interface project Service for web-based analysis (SWAN) The result—Higgs Boson discovery 5. Pharmacy industry applications and usage The complexity design for data applications Complexities in transformation of data Google deep mind Case study Additional reading 6. Visualization, storyboarding and applications Let us look at some of the use cases of big data applications Visualization The evolving role of the data scientist 7. Banking industry applications and usage The coming of age with uber banking Business challenges facing banks today Positively impacting the banking experience requires data, which is available and we need to orchestrate models of usage an ... The customer journey and the continuum Create modern data applications The use cases of analytics and big data applications in banking today Fraud and compliance tracking Client chatbots for call center Antimoney laundering detection Algorithmic trading Recommendation engines Predicting customer churn Additional reading 8. Travel and tourism industry applications and usage Travel and big data Real-time conversion optimization Optimized disruption management Niche targeting and unique selling propositions “Smart” social media listening and sentiment analysis Hospitality industry and big data Analytics and travel industry Examples of the use of predictive analytics Develop applications using data and agile API Additional reading 9. Governance Definition Metadata and master data Master data Data management in big data infrastructure Processing complexity of big data Processing limitations Governance model for building an application Use cases of governance Machine learning Additional reading 10. Building the big data application Risk assessment questions Information security policy Information security infrastructure Security of third-party access Asset classification and control Information classification Security in job definition and resourcing User training Responding to security/threat incidents Physical and environmental security Communications and operations management Media handling and security Exchange of information and software Business requirements for access control Mobile computing and telecommuting Business continuity management Aspects of business continuity management Additional reading 11. Data discovery and connectivity Challenges before you start with AI Strategies you can follow to start with AI Compliance and regulations Use cases from industry vendors Index A B C D E F G H I J L M N O P Q R S T U V W X Y Back Cover Helping data managers and their organizations make the most of unstructured data with an existing data warehouse, this book provides readers with what they need to know to make sense of how Big Data fits into the world of Data Warehousing. -- Edited summary from book