Guide to Big Data Applications (Studies in Big Data (26))

This handbook brings together a variety of approaches to the uses of big data in multiple fields, primarily science, medicine, and business. This single resource features contributions from researchers around the world from a variety of fields, where they share their findings and experience. This book is intended to help spur further innovation in big data. The research is presented in a way that allows readers, regardless of their field of study, to learn from how applications have proven successful and how similar applications could be used in their own field. Contributions stem from researchers in fields such as physics, biology, energy, healthcare, and business. The contributors also discuss important topics such as fraud detection, privacy implications, legal perspectives, and ethical handling of big data. Foreword......Page 7 Preface......Page 9 Acknowledgements......Page 12 Contents......Page 13 List of Reviewers......Page 15 Part I General......Page 16 1.1 Introduction......Page 17 1.1.3 Better Customer Relationships......Page 18 1.2 From Value Disciplines to Digital Disciplines......Page 19 1.2.2 Solution Leadership......Page 20 1.2.4 Accelerated Innovation......Page 21 1.3.1 Real-Time Process and Resource Optimization......Page 22 1.3.3 Digital-Physical Substitution and Fusion......Page 24 1.3.4 Exhaust-Data Monetization......Page 25 1.3.6 Beyond Business......Page 26 1.4.1 Digital-Physical Mirroring......Page 27 1.4.3 Product/Service Usage Optimization......Page 28 1.4.6 Long-Term Product Improvement......Page 29 1.4.9 Transformations......Page 30 1.5 Collective Intimacy......Page 31 1.5.3 Recommendations......Page 33 1.5.5 Beyond Business......Page 34 1.6 Accelerated Innovation......Page 35 1.6.2 Contest Economics......Page 36 1.6.4 Beyond Business......Page 37 1.7 Integrated Disciplines......Page 38 References......Page 39 2.1 Introduction......Page 42 2.2 Information Privacy Defined......Page 43 2.2.1 Is It Personally Identifiable Information?......Page 44 2.3 Big Data: Understanding the Challenges to Privacy......Page 46 2.3.1 Big Data: The Antithesis of Data Minimization......Page 48 2.3.2 Predictive Analysis: Correlation Versus Causation......Page 49 2.3.3 Lack of Transparency/Accountability......Page 50 2.4 Privacy by Design and the 7 Foundational Principles......Page 51 2.4.1 The 7 Foundational Principles......Page 53 2.5.1 Being Proactive About Privacy Through Prevention......Page 54 2.5.2 Data Minimization as the Default Through De-identification......Page 55 2.5.3 Embedding Privacy at the Design Stage......Page 56 2.5.4 Aspire for Positive-Sum Without Diminishing Functionality......Page 57 2.6 Conclusion......Page 58 References......Page 59 3.1 Introduction......Page 62 3.2.1.1 Server/Client Architecture......Page 66 3.2.1.2 Decentralized Architecture......Page 68 3.2.2.2 Alternating Direction Method of Multipliers......Page 69 3.3.1 Applications Based on the Newton-Raphson Method......Page 71 3.3.2 Applications Based on ADMM......Page 75 3.3.2.1 Regression......Page 76 3.3.2.2 Classification......Page 77 3.3.2.3 Convergence and Robustness for Decentralized Data Analysis......Page 78 3.4.1 Regression......Page 79 3.4.2 Classification......Page 81 3.4.3 Evaluation......Page 85 3.5 Asynchronous Optimization......Page 86 3.5.1 Asynchronous Optimization Based on Fixed-Point Algorithms......Page 87 3.5.3 Asynchronous Alternating Direction Method of Multipliers......Page 88 3.6 Discussion and Conclusion......Page 89 References......Page 91 4.1 Introduction......Page 96 4.2.1 Background......Page 98 4.2.2 Neural Network Language Model......Page 99 4.2.2.1 Restricted Boltzmann Machine (RBM)......Page 102 4.2.2.2 Recurrent and Recursive Neural Network......Page 103 4.2.2.3 Convolutional Neural Network (CNN)......Page 106 4.2.2.4 Hierarchical Neural Language Model (HNLM)......Page 107 4.2.3 Sparse Coding Approach......Page 108 4.2.4 Evaluations of Word Embedding......Page 110 4.3 Word Embedding Applications......Page 111 4.4 Conclusion and Future Work......Page 113 References......Page 114 Part II Applications in Science......Page 118 5.1 Introduction......Page 119 5.2.1 Sample Applications......Page 121 5.3.1 RapidMiner......Page 123 5.4.1 Background......Page 124 5.4.2 Dataset......Page 125 5.4.3 Data Analysis......Page 126 5.4.4 Summary......Page 127 5.5 Case Study II: Big Data in Environmental Microbiology......Page 129 5.5.2 Genome Dataset......Page 130 5.5.3 Analysis of Big Data......Page 131 5.6 Discussion and Future Work......Page 132 References......Page 134 6.1 Introduction......Page 137 6.2.1.2 Ingestion......Page 138 6.2.1.4 Batch Event Processing......Page 139 6.3 High-Performance and Big Data Deployment Types......Page 140 6.3.2 Cloud-Based Hardware Deployment......Page 141 6.3.5 Summary......Page 142 6.4.1.1 Data Ingestion Stacks......Page 143 6.4.1.4 Indexing and Querying Engine......Page 145 6.4.2.1 Data Ingestion Cluster......Page 146 6.4.2.3 Data Stores......Page 148 6.4.2.5 Batch-Processing Frameworks......Page 149 6.4.2.6 Interoperability Between Frameworks......Page 150 6.4.2.7 Summary......Page 151 6.4.4.1 Software Defined Infrastructure (SDI)......Page 152 6.4.4.3 Intelligent Software for Performance Management......Page 154 6.5 Designing Data Pipelines for High Performance......Page 155 6.6 Conclusions......Page 156 References......Page 157 7.1 Introduction......Page 160 7.2 Improve Classification Accuracy of Bayesian Inversion Through Big Data Learning......Page 161 7.2.1 Bayesian Inversion and Measurement Errors......Page 162 7.2.2 Bayesian Graphical Model......Page 163 7.2.3 Statistical Inference for the Gaussian Mixture Model......Page 166 7.2.3.1 Distributed Markov Chain Monte Carlo for Big Data......Page 168 7.2.4 Tests from Synthetic Well Integrity Logging Data......Page 169 7.3 Proactive Geosteering and Formation Evaluation......Page 171 7.3.1.1 The Deterministic Inversion Method......Page 173 7.3.1.2 The Statistical Inversion Method......Page 175 7.3.2 Hamiltonian Monte Carlo and MapReduce......Page 177 7.3.3.1 Tests from Synthetic Data......Page 178 7.3.3.2 Test from Field Data......Page 180 References......Page 182 8.1 Introduction......Page 185 8.2 The Value of Big Data for the Petroleum Industry......Page 186 8.2.1.1 SaaS (Software as a Service)......Page 187 8.2.1.3 Reduced IT Support......Page 188 8.2.2 Collaboration......Page 189 8.2.2.1 Ideal Ecosystem......Page 190 8.2.2.2 Collaboration Enhancement by Universal Cloud-based Database Organization......Page 191 8.2.2.4 Collaboration Enhancement by Low Cost Subscription Model......Page 192 8.2.3.1 Workflow Variety......Page 193 8.3.1 Big Data......Page 194 8.3.3 Browser......Page 195 8.4.1 Past Steps......Page 196 8.4.2.1 What Is GeoFit?......Page 197 8.4.2.2 Why Use a Universal Cloud-Based Hybrid Database?......Page 198 8.4.2.3 Workflow Engine......Page 200 8.4.2.5 Viewing......Page 201 8.4.3.1 Project Structure......Page 203 8.5 Future of Tools, Example......Page 204 8.6.1.1 Planning Storage......Page 206 8.6.1.2 Planning CPU Capacity......Page 207 8.6.1.3 Planning Memory Requirements......Page 208 8.6.2 Eventual Consistency......Page 209 8.6.3 Fault Tolerance......Page 210 8.7 Big Data is the future of O&G......Page 212 References......Page 213 9.1 Introduction......Page 215 9.1.1 Organization of the Chapter and Summary of Results......Page 218 9.2 A Brief Review of the Statistics of Friendship Paradoxes: What are Strong Paradoxes, and Why Should We Measure Them?......Page 219 9.2.2 What Does Feld's Argument Imply?......Page 220 9.2.4 Beyond Random-Wiring Assumptions: Why Weak and Strong Friendship Paradoxes are Ubiquitous in Undirected Networks......Page 222 9.2.6 Strong Degree-Based Paradoxes in Directed Networks and Strong Generalized Paradoxes are Nontrivial......Page 223 9.3.1 Definition of the Network and Core Questions......Page 224 9.3.2 All Four Degree-Based Paradoxes Occur in the Quora Follow Network......Page 225 9.3.3 Anatomy of a Strong Degree-Based Paradox in Directed Networks......Page 227 9.3.4 Summary and Implications......Page 229 9.4.1 What are Upvotes and Downvotes?......Page 230 9.4.2 The Downvoting Network and the Core Questions......Page 231 9.4.3 The Downvoting Paradox is Absent in the Full Downvoting Network......Page 232 9.4.4 The Downvoting Paradox Occurs When The Downvotee and Downvoter are Active Contributors......Page 235 9.4.5 Does a ``Content-Contribution Paradox'' Explain the Downvoting Paradox?......Page 238 9.4.6 Summary and Implications......Page 241 9.5.1 Content Dynamics in the Follow Network......Page 242 9.5.2 Core Questions and Methodology......Page 245 9.5.3 Demonstration of the Existence of the Paradox......Page 247 9.5.4 Can We Measure the Impact of the Paradox?......Page 249 9.5.5 Summary and Implications......Page 250 9.6 Conclusion......Page 251 References......Page 253 10.1 Context and Motivation......Page 255 10.2.2 Deduplication Level......Page 256 10.2.4 Client- vs Server-Side Deduplication......Page 257 10.4 Secure Image Deduplication Through Image Compression......Page 258 10.6 Proposed Image Deduplication Scheme......Page 259 10.6.1 Image Compression......Page 260 10.6.2 Partial Encryption of the Compressed Image......Page 261 10.6.3 Image Hashing from the Compressed Image......Page 262 10.6.4.1 Experimental Settings......Page 263 10.6.4.2 Deduplication Analysis......Page 264 10.6.4.3 Performance Analysis......Page 267 10.6.5 Security Analysis......Page 268 10.7 Secure Video Deduplication Scheme in Cloud Storage Environment Using H.264 Compression......Page 269 10.9 Proposed Video Deduplication Scheme......Page 270 10.9.1 H.264 Video Compression......Page 271 10.9.2 Signature Generation from the Compressed Videos......Page 272 10.9.3 Selective Encryption of the Compressed Videos......Page 273 10.9.4 Experimental Results......Page 275 10.9.5 Security Analysis......Page 278 10.10 Chapter Summary......Page 279 References......Page 280 11.1 Introduction......Page 282 11.2 Searchable Encryption Models......Page 283 11.3 Text......Page 284 11.3.1.1 Encrypted Indexes......Page 285 11.3.1.2 Bloom Filters......Page 287 11.3.2 Private/Public Search Scheme: Cloud Document Storage Extended......Page 288 11.3.3 Public/Private Search Scheme: Email Filtering System......Page 289 11.3.3.1 Identity-Based Encryption (IBE)......Page 290 11.3.3.2 An IBE-Based Secure Email Filtering System......Page 291 11.3.4 Public/Public Search Scheme: Delegated Investigation of Secured Audit Logs......Page 292 11.3.4.1 Symmetric Scheme......Page 293 11.3.4.2 Asymmetric Scheme......Page 294 11.4.1 Order Preserving Encryption......Page 295 11.5.1 Keyword Based Media Search......Page 296 11.5.2 Content Based Media Search......Page 297 11.5.2.1 Media Search Using OPE......Page 298 11.5.2.2 Homomorphic Encryption......Page 299 11.6 Other Applications......Page 300 11.7 Conclusions......Page 301 References......Page 302 12.1.1 Motivations of Big-Data Enabled Structural Health Monitoring......Page 303 12.1.2 Overview of the Proposed MS-SHM-Hadoop......Page 305 12.1.3 Organization of This Chapter......Page 306 12.2.1 Infrastructure of MS-SHM-Hadoop......Page 307 12.2.2 Flowchart of the MS-SHM-Hadoop......Page 309 12.3 Acquisition of Sensory Data and Integration of Structure-Related Data......Page 310 12.4.1 The Features Used in Nationwide Civil Infrastructure Survey......Page 312 12.4.2.1 Overview of Machine-Learning-Enabled Life-Expectancy Estimation......Page 315 12.4.2.2 Weibull Linear Regression Model......Page 316 12.4.2.5 Deep Learning Models......Page 317 12.4.3.3 Dimensionality Reduction......Page 319 12.5 Global Structural Integrity Analysis......Page 320 12.5.2.1 Data Query for the Measured Global Characteristics of the Targeted Civil Infrastructure......Page 322 12.5.2.3 Assessment of the Integrity Level of Civil Infrastructure......Page 324 12.6.1 Deep-Learning-Enabled Component Reliability Analysis......Page 325 12.6.2 Probe Prolongation Strategies via Simulating Crack Initialization and Growth......Page 327 12.7 Civil Infrastructure's Reliability Analysis Based on Bayesian Network......Page 328 References......Page 329 Part III Applications in Medicine......Page 334 13.1 Introduction......Page 335 13.2.1 Epilepsy......Page 338 13.2.2 Parallel Computing......Page 341 13.2.3 Challenges......Page 342 13.2.4 Current State of Art......Page 343 13.3 Nonlinear Dynamical Systems with Chaos......Page 344 13.4 Lyapunov Exponents......Page 349 13.4.1 Numerical Computation of Lyapunov Exponents......Page 350 13.4.1.1 The Wolf Algorithm......Page 351 13.4.1.2 The Rosenstein and Kantz Algorithm......Page 353 13.5 Rapid Prototyping HPCmatlab Platform......Page 354 13.5.1 Bigdata and Matlab......Page 355 13.5.2 Parallel Computing and HPCmatlab......Page 356 13.5.2.1 Parallel Computation of Lyapunov Exponents......Page 357 13.5.2.2 Epilepsy as a Big Data Problem......Page 359 13.6 Case Study: Epileptic Seizure Prediction and Control......Page 360 13.7 Future Research Opportunities and Conclusions......Page 367 Appendix 1: Electrical Stimulation and Experiment Setup......Page 369 Appendix 2: Preparation of Animals......Page 370 References......Page 371 14.1 Introduction......Page 376 14.2 Capture Reliably......Page 379 14.2.2 Data Sparsity......Page 380 14.2.3 Feature Selection......Page 381 14.2.4 State-of-the Art and Novel Algorithms......Page 383 14.2.5 Physiological Precision and Stratified Medicine......Page 384 14.2.6 The Green Button: A Case Study on Capture Reliably Principle of the CAPE Roadmap......Page 385 14.3.1 Networks Medicine......Page 386 14.3.3 Agent Based Models......Page 388 14.3.4 Prevalence of Symptoms on a Single Indian Healthcare Day on a Nationwide Scale (POSEIDON): A Case Study on Approach Systemically Principal of the CAPE Roadmap......Page 389 14.4 Phenotype Deeply*-4pt......Page 391 14.4.2 Heart Rate Variability: A Case Study on Phenotype Deeply Principle of the CAPE Roadmap......Page 392 14.5 Enable Decisions......Page 397 14.5.1 Causal Modeling Through Bayesian Networks......Page 398 14.5.4 SAFE-ICU Initiative: A Full Spectrum Case Study in Biomedical Data-Science......Page 399 References......Page 401 15.1 Introduction......Page 405 15.2.1 Data Description and Preparation......Page 406 15.2.2 Data Preparation Process for Sequence Analysis......Page 408 15.3 Results......Page 409 15.3.1 Top 20 Diseases in TUD and Non-TUD Patients......Page 411 15.3.2 Top 20 Diseases in TUD Patients and Corresponding Prevalence in Non-TUD Patients......Page 413 15.3.3 Top 15 Comorbidities in TUD Patients Across Two Hospital Visits (Second Iteration) and Corresponding Prevalence in Non-TUD Patients......Page 414 15.3.4 Comorbidities in TUD Patients Across Three Hospital Visits (Third Iteration) and Comparison with Non-TUD Patients......Page 415 15.4 Discussion and Concluding Remarks......Page 416 References......Page 417 16 The Impact of Big Data on the Physician......Page 418 16.1.1 Defining Quality Care......Page 419 16.1.2 Choosing the Best Doctor......Page 420 16.1.3 Choosing the Best Hospital......Page 421 16.1.5 What is mHealth?......Page 423 16.1.6 mHealth from the Provider Side......Page 424 16.1.7 mHealth from the Patient Side......Page 425 16.1.9 Accessibility......Page 428 16.1.10 Privacy and Security......Page 429 16.1.11 Regulation and Liability......Page 430 16.1.12 Patient Education and Partnering with Patients......Page 431 16.1.13 Developing Online Health Communities Through Social Media: Creating Data that Fuels Research......Page 432 16.1.15 Beyond the Package Insert: Iodine.com......Page 434 16.1.16 Data Inspires the Advent of New Models of Medical Care Delivery......Page 436 16.2.1.1 Imagine the Following Scenario......Page 437 16.2.2 Probabilistic Systems......Page 439 16.2.4 Data Driven Approaches......Page 440 16.2.5 Challenges and Areas of Exploration......Page 442 16.2.6 Using Big Data to Improve Treatment Options......Page 443 References......Page 447 Part IV Applications in Business......Page 452 17.1 Introduction......Page 453 17.2.1 The Nature of Bank Information Activities......Page 454 17.2.1.1 The Corporate Culture of Banks......Page 457 17.2.2 Analytical Information Needs in Banks......Page 458 17.3.1 Tradeoffs of Big Data Analytics......Page 459 17.3.2 Data Collection and Integration......Page 460 17.3.3 Data Quality Challenges......Page 462 17.3.4 Discovery and Detection......Page 463 17.3.4.1 Risks......Page 465 17.3.4.2 Customers......Page 466 17.3.5 Big Data Approaches in Banking Supervision......Page 468 17.4 Big Data Analysis Methods and Instruments in Banking......Page 471 17.4.1 Big Data Processing Methods: Data Mining, Text Mining, Machine Learning, Visualization......Page 472 17.4.1.1 Visualization......Page 474 17.4.2 Analytical Platforms......Page 475 17.4.3 Examples of Fraud Detection......Page 478 17.5 Managerial Implications......Page 484 References......Page 486 18.1 Introduction......Page 489 18.2 Multi-Touch Attribution (MTA)......Page 490 18.3 Granular Audience Targeting......Page 492 18.4 Forecasting......Page 495 18.5 Predictive Analytics......Page 498 18.6 Content Marketing......Page 500 18.7 Weaving Big Data in Applications......Page 502 18.8 Summary......Page 503 References......Page 504 19.2.1 The Rise of Social Media Data......Page 505 19.2.2 The Quick Service Restaurant Sector......Page 506 19.3 Analysis of Numeric and Text Reviews in Yelp......Page 507 19.3.1 Description of Numeric Ratings......Page 508 19.3.2 Comparison Between Non-Franchise and Franchise Locations......Page 509 19.3.4 Analysis of All U.S. Locations......Page 513 19.3.5 Description of Text Reviews......Page 514 19.3.6 Caution Related to Analysis of Reviews......Page 516 19.4 Guide to Using R to Extract Yelp Data......Page 517 References......Page 523 Author Biographies......Page 525 Index......Page 555