The Metrics Manifesto : Confronting Security with Data

Security professionals are trained skeptics. They poke and prod at other people’s digital creations, expecting them to fail in unexpected ways. Shouldn’t that same skeptical power be turned inward? Shouldn’t practitioners ask: “How do I know that my enterprise security capabilities work? Are they scaling, accelerating, or slowing as the business exposes more value to more people and through more channels at higher velocities?” This is the start of the modern measurement mindset--the mindset that seeks to confront security with data. The Metrics Manifesto: Confronting Security with Data delivers an examination of security metrics with R, the popular open-source programming language and software development environment for statistical computing. This insightful and up-to-date guide offers readers a practical focus on applied measurement that can prove or disprove the efficacy of information security measures taken by a firm. The book’s detailed chapters combine topics like security, predictive analytics, and R programming to present an authoritative and innovative approach to security metrics. The author and security professional examines historical and modern methods of measurement with a particular emphasis on Bayesian Data Analysis to shed light on measuring security operations. Readers will learn how processing data with R can help measure security improvements and changes as well as help technology security teams identify and fix gaps in security. The book also includes downloadable code for people who are new to the R programming language. Perfect for security engineers, risk engineers, IT security managers, CISOs, and data scientists comfortable with a bit of code, The Metrics Manifesto offers readers an invaluable collection of information to help professionals prove the efficacy of security measures within their company. Cover 1 Title Page 5 Copyright Page 6 Contents 9 Foreword 11 Preface: How This Book Came to Be 13 About the Technical Review Team 15 Robert D. Brown III 15 Anuj Gargeya Malkapuram 15 Kaela Seiersen 16 Chapter 1 Introduction: The Manifesto and the BOOM! Framework 17 What’s Next: Caveats and Epiphanies 18 The Metrics Manifesto and BOOM! 20 The (Modern) Metrics Manifesto 20 BOOM: Baseline Objectives and Optimization Measurements 21 Bullet Holes and Bombers 22 BOOM Defined 22 Survival Analysis 23 Administrivia and Sundry Items 28 Notes 29 Chapter 2 Time to Event Metrics 31 Threat Hunting with Dr. Snow 32 From Cholera to Security 33 Life Tables 34 Life Table Structure 36 Making a Life Table in R 38 Code Explained in Detail 40 The Survival Functions 42 Life Table Detail 43 Basic Life Table Metrics 45 Conclusion 48 Notes 48 Chapter 3 Counting on Uncertainty: Preparing for Burndown, Arrival, Wait-Times, and Escape Rates 49 Gamblers, Scientists, and a Theologian 49 The Persistence and Dominance of Bayes 50 A Bayesian Primer 51 Metrics Example: Phishing for Improvement 53 From ABC to Canonical Bayes 63 Notes 68 Chapter 4 Burndown Rates: Shifting Right the Bayesian Way 69 The Day 1 Metric 72 Get Small Data 73 Graph the Updated Model 79 Precision and Accuracy: Burndown and Updating over Time 85 Building a Bayesian Burndown Chart 86 Final: Comparing Teams 93 Wrapping Up 94 Notes 95 Chapter 5 Risk Arrival Rates: Shift Left Security Metrics 97 Introduction: Random Bombs and Horse Kicks 97 From Burndown to Arrival 101 Simulating Arrivals 102 Arrival Prediction 107 Bayes Meets Arrival Rates 109 Building a Simple Bayesian Arrival Model 111 Advanced Prediction 120 Wrapping Up Arrivals 125 Notes 125 Chapter 6 Wait-Time Rates: Between Arrival and Departure Is...Waiting 127 Bayesian Wait-Times 136 Mitigatable Surprise 145 NVD Analysis Decomposed 149 Step 1: Download Data 150 Step 2: Get Priors 151 Step 3: Get Rates (from Data) 152 Steps 4 and 5: Simulate Data and Make Graphs 153 Summary 159 Notes 159 Chapter 7 Escape Rates 161 What Is an Escape Rate? 161 Naive Escape Rates 162 Quick Bayesian Recap 165 Coding up Bayesian Escape Rates 165 Functional Decomposition 172 Escape Rates in 10 Lines of Code 176 Chapter Summary 177 Notes 178 Chapter 8 Optimization Basics with Bayesian Linear Regression 179 Grid Approximation 181 Introducing Grid Approximation 181 Steps Toward Optimization: Using MCMC-Based Regression 187 Markov Chain Monte Carlo (MCMC) Conceptual Primer 188 A Brief Introduction to Regression Analysis 194 MCMC Posterior Quality 201 Posterior Policy Prediction 208 Final Chapter Thoughts 211 Notes 211 Chapter 9 ABC A/B Testing and Security ROI 213 Get Better ROI for Security 213 Buying Security with Predictive Analytics 213 The Use Case: Web Application and API Scanning 214 Step 1: Model Your Beliefs 215 Step 2: Mash up Data with Beliefs 216 Step 3: Forecast the Financial Impact of Errors 220 Code Details and Design 224 Conclusion 237 Notes 237 Chapter 10 Dashboarding with BOOM! 239 BOOM Metrics Objects 240 Default Mode (Rate Mode) 240 KPI Mode 241 Mixed Mode 243 Child Metrics Objects 243 Count Metrics 243 Scaling, Accelerating, Slowing 244 Survival Analysis 245 Subsetting by Date 246 Data Upload 247 KPI Analysis – Scoring the Scores 247 KPI Sliders 249 Making Shiny Dashboards 251 Shiny Code 253 Shiny Server Code 255 Conclusion 259 Notes 259 Chapter 11 Simulating Data Like a Pro 261 Introduction 261 Warming Up to Simulation 262 Why Simulate First? 263 Security Data Types and Dimensions 264 Let’s Make Some Data: Part 1 266 Let’s Code It! 267 Ask the Expert 269 Let’s Make Some Data: Part 2 272 Modeling Vulnerability Management 273 The Team Data Structure 274 Enriching and Tidying 279 Notes 295 Epilogue: A Short One-for-One Substitution Guide 297 BOOM for CIS Metrics 297 Focusing on Outcomes 305 Next Steps 307 Notes 308 Index 309 EULA 317 Provides predictive security metrics with R--security, analytics, and programming Massive data breaches and discussions surrounding improving technology security have been topics of intense interest over the past several years. Security failures by organizations such as Equifax, Uber, the U.S Securities and Exchange Commission, and the Republican National Committee, amongst many others, impacted millions of Americans. There is no disputing the importance of effective cybersecurity technologies and practices, yet measuring security effectiveness within corporations and other entities has proved to be a challenge. The Metrics Manifesto examines security metrics with R, the popular open-source programming language and software development environment for statistical computing. This timely, fully up-to-date guide focuses on applied measurement that proves or disproves information security effectiveness. Comprehensive, detailed chapters discuss security, predictive analytics, and programming with R. Author Richard Seiersen presents an innovative approach to security metrics, looking to fields such as the sciences and professional sports to improve measurement. A valuable tool for discovering how to improve IT security procedures, this important book: Uncovers the truths about an organization's security programs Explains how processing data with R can measure security improvements Helps technology security teams identify and rectify security weaknesses Offer practical insights from a leading security expert with two decade's experience in information security, risk management, and product development Includes a downloadable applied tutorial new R users The Metrics Manifesto: Confronting Security with Data is an essential resource for IT security managers, risk managers, statisticians, and other security professionals. "This book is predictive security metrics with R. It is a quantitative shift in security strategy and tactics that looks to the sciences, professional sports, and others for cues on measuring improvement (security, predictive analytics, and programming in R). The ultimate goal of this book is to show the truths about a corporation's security programs. This is done by confronting the program with data. That means the data coming from the program should unambiguously prove whether or not the technology team is improving the program. This book will be tool to be used in discovery how to improve IT security procedures"-- Provided by publisher