Network security through data analysis : from data to action

Traditional intrusion detection and logfile analysis are no longer enough to protect today’s complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. You’ll understand how your network is used, and what actions are necessary to harden and defend the systems within it. In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics. You’ll learn how to: • Use sensors to collect network, service, host, and active domain data • Work with the SiLK toolset, Python, and other tools and techniques for manipulating data you collect • Detect unusual phenomena through exploratory data analysis (EDA), using visualization and mathematical techniques • Analyze text data, traffic behavior, and communications mistakes • Identify significant structures in your network with graph analysis • Examine insider threat data and acquire threat intelligence • Map your network and identify significant hosts within it • Work with operations to develop defenses and analysis techniques Copyright 6 Table of Contents 7 Preface 15 Audience 17 Contents of This Book 18 Changes Between Editions 21 Conventions Used in This Book 22 Using Code Examples 22 O’Reilly Safari 23 How to Contact Us 23 Acknowledgments 24 Part I. Data 25 Chapter 1. Organizing Data: Vantage, Domain, Action, and Validity 27 Domain 29 Vantage 30 Choosing Vantage 32 Actions: What a Sensor Does with Data 33 Validity and Action 35 Internal Validity 37 External Validity 38 Construct Validity 39 Statistical Validity 39 Attacker and Attack Issues 40 Further Reading 40 Chapter 2. Vantage: Understanding Sensor Placement in Networks 43 The Basics of Network Layering 43 Network Layers and Vantage 46 Network Layers and Addressing 50 MAC Addresses 51 IPv4 Format and Addresses 52 IPv6 Format and Addresses 52 Validity Challenges from Middlebox Network Data 53 Further Reading 58 Chapter 3. Sensors in the Network Domain 59 Packet and Frame Formats 60 Rolling Buffers 60 Limiting the Data Captured from Each Packet 61 Filtering Specific Types of Packets 61 What If It’s Not Ethernet? 65 NetFlow 65 NetFlow v5 Formats and Fields 66 NetFlow Generation and Collection 68 Data Collection via IDS 68 Classifying IDSs 69 IDS as Classifier 70 Improving IDS Performance 74 Enhancing IDS Detection 75 Configuring Snort 76 Enhancing IDS Response 81 Prefetching Data 82 Middlebox Logs and Their Impact 83 VPN Logs 84 Proxy Logs 84 NAT Logs 85 Further Reading 85 Chapter 4. Data in the Service Domain 87 What and Why 87 Logfiles as the Basis for Service Data 89 Accessing and Manipulating Logfiles 89 The Contents of Logfiles 91 The Characteristics of a Good Log Message 91 Existing Logfiles and How to Manipulate Them 94 Stateful Logfiles 96 Further Reading 99 Chapter 5. Sensors in the Service Domain 101 Representative Logfile Formats 102 HTTP: CLF and ELF 102 Simple Mail Transfer Protocol (SMTP) 106 Sendmail 106 Microsoft Exchange: Message Tracking Logs 108 Additional Useful Logfiles 109 Staged Logging 109 LDAP and Directory Services 110 File Transfer, Storage, and Databases 110 Logfile Transport: Transfers, Syslog, and Message Queues 111 Transfer and Logfile Rotation 111 Syslog 111 Further Reading 113 Chapter 6. Data and Sensors in the Host Domain 115 A Host: From the Network’s View 116 The Network Interfaces 117 The Host: Tracking Identity 120 Processes 122 Structure 122 Filesystem 125 Historical Data: Commands and Logins 127 Other Data and Sensors: HIPS and AV 128 Further Reading 129 Chapter 7. Data and Sensors in the Active Domain 131 Discovery, Assessment, and Maintenance 131 Discovery: ping, traceroute, netcat, and Half of nmap 132 Checking Connectivity: Using ping to Connect to an Address 132 Tracerouting 134 Using nc as a Swiss Army Multitool 136 nmap Scanning for Discovery 137 Assessment: nmap, a Bunch of Clients, and a Lot of Repositories 139 Basic Assessment with nmap 139 Using Active Vantage Data for Verification 143 Further Reading 144 Part II. Tools 145 Chapter 8. Getting Data in One Place 147 High-Level Architecture 149 The Sensor Network 150 The Repository 151 Query Processing 153 Real-Time Processing 154 Source Control 154 Log Data and the CRUD Paradigm 155 A Brief Introduction to NoSQL Systems 157 Further Reading 160 Chapter 9. The SiLK Suite 161 What Is SiLK and How Does It Work? 161 Acquiring and Installing SiLK 162 The Datafiles 162 Choosing and Formatting Output Field Manipulation: rwcut 163 Basic Field Manipulation: rwfilter 168 Ports and Protocols 169 Size 170 IP Addresses 170 Time 172 TCP Options 172 Helper Options 174 Miscellaneous Filtering Options and Some Hacks 175 rwfileinfo and Provenance 176 Combining Information Flows: rwcount 178 rwset and IP Sets 181 rwuniq 185 rwbag 186 Advanced SiLK Facilities 187 PMAPs 187 Collecting SiLK Data 189 YAF 190 rwptoflow 192 rwtuc 193 rwrandomizeip 194 Further Reading 195 Chapter 10. Reference and Lookup: Tools for Figuring Out Who Someone Is 197 MAC and Hardware Addresses 198 IP Addressing 200 IPv4 Addresses, Their Structure, and Significant Addresses 200 IPv6 Addresses, Their Structure, and Significant Addresses 202 IP Intelligence: Geolocation and Demographics 204 DNS 205 DNS Name Structure 205 Forward DNS Querying Using dig 207 The DNS Reverse Lookup 215 Using whois to Find Ownership 216 DNS Blackhole Lists 219 Search Engines 221 General Search Engines 221 Scanning Repositories, Shodan et al 222 Further Reading 222 Part III. Analytics 223 An Overview of Attacker Behavior 223 Further Reading 226 Chapter 11. Exploratory Data Analysis and Visualization 227 The Goal of EDA: Applying Analysis 229 EDA Workflow 231 Variables and Visualization 232 Univariate Visualization 233 Histograms 234 Bar Plots (Not Pie Charts) 236 The Five-Number Summary and the Boxplot 236 Generating a Boxplot 238 Bivariate Description 239 Scatterplots 239 Multivariate Visualization 241 Other Visualizations and Their Role 242 Operationalizing Security Visualization 246 Fitting and Estimation 252 Is It Normal? 252 Simply Visualizing: Projected Values and QQ Plots 252 Fit Tests: K-S and S-W 255 Further Reading 257 Chapter 12. On Analyzing Text 259 Text Encoding 259 Unicode, UTF, and ASCII 262 Encoding for Attackers 263 Basic Skills 266 Finding a String 266 Manipulating Delimiters 267 Splitting Along Delimiters 267 Regular Expressions 268 Techniques for Text Analysis 271 N-Gram Analysis 271 Jaccard Distance 271 Hamming Distance 272 Levenshtein Distance 272 Entropy and Compressibility 274 Homoglyphs 275 Further Reading 276 Chapter 13. On Fumbling 277 Fumbling: Misconfiguration, Automation, and Scanning 277 Lookup Failures 278 Automation 278 Scanning 279 Identifying Fumbling 279 IP Fumbling: Dark Addresses and Spread 281 TCP Fumbling: Failed Sessions 283 ICMP Messages and Fumbling 288 Fumbling at the Service Level 289 HTTP Fumbling 289 SMTP Fumbling 291 DNS Fumbling 291 Detecting and Analyzing Fumbling 292 Building Fumbling Alarms 292 Forensic Analysis of Fumbling 294 Engineering a Network to Take Advantage of Fumbling 295 Chapter 14. On Volume and Time 297 The Workday and Its Impact on Network Traffic Volume 297 Beaconing 300 File Transfers/Raiding 303 Locality 306 DDoS, Flash Crowds, and Resource Exhaustion 309 DDoS and Routing Infrastructure 310 Applying Volume and Locality Analysis 316 Data Selection 316 Using Volume as an Alarm 319 Using Beaconing as an Alarm 319 Using Locality as an Alarm 319 Engineering Solutions 320 Further Reading 320 Chapter 15. On Graphs 323 Graph Attributes: What Is a Graph? 323 Labeling, Weight, and Paths 327 Components and Connectivity 332 Clustering Coefficient 333 Analyzing Graphs 335 Using Component Analysis as an Alarm 335 Using Centrality Analysis for Forensics 336 Using Breadth-First Searches Forensically 337 Using Centrality Analysis for Engineering 339 Further Reading 339 Chapter 16. On Insider Threat 341 Insider Threat Versus Other Classes of Attacks 342 Avoiding Toxicity 345 Modes of Attack 346 Data Theft and Exfiltration 346 Credential Theft 347 Sabotage 347 Insider Threat Data: Logistics and Collection 347 Applying Sector-Based Workflow to Insider Threat 348 Physical Data Sources 350 Keeping Track of User Identity 350 Further Reading 350 Chapter 17. On Threat Intelligence 353 Defining Threat Intelligence 353 Data Types 354 Creating a Threat Intelligence Program 357 Identifying Goals 357 Starting with Free Sources 359 Determining Data Output 359 Purchasing Sources 359 Brief Remarks on Creating Threat Intelligence 361 Further Reading 361 Chapter 18. Application Identification 363 Mechanisms for Application Identification 363 Port Number 364 Application Identification by Banner Grabbing 368 Application Identification by Behavior 371 Application Identification by Subsidiary Site 375 Application Banners: Identifying and Classifying 375 Non-Web Banners 375 Web Client Banners: The User-Agent String 376 Further Reading 378 Chapter 19. On Network Mapping 379 Creating an Initial Network Inventory and Map 379 Creating an Inventory: Data, Coverage, and Files 380 Phase I: The First Three Questions 382 Phase II: Examining the IP Space 384 Phase III: Identifying Blind and Confusing Traffic 389 Phase IV: Identifying Clients and Servers 392 Identifying Sensing and Blocking Infrastructure 395 Updating the Inventory: Toward Continuous Audit 395 Further Reading 396 Chapter 20. On Working with Ops 397 Ops Environments: An Overview 397 Operational Workflows 398 Escalation Workflow 399 Sector Workflow 401 Hunting Workflow 403 Hardening Workflow 404 Forensic Workflow 406 Switching Workflows 407 Further Readings 408 Chapter 21. Conclusions 409 Index 411 About the Author 426 Colophon 426 From Data to Action. Second edition. Traditional intrusion detection and logfile analysis are no longer enough to protect todays complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. Youll understand how your network is used, and what actions are necessary to harden and defend the systems within it.In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics.Youll learn how to:Use sensors to collect network, service, host, and active domain data Work with the SiLK toolset, Python, and other tools and techniques for manipulating data you collect Detect unusual phenomena through exploratory data analysis (EDA), using visualization and mathematical techniques Analyze text data, traffic behavior, and communications mistakes Identify significant structures in your network with graph analysis Examine insider threat data and acquire threat intelligence Map your network and identify significant hosts within it Work with operations to develop defenses and analysis techniques Traditional intrusion detection and logfile analysis are no longer enough to protect today’s complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. You’ll understand how your network is used, and what actions are necessary to harden and defend the systems within it. In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics. -- Provided by publisher