Network security through data analysis : building situational awareness

Copyright 4 Table of Contents 5 Preface 11 Audience 13 Contents of This Book 14 Conventions Used in This Book 16 Using Code Examples 17 Safari庐 Books Online 17 How to Contact Us 18 Acknowledgements 18 Part聽I.聽Data 21 Chapter聽1.聽Sensors and Detectors: An Introduction 23 Vantages: How Sensor Placement Affects Data Collection 24 Domains: Determining Data That Can Be Collected 27 Actions: What a Sensor Does with Data 30 Conclusion 33 Chapter聽2.聽Network Sensors 35 Network Layering and Its Impact on Instrumentation 36 Network Layers and Vantage 38 Network Layers and Addressing 43 Packet Data 44 Packet and Frame Formats 44 Rolling Buffers 45 Limiting the Data Captured from Each Packet 45 Filtering Specific Types of Packets 45 What If It鈥檚 Not Ethernet? 49 NetFlow 50 NetFlow v5 Formats and Fields 50 NetFlow Generation and Collection 52 Further Reading 53 Chapter聽3.聽Host and Service Sensors: Logging Traffic at the Source 55 Accessing and Manipulating Logfiles 56 The Contents of Logfiles 58 The Characteristics of a Good Log Message 58 Existing Logfiles and How to Manipulate Them 61 Representative Logfile Formats 63 HTTP: CLF and ELF 63 SMTP 67 Microsoft Exchange: Message Tracking Logs 69 Logfile Transport: Transfers, Syslog, and Message Queues 70 Transfer and Logfile Rotation 71 Syslog 71 Further Reading 73 Chapter聽4.聽Data Storage for Analysis: Relational Databases, Big Data, and Other Options 75 Log Data and the CRUD Paradigm 76 Creating a Well-Organized Flat File System: Lessons from SiLK 77 A Brief Introduction to NoSQL Systems 79 What Storage Approach to Use 82 Storage Hierarchy, Query Times, and Aging 84 Part聽II.聽Tools 87 Chapter聽5.聽The SiLK Suite 89 What Is SiLK and How Does It Work? 89 Acquiring and Installing SiLK 90 The Datafiles 90 Choosing and Formatting Output Field Manipulation: rwcut 91 Basic Field Manipulation: rwfilter 96 Ports and Protocols 97 Size 98 IP Addresses 98 Time 100 TCP Options 100 Helper Options 102 Miscellaneous Filtering Options and Some Hacks 102 rwfileinfo and Provenance 103 Combining Information Flows: rwcount 106 rwset and IP Sets 108 rwuniq 111 rwbag 113 Advanced SiLK Facilities 113 pmaps 113 Collecting SiLK Data 115 YAF 116 rwptoflow 118 rwtuc 118 Further Reading 120 Chapter聽6.聽An Introduction to R for Security Analysts 121 Installation and Setup 122 Basics of the Language 122 The R Prompt 122 R Variables 124 Writing Functions 129 Conditionals and Iteration 131 Using the R Workspace 133 Data Frames 134 Visualization 137 Visualization Commands 137 Parameters to Visualization 138 Annotating a Visualization 140 Exporting Visualization 141 Analysis: Statistical Hypothesis Testing 141 Hypothesis Testing 142 Testing Data 144 Further Reading 147 Chapter聽7.聽Classification and Event Tools: IDS, AV, and SEM 149 How an IDS Works 150 Basic Vocabulary 150 Classifier Failure Rates: Understanding the Base-Rate Fallacy 154 Applying Classification 156 Improving IDS Performance 158 Enhancing IDS Detection 158 Enhancing IDS Response 163 Prefetching Data 164 Further Reading 165 Chapter聽8.聽Reference and Lookup: Tools for Figuring Out Who Someone Is 167 MAC and Hardware Addresses 167 IP Addressing 170 IPv4 Addresses, Their Structure, and Significant Addresses 170 IPv6 Addresses, Their Structure and Significant Addresses 172 Checking Connectivity: Using ping to Connect to an Address 173 Tracerouting 175 IP Intelligence: Geolocation and Demographics 177 DNS 178 DNS Name Structure 178 Forward DNS Querying Using dig 179 The DNS Reverse Lookup 187 Using whois to Find Ownership 188 Additional Reference Tools 191 DNSBLs 191 Chapter聽9.聽More Tools 195 Visualization 195 Graphviz 195 Communications and Probing 198 netcat 199 nmap 200 Scapy 201 Packet Inspection and Reference 204 Wireshark 204 GeoIP 205 The NVD, Malware Sites, and the C*Es 206 Search Engines, Mailing Lists, and People 207 Further Reading 208 Part聽III.聽Analytics 209 Chapter聽10.聽Exploratory Data Analysis and Visualization 211 The Goal of EDA: Applying Analysis 213 EDA Workflow 214 Variables and Visualization 216 Univariate Visualization: Histograms, QQ Plots, Boxplots, and Rank Plots 217 Histograms 218 Bar Plots (Not Pie Charts) 220 The Quantile-Quantile (QQ) Plot 221 The Five-Number Summary and the Boxplot 223 Generating a Boxplot 224 Bivariate Description 227 Scatterplots 227 Contingency Tables 230 Multivariate Visualization 231 Operationalizing Security Visualization 233 Further Reading 240 Chapter聽11.聽On Fumbling 241 Attack Models 241 Fumbling: Misconfiguration, Automation, and Scanning 244 Lookup Failures 244 Automation 245 Scanning 245 Identifying Fumbling 246 TCP Fumbling: The State Machine 246 ICMP Messages and Fumbling 249 Identifying UDP Fumbling 251 Fumbling at the Service Level 251 HTTP Fumbling 251 SMTP Fumbling 253 Analyzing Fumbling 253 Building Fumbling Alarms 254 Forensic Analysis of Fumbling 255 Engineering a Network to Take Advantage of Fumbling 256 Further Reading 256 Chapter聽12.聽Volume and Time Analysis 257 The Workday and Its Impact on Network Traffic Volume 257 Beaconing 260 File Transfers/Raiding 263 Locality 266 DDoS, Flash Crowds, and Resource Exhaustion 269 DDoS and Routing Infrastructure 270 Applying Volume and Locality Analysis 276 Data Selection 276 Using Volume as an Alarm 278 Using Beaconing as an Alarm 279 Using Locality as an Alarm 279 Engineering Solutions 280 Further Reading 280 Chapter聽13.聽Graph Analysis 281 Graph Attributes: What Is a Graph? 281 Labeling, Weight, and Paths 285 Components and Connectivity 290 Clustering Coefficient 291 Analyzing Graphs 293 Using Component Analysis as an Alarm 293 Using Centrality Analysis for Forensics 295 Using Breadth-First Searches Forensically 295 Using Centrality Analysis for Engineering 297 Further Reading 297 Chapter聽14.聽Application Identification 299 Mechanisms for Application Identification 299 Port Number 300 Application Identification by Banner Grabbing 303 Application Identification by Behavior 306 Application Identification by Subsidiary Site 310 Application Banners: Identifying and Classifying 311 Non-Web Banners 311 Web Client Banners: The User-Agent String 312 Further Reading 314 Chapter聽15.聽Network Mapping 315 Creating an Initial Network Inventory and Map 315 Creating an Inventory: Data, Coverage, and Files 316 Phase I: The First Three Questions 317 Phase II: Examining the IP Space 320 Phase III: Identifying Blind and Confusing Traffic 325 Phase IV: Identifying Clients and Servers 329 Identifying Sensing and Blocking Infrastructure 331 Updating the Inventory: Toward Continuous Audit 331 Further Reading 332 Index 333 About the Author 346 Traditional intrusion detection and logfile analysis are no longer enough to protect today’s complex networks. In this practical guide, security researcher Michael Collins shows you several techniques and tools for collecting and analyzing network traffic datasets. You’ll understand how your network is used, and what actions are necessary to protect and improve it.Divided into three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. It’s ideal for network administrators and operational security analysts familiar with scripting.Explore network, host, and service sensors for capturing security dataStore data traffic with relational databases, graph databases, Redis, and HadoopUse SiLK, the R language, and other tools for analysis and visualizationDetect unusual phenomena through Exploratory Data Analysis (EDA)Identify significant structures in networks with graph analysisDetermine the traffic that’s crossing service ports in a networkExamine traffic volume and behavior to spot DDoS and database raidsGet a step-by-step process for network mapping and inventory System and network administrators have traditionally monitored their systems through general tools such as intrusion detection and logfile analysis. But modern, complex networks, suffering from more and more sophisticated attacks, deserve more analytical tools. Michael Collins, a leading researcher in security, introduces the techniques needed in this book and highlights some of the computing tools that will help catch problems. The book is divided into three large sections: data collection, analysis, and taking action. These can be iterative, as each discovery alerts the administrator to data that should be collected. Several forms of analysis and visualization are covered. Topics include: What data to capture on your systems Data fusion Structures and storage systems for data Using R, SiLK, and Python for analysis Visualization and exploratory data analysis Graph analysis Network mapping Address forensics: determining where traffic originates Handling malware Traditional intrusion detection and logfile analysis are no longer enough to protect today's complex networks. In this practical guide, security researcher Michael Collins shows you several techniques and tools for collecting and analyzing network traffic datasets. You'll understand how your network is used, and what actions are necessary to protect and improve it. Divided into three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. It's ideal for network administrators and operational security analysts familiar with scripting.