Network security through data analysis : from data to action

Traditional intrusion detection and logfile analysis are no longer enough to protect today’s complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. You’ll understand how your network is used, and what actions are necessary to harden and defend the systems within it. In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics. You’ll learn how to: * Use sensors to collect network, service, host, and active domain data * Work with the SiLK toolset, Python, and other tools and techniques for manipulating data you collect * Detect unusual phenomena through exploratory data analysis (EDA), using visualization and mathematical techniques * Analyze text data, traffic behavior, and communications mistakes * Identify significant structures in your network with graph analysis * Examine insider threat data and acquire threat intelligence * Map your network and identify significant hosts within it * Work with operations to develop defenses and analysis techniques Copyright Table of Contents Preface Audience Contents of This Book Changes Between Editions Conventions Used in This Book Using Code Examples O'Reilly Safari How to Contact Us Acknowledgments Part I. Data Chapter 1. Organizing Data: Vantage, Domain, Action, and Validity Domain Vantage Choosing Vantage Actions: What a Sensor Does with Data Validity and Action Internal Validity External Validity Construct Validity Statistical Validity Attacker and Attack Issues Further Reading Chapter 2. Vantage: Understanding Sensor Placement in Networks The Basics of Network Layering. Network Layers and VantageNetwork Layers and Addressing MAC Addresses IPv4 Format and Addresses IPv6 Format and Addresses Validity Challenges from Middlebox Network Data Further Reading Chapter 3. Sensors in the Network Domain Packet and Frame Formats Rolling Buffers Limiting the Data Captured from Each Packet Filtering Specific Types of Packets What If It's Not Ethernet? NetFlow NetFlow v5 Formats and Fields NetFlow Generation and Collection Data Collection via IDS Classifying IDSs IDS as Classifier Improving IDS Performance Enhancing IDS Detection Configuring Snort. Enhancing IDS ResponsePrefetching Data Middlebox Logs and Their Impact VPN Logs Proxy Logs NAT Logs Further Reading Chapter 4. Data in the Service Domain What and Why Logfiles as the Basis for Service Data Accessing and Manipulating Logfiles The Contents of Logfiles The Characteristics of a Good Log Message Existing Logfiles and How to Manipulate Them Stateful Logfiles Further Reading Chapter 5. Sensors in the Service Domain Representative Logfile Formats HTTP: CLF and ELF Simple Mail Transfer Protocol (SMTP) Sendmail Microsoft Exchange: Message Tracking Logs. Additional Useful LogfilesStaged Logging LDAP and Directory Services File Transfer, Storage, and Databases Logfile Transport: Transfers, Syslog, and Message Queues Transfer and Logfile Rotation Syslog Further Reading Chapter 6. Data and Sensors in the Host Domain A Host: From the Network's View The Network Interfaces The Host: Tracking Identity Processes Structure Filesystem Historical Data: Commands and Logins Other Data and Sensors: HIPS and AV Further Reading Chapter 7. Data and Sensors in the Active Domain Discovery, Assessment, and Maintenance. Discovery: ping, traceroute, netcat, and Half of nmapChecking Connectivity: Using ping to Connect to an Address Tracerouting Using nc as a Swiss Army Multitool nmap Scanning for Discovery Assessment: nmap, a Bunch of Clients, and a Lot of Repositories Basic Assessment with nmap Using Active Vantage Data for Verification Further Reading Part II. Tools Chapter 8. Getting Data in One Place High-Level Architecture The Sensor Network The Repository Query Processing Real-Time Processing Source Control Log Data and the CRUD Paradigm A Brief Introduction to NoSQL Systems. From Data to Action. Second edition. Traditional intrusion detection and logfile analysis are no longer enough to protect todays complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. Youll understand how your network is used, and what actions are necessary to harden and defend the systems within it.In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics.Youll learn how to:Use sensors to collect network, service, host, and active domain data Work with the SiLK toolset, Python, and other tools and techniques for manipulating data you collect Detect unusual phenomena through exploratory data analysis (EDA), using visualization and mathematical techniques Analyze text data, traffic behavior, and communications mistakes Identify significant structures in your network with graph analysis Examine insider threat data and acquire threat intelligence Map your network and identify significant hosts within it Work with operations to develop defenses and analysis techniques Traditional intrusion detection and logfile analysis are no longer enough to protect today’s complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. You’ll understand how your network is used, and what actions are necessary to harden and defend the systems within it. In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics. -- Provided by publisher