The Antivirus Hacker's Handbook

Hack your antivirus software to stamp out future vulnerabilities The Antivirus Hacker's Handbook guides you through the process of reverse engineering antivirus software. You explore how to detect and exploit vulnerabilities that can be leveraged to improve future software design, protect your network, and anticipate attacks that may sneak through your antivirus' line of defense. You'll begin building your knowledge by diving into the reverse engineering process, which details how to start from a finished antivirus software program and work your way back through its development using the functions and other key elements of the software. Next, you leverage your new knowledge about software development to evade, attack, and exploit antivirus software—all of which can help you strengthen your network and protect your data. While not all viruses are damaging, understanding how to better protect your computer against them can help you maintain the integrity of your network. Discover how to reverse engineer your antivirus software Explore methods of antivirus software evasion Consider different ways to attack and exploit antivirus software Understand the current state of the antivirus software market, and get recommendations for users and vendors who are leveraging this software The Antivirus Hacker's Handbook is the essential reference for software reverse engineers, penetration testers, security researchers, exploit writers, antivirus vendors, and software engineers who want to understand how to leverage current antivirus software to improve future applications. Cover Title Page Copyright Contents Introduction Part I Antivirus Basics Chapter 1 Introduction to Antivirus Software What Is Antivirus Software? Antivirus Software: Past and Present Antivirus Scanners, Kernels, and Products Typical Misconceptions about Antivirus Software Antivirus Features Basic Features Making Use of Native Languages Scanners Signatures Compressors and Archives Unpackers Emulators Miscellaneous File Formats Advanced Features Packet Filters and Firewalls Self-Protection Anti-Exploiting Summary Chapter 2 Reverse-Engineering the Core Reverse-Engineering Tools Command-Line Tools versus GUI Tools Debugging Symbols Tricks for Retrieving Debugging Symbols Debugging Tricks Backdoors and Configuration Settings Kernel Debugging Debugging User-Mode Processes with a Kernel-Mode Debugger Analyzing AV Software with Command-Line Tools Porting the Core A Practical Example: Writing Basic Python Bindings for Avast for Linux A Brief Look at Avast for Linux Writing Simple Python Bindings for Avast for Linux The Final Version of the Python Bindings A Practical Example: Writing Native C/C++ Tools for Comodo Antivirus for Linux Other Components Loaded by the Kernel Summary Chapter 3 The Plug-ins System Understanding How Plug-ins Are Loaded A Full-Featured Linker in Antivirus Software Understanding Dynamic Loading Advantages and Disadvantages of the Approaches for Packaging Plug-ins Types of Plug-ins Scanners and Generic Routines File Format and Protocol Support Heuristics Bayesian Networks Bloom Filters Weights-Based Heuristics Some Advanced Plug-ins Memory Scanners Non-native Code Scripting Languages Emulators Summary Chapter 4 Understanding Antivirus Signatures Typical Signatures Byte-Streams Checksums Custom Checksums Cryptographic Hashes Advanced Signatures Fuzzy Hashing Graph-Based Hashes for Executable Files Summary Chapter 5 The Update System Understanding the Update Protocols Support for SSL/TLS Verifying the Update Files Dissecting an Update Protocol When Protection Is Done Wrong Summary Part II Antivirus Software Evasion Chapter 6 Antivirus Software Evasion Who Uses Antivirus Evasion Techniques? Discovering Where and How Malware Is Detected Old Tricks for Determining Where Malware Is Detected: Divide and Conquer Evading a Simple Signature-Based Detection with the Divide and Conquer Trick Binary Instrumentation and Taint Analysis Summary Chapter 7 Evading Signatures File Formats: Corner Cases and Undocumented Cases Evading a Real Signature Evasion Tips and Tricks for Specific File Formats PE Files JavaScript String Encoding Executing Code on the Fly Hiding the Logic: Opaque Predicates and Junk Code PDF Summary Chapter 8 Evading Scanners Generic Evasion Tips and Tricks Fingerprinting Emulators Advanced Evasion Tricks Taking Advantage of File Format Weaknesses Using Anti-emulation Techniques Using Anti-disassembling Techniques Disrupting Code Analyzers through Anti-analysis More Anti-Anti-Anti... Causing File Format Confusion Automating Evasion of Scanners Initial Steps Installing ClamAV Installing Avast Installing AVG Installing F-Prot Installing Comodo Installing Zoner Antivirus MultiAV Configuration peCloak Writing the Final Tool Summary Chapter 9 Evading Heuristic Engines Heuristic Engine Types Static Heuristic Engines Bypassing a Simplistic Static Heuristic Engine Dynamic Heuristic Engines Userland Hooks Bypassing a Userland HIPS Kernel-Land Hooks Summary Chapter 10 Identifying the Attack Surface Understanding the Local Attack Surface Finding Weaknesses in File and Directory Privileges Escalation of Privileges Incorrect Privileges in Files and Folders Incorrect Access Control Lists Kernel-Level Vulnerabilities Exotic Bugs Exploiting SUID and SGID Binaries on Unix-Based Platforms ASLR and DEP Status for Programs and Binaries Exploiting Incorrect Privileges on Windows Objects Exploiting Logical Flaws Understanding the Remote Attack Surface File Parsers Generic Detection and File Disinfection Code Network Services, Administration Panels, and Consoles Firewalls, Intrusion Detection Systems, and Their Parsers Update Services Browser Plug-ins Security Enhanced Software Summary Chapter 11 Denial of Service Local Denial-of-Service Attacks Compression Bombs Creating a Simple Compression Bomb Bugs in File Format Parsers Attacks against Kernel Drivers Remote Denial-of-Service Attacks Compression Bombs Bugs in File Format Parsers Summary Part III Analysis and Exploitation Chapter 12 Static Analysis Performing a Manual Binary Audit File Format Parsers Remote Services Summary Chapter 13 Dynamic Analysis Fuzzing What Is a Fuzzer? Simple Fuzzing Automating Fuzzing of Antivirus Products Using Command-Line Tools Porting Antivirus Kernels to Unix Fuzzing with Wine Problems, Problems, and More Problems Finding Good Templates Finding Template Files Maximizing Code Coverage Blind Code Coverage Fuzzer Using Blind Code Coverage Fuzzer Nightmare, the Fuzzing Suite Configuring Nightmare Finding Samples Configuring and Running the Fuzzer Summary Chapter 14 Local Exploitation Exploiting Backdoors and Hidden Features Finding Invalid Privileges, Permissions, and ACLs Searching Kernel-Land for Hidden Features More Logical Kernel Vulnerabilities Summary Chapter 15 Remote Exploitation Implementing Client-Side Exploitation Exploiting Weakness in Sandboxing Exploiting ASLR, DEP, and RWX Pages at Fixed Addresses Writing Complex Payloads Taking Advantage of Emulators Exploiting Archive Files Finding Weaknesses in Intel x86, AMD x86_64, and ARM Emulators Using JavaScript, VBScript, or ActionScript Determining What an Antivirus Supports Launching the Final Payload Exploiting the Update Services Writing an Exploit for an Update Service Server-Side Exploitation Differences between Client-Side and Server-Side Exploitation Exploiting ASLR, DEP, and RWX Pages at Fixed Addresses Summary Part IV Current Trends and Recommendations Chapter 16 Current Trends in Antivirus Protection Matching the Attack Technique with the Target The Diversity of Antivirus Products Zero-Day Bugs Patched Bugs Targeting Home Users Targeting Small to Medium-Sized Companies Targeting Governments and Big Companies The Targets of Governments Summary Chapter 17 Recommendations and the Possible Future Recommendations for Users of Antivirus Products Blind Trust Is a Mistake Isolating Machines Improves Protection Auditing Security Products Recommendations for Antivirus Vendors Engineering Is Different from Security Exploiting Antivirus Software Is Trivial Perform Audits Fuzzing Use Privileges Safely Reduce Dangerous Code in Parsers Improve the Safety of Update Services and Protocols Remove or Disable Old Code Summary Index EULA Analyzing how hacks are done, so as to stop them in the future Reverse engineering is the process of analyzing hardware or software and understanding it, without having access to the source code or design documents. Hackers are able to reverse engineer systems and exploit what they find with scary results. Now the good guys can use the same tools to thwart these threats. Practical Reverse Engineering goes under the hood of reverse engineering for security analysts, security engineers, and system programmers, so they can learn how to use these same processes to stop hackers in their tracks. The book covers x86, x64, and ARM (the first book to cover all three); Windows kernel-mode code rootkits and drivers; virtual machine protection techniques; and much more. Best of all, it offers a systematic approach to the material, with plenty of hands-on exercises and real-world examples. Offers a systematic approach to understanding reverse engineering, with hands-on exercises and real-world examples Covers x86, x64, and advanced RISC machine (ARM) architectures as well as deobfuscation and virtual machine protection techniques Provides special coverage of Windows kernel-mode code (rootkits/drivers), a topic not often covered elsewhere, and explains how to analyze drivers step by step Demystifies topics that have a steep learning curve Includes a bonus chapter on reverse engineering tools Practical Reverse Engineering: Using x86, x64, ARM, Windows Kernel, and Reversing Tools provides crucial, up-to-date guidance for a broad range of IT professionals. This book provides a systematic approach to reverse engineering. Reverse engineering is not about reading assembly code, but actually understanding how different pieces/components in a system work. To reverse engineer a system is to understand how it is constructed and how it works. The book provides: Coverage of x86, x64, and ARM. In the past x86 was the most common architecture on the PC; however, times have changed and x64 is becoming the dominant architecture. It brings new complexity and constructs previously not present in x86. ARM (Advanced RISC Machine) is very common in embedded / consumer electronic devices; for example, most if not all cell phones run on ARM. All of apple's i-devices run on ARM. This book will be the first book to cover all three. Discussion of Windows kernel-mode code (rootkits/drivers). This topic has a steep learning curve so most practitioners stay away from this area because it is highly complex. However, this book will provide a concise treatment of this topic and explain how to analyze drivers step-by-step. The book uses real world examples from the public domain. The best way to learn is through a combination of concept discussions, examples, and exercises. This book uses real-world trojans / rootkits as examples congruent with real-life scenarios and Hands-on exercises The x86 is little-endian architecture based on the Intel 8086 processor.