Linux(R) Debugging and Performance Tuning: Tips and Techniques (Prentice Hall Open Source Software Development Series)

A unique resource that shows Linux kernel and Linux application programmers how to write bug-free code that's optimized for performance. This resource includes: debugging tools and techniques for the Linux kernel itself and Linux applications; performance tools and techniques that show how to supercharge the kernel and the applications that run on it; and sophisticated profiling techniques that teach programmers how to uncover application bottlenecks. Like all books in the Arnold Robbins series, this book is written by Linux practitioners for Linux practitioners, and like all books in the series, this book contains many hundreds of lines of example program code from the Linux kernel and from real-world Linux applications. Though this book presents debugging strategy, it is fundamentally a hands-on Linux software debugging and performance tuning book. Cover......Page 1 Contents......Page 8 Foreword......Page 16 Acknowledgments......Page 18 About the Author......Page 20 Introduction......Page 22 Chapter 1 Profiling......Page 28 stopwatch......Page 30 date......Page 31 time......Page 32 clock......Page 33 gettimeofday......Page 38 Performance Tuning Using GNU gprof......Page 40 gcc Option Needed for gprof......Page 42 Building the sample1 Program and Using gprof......Page 43 kprof......Page 58 Building Graphviz, the Graph Feature......Page 59 Summary......Page 62 Web Resources for Profiling......Page 63 Chapter 2 Code Coverage......Page 64 What Is Code Coverage Analysis?......Page 65 gcov Can't Find Logic Errors......Page 66 Types of Code Coverage......Page 67 gcc Options Needed for gcov......Page 68 Web Resource for gcov......Page 79 Chapter 3 GNU Debugger (gdb)......Page 80 Installing gdb......Page 82 gdb Commands......Page 83 Some Useful gdb Commands......Page 84 Compiling a Program That Is to Be Debugged with gdb......Page 86 Invoking the gdb Debugger......Page 87 Debugging Using a Core File......Page 90 Running the Program and Getting the Core Dump......Page 91 Graphical gdb Interfaces......Page 92 Installing ddd......Page 93 ddd Ways to View a Program......Page 94 Insight Features......Page 97 Installing Insight......Page 98 Debugging Symbols Increase the Executable's Size......Page 104 Debugging Techniques......Page 105 Web Resources for GNU Debugger......Page 107 Chapter 4 Memory Management Debugging......Page 108 Dynamic Memory Functions......Page 109 MEMWATCH......Page 111 YAMD......Page 113 Electric Fence......Page 120 Valgrind......Page 124 Installing Valgrind......Page 125 Losing Your Memory......Page 128 Cache Profiling......Page 133 Web Resources for Memory Checkers......Page 136 Chapter 5 System Information (/proc)......Page 138 What Is /proc?......Page 139 Ways an Administrator Can Use /proc......Page 140 Process-Specific Subdirectories......Page 141 Summary......Page 175 Web Resources for /proc......Page 176 Chapter 6 System Tools......Page 178 Processes......Page 179 Task States......Page 180 ps Reports the Process Status......Page 181 pgrep Lists the Process IDs That Match the Input Criteria......Page 195 pstree Displays a Tree of Processes......Page 196 strace Traces System Calls......Page 197 The Magic Key Sequence Gets a Back Trace......Page 201 lsof Lists Open Files......Page 203 ifconfig Configures the Network Interface......Page 205 tcpdump Dumps Traffic on a Network......Page 206 netstat Shows the Network Status......Page 207 Summary......Page 211 Web Resources for Systems Tools......Page 212 Chapter 7 System Error Messages......Page 214 Typical Distribution /boot Layout of the Kernel Files......Page 215 Oops Analysis......Page 217 Building the Kernel......Page 220 Processing the Oops Using ksymoops......Page 223 The gcc -S Option Generates Assembly Code......Page 226 An Oops in the 3c59x Network Driver 2.4.7ac11......Page 227 A Linux-2.5.1-pre5 Oops......Page 231 perror Describes a System or MySQL Error Code......Page 235 Web Resources for Oops......Page 236 Chapter 8 Event Logging......Page 238 syslogd Key Options......Page 240 syslog.conf......Page 241 klogd......Page 244 Logger Command......Page 245 Provide Sufficient Disk Space......Page 246 Log File Management......Page 248 Debugging Using Syslog Messages......Page 249 Error Logging Subsystem for Event Logging......Page 253 Benefits of Event-Type Logging......Page 254 Event Logging Utilities......Page 255 Remote Logging......Page 256 Setting up the Event Consolidator......Page 257 evlog Packages......Page 258 Web Resource for Event Logging......Page 259 Chapter 9 Linux Trace Toolkit......Page 260 Architecture Component Tasks......Page 262 Package and Installation......Page 263 Building the Kernel......Page 267 Building LTT User-Level Tools......Page 269 Data Acquisition......Page 270 Stopping Data Recording......Page 271 Ways to Analyze Tracing Processes......Page 272 tracevisualizer......Page 275 Tracing Example for Tarring a Subdirectory......Page 280 tracedcore filename......Page 283 traceanalyze filename......Page 284 tracedump filename......Page 285 Web Resource for the Linux Trace Toolkit......Page 286 Chapter 10 oprofile: a Profiler Supported by the Kernel......Page 288 Instrumentation......Page 289 oprofile: a System-Wide Profiler......Page 290 Building the Kernel......Page 291 Utilities for oprofile......Page 294 General Profile Steps......Page 295 Examining a Single Executable's Profile......Page 299 System-Wide Symbol Summary, Including Per-Application Libraries......Page 303 Hardware Counters......Page 304 Minimizing Cache Misses......Page 305 Padding and Aligning Structures......Page 306 Loop Grouping......Page 307 Blocking......Page 308 The Prospect Profiler......Page 314 Summary......Page 315 Web Resources for Profiling......Page 316 Chapter 11 User-Mode Linux......Page 318 UML: Introduction to the Kernel and Root File System......Page 320 Patching and Building the Kernel......Page 323 UML Kernel Options......Page 324 Building the Kernel......Page 330 Root Image......Page 331 File Systems......Page 332 Setting up a gdb Session......Page 333 A Typical gdb Session......Page 334 GDB Thread Analysis......Page 335 UML Networking Support......Page 337 Adding File Systems......Page 338 UML Utilities......Page 339 Summary......Page 340 Web Resources for User-Mode Linux......Page 341 Chapter 12 Dynamic Probes......Page 342 General Steps for Using Probes......Page 345 Kprobes: Kernel Dynamic Probes......Page 346 Building the Kernel......Page 347 Kprobe Interfaces......Page 348 Registering and Unregistering Kprobes......Page 349 Probe Example for sys_open......Page 351 Makefile for a Basic Kernel Module......Page 353 Finding an Offset in sys_open......Page 355 Jumper Probes......Page 356 Uses of Kprobes......Page 357 Web Resource for Dynamic Probes......Page 358 Chapter 13 Kernel-Level Debuggers (kgdb and kdb)......Page 360 kgdb......Page 362 kgdb Kernel Patching......Page 363 Checking the Null Modem Serial Cable Setup......Page 368 A Typical gdb Session......Page 370 Using kgdb to Stop and View the Kernel Source......Page 371 gdb Thread Analysis......Page 374 kdb Kernel Patching......Page 375 Building the Kernel......Page 377 kdb Activation......Page 379 kdb Commands......Page 380 kdb Debug Sessions......Page 381 Summary......Page 395 Web Resources for kgdb and kdb......Page 396 Chapter 14 Crash Dump......Page 398 Crash Dump Kernel Options......Page 401 Patching and Building the Kernel......Page 403 Building the Kernel......Page 404 General Crash Dump Steps......Page 406 LKCD Commands......Page 412 System Panic and Crash Dump Taken......Page 413 Client......Page 418 diskdump: a Crash Dump Facility......Page 419 Viewing an mcore Crash Dump......Page 420 Summary......Page 437 Web Resources for Crash Dump......Page 438 A......Page 440 C......Page 441 D......Page 443 E......Page 444 H......Page 445 K......Page 446 L......Page 447 M......Page 448 P......Page 449 S......Page 451 U......Page 453 W-Z......Page 454

“This is an excellent introduction to Linux programming. The topics are well chosen and lucidly presented. I learned things myself, especially about internationalization, and I’ve been at this for quite a while.”

—Chet Ramey, Coauthor and Maintainer of the Bash shell

“This is a good introduction to Linux programming. Arnold’s technique of showing how experienced programmers use the Linux programming interfaces is a nice touch, much more useful than the canned programming examples found in most books.”

—Ulrich Drepper, Project Lead, GNU C library

“A gentle yet thorough introduction to the art of UNIX system programming, Linux Programming by Example uses code from a wide range of familiar programs to illustrate each concept it teaches. Readers will enjoy an interesting mix of in-depth API descriptions and portability guidelines, and will come away well prepared to begin reading and writing systems applications. Heartily recommended.”

—Jim Meyering, Coauthor and Maintainer of the GNU Core Utility Programs

Learn Linux® programming, hands-on... from real source code

This book teaches Linux programming in the most effective way possible: by showing and explaining well-written programs. Drawing from both V7 Unix® and current GNU source code, Arnold Robbins focuses on the fundamental system call APIs at the core of any significant program, presenting examples from programs that Linux/Unix users already use every day. Gradually, one step at a time, Robbins teaches both high-level principles and “under the hood” techniques. Along the way, he carefully addresses real-world issues like performance, portability, and robustness. Coverage includes:

• Memory management
• File I/O
• File metadata
• Processes
• Users and groups
• Sorting and searching
• Argument parsing
• Extended interfaces
• Signals
• Internationalization
• Debugging
• And more...

Just learning to program? Switching from Windows®? Already developing with Linux but interested in exploring the system call interface further? No matter which, quickly and directly, this book will help you master the fundamentals needed to build serious Linux software.

Companion Web Sites, authors.phptr.com/robbins and www.linux-by-example.com, include all code examples.

Learn Linux kernel programming, hands-on: a uniquely effective top-down approach

The Linux® Kernel Primer is the definitive guide to Linux kernel programming. The authors' unique top-down approach makes kernel programming easier to understand by systematically tracing functionality from user space into the kernel and carefully associating kernel internals with user-level programming fundamentals. Their approach helps you build on what you already know about Linux, gaining a deep understanding of how the kernel works and how its elements fit together.

One step at a time, the authors introduce all the tools and assembly language programming techniques required to understand kernel code and control its behavior. They compare x86 and PowerPC implementations side-by-side, illuminating cryptic functionality through carefully-annotated source code examples and realistic projects. The Linux® Kernel Primer is the first book to offer in-depth coverage of the rapidly growing PowerPC Linux development platform, and the only book to thoroughly discuss kernel configuration with the Linux build system. Coverage includes

• Data structures
• x86 and PPC assembly language
• Viewing kernel internals
• Linux process model
• User and kernel space
• Interrupts and exceptions
• Memory allocation and tracking
• Tracing subsystem behavior
• I/O interactions
• Filesystems and file operations
• Scheduling and synchronization
• Kernel boot process
• Kernel build system
• Configuration options
• Device drivers
• And more...

If you know C, this book teaches you all the skills and techniques you need to succeed with Linux kernel programming. Whether you're a systems programmer, software engineer, systems analyst, test professional, open source project contributor, or simply a Linux enthusiast, you'll find it indispensable.

Learn Linux kernel programming, hands-on : a uniquely effective top-down approach The Linux ® Kernel Primer is the definitive guide to Linux kernel programming. The authors' unique top-down approach makes kernel programming easier to understand by systematically tracing functionality from user space into the kernel and carefully associating kernel internals with user-level programming fundamentals. Their approach helps you build on what you already know about Linux, gaining a deep understanding of how the kernel works and how its elements fit together. One step at a time, the authors introduce all the tools and assembly language programming techniques required to understand kernel code and control its behavior. They compare x86 and PowerPC implementations side-by-side, illuminating cryptic functionality through carefully-annotated source code examples and realistic projects. The Linux ® Kernel Primer is the first book to offer in-depth coverage of the rapidly growing PowerPC Linux development platform, and the only book to thoroughly discuss kernel configuration with the Linux build system. Coverage includes Data structures x86 and PPC assembly language Viewing kernel internals Linux process model User and kernel space Interrupts and exceptions Memory allocation and tracking Tracing subsystem behavior I/O interactions Filesystems and file operations Scheduling and synchronization Kernel boot process Kernel build system Configuration options Device drivers And more... If you know C, this book teaches you all the skills and techniques you need to succeed with Linux kernel programming. Whether you're a systems programmer, software engineer, systems analyst, test professional, open source project contributor, or simply a Linux enthusiast, you'll find it indispensable. © Copyright Pearson Education. All rights reserved

Learn Linux debugging and optimization-at kernel and application levels-hands-on!

This is the definitive guide to Linux software debugging and performance optimization at both the kernel and application levels. Using extensive Linux code examples, Steve Best systematically introduces open source tools and best-practice techniques for delivering bug-free, well-tuned code.

Drawing on his exceptional experience optimizing Linux systems for IBM, Best covers issues ranging from memory management and I/O to system processes and kernel bug messages. You'll walk through real debugging sessions, discovering the strategies experts use to debug even the most complex application- and kernel-related problems. You'll master sophisticated profiling techniques for identifying and resolving bottlenecks more quickly and learn how to capture the right data in the event of trouble. Coverage includes

• Bottleneck identification
• Code coverage
• Debuggers: gdb, kgdb, and KDB
• Memory management
• /proc kernel data analysis
• System process monitoring
• Network performance
• Oops bug messages
• Syslog and event logging
• Execution traces
• Profiling kernel behavior
• Cache misses
• User-Mode Linux
• Dynamic probes
• Crash dump analysis
• And more...

Linux® Debugging and Performance Tuning will be indispensable for every developer who needs to supercharge the Linux kernel and applications, and for every administrator and support specialist who must resolve Linux reliability or performance issues.

Annotation Linux Programming by Example introduces new Linux programmers to the core Linux programming interfaces in a gradual, consistent fashion, progressing intuitively from the basic to the more complex. It covers I/O, file metainformation, users and groups, processes, basic interprocess communication (pipes), general purpose APIs, signals, internationalization, and ends with a chapter on debugging Linux programs. Programmers know that the best way to learn about programming is to study well-written programs. This book teaches the fundamental Linux programming interfaces, those that form the core of any significant program, by presenting example code from real-world production programs that Linux users use every day. By looking at concrete programs, its possible not only to see how to use the Linux programming interfaces, but also to examine the real-world issues (performance, portability, robustness) that arise in writing Linux software. This book is the FIRST in a new series of books featuring Arnold Robbins as Series Editor. The books will all be branded "Linux Programming by Example" and cover programming topics for the new Linux programmer and Windows programmers making the switch. Top Reasons people will buy this book: -Covers Linux on PPC -Top-down approach traces functionality from user space into the kernel -Lots of code commentary and examples. It walks you through the actual source code implementation. -Side by side comparison of x86 and PPC -Hands on Examples and Projects -Covers the kernel build system. The "Linux Kernel Primer" offers a comprehensive view of the underpinnings of the Linux kernel. This book starts with a guide of the necessary tools a developer needs to be able to understand and manipulate the source code including cryptic programming fundamentals found throughout the kernel code. It then follows up with an in depth analysis of the major subsystems including process management, memory management, scheduling, I/O, and filesystems. This book also provides information necessary to get started developing on the Linux kernel. The specifics of Intel and PowerPC architecture implementations are covered side by side providing perspective on architecture specific features and how Linux make use of them. Similar in approach to Kernighan's "Practice of Programming" 020161586X Preface Files and Users Introduction Arguments, Options, and the Environment User-Level Memory Management Files and File I/O Directories and File Metadata General Library Interfaces - Part 1 Putting it all together: ls Filesystems and Directory Walks Processes, IPC, and Internationalization Process Management and Pipes Signals Permissions and User and Group ID Numbers General Library Interfaces - Part 2 Internationalization and Localization Extended Interfaces Debugging and Final Project Debugging A Project That Ties Everything Together Appendixes Teach Yourself Programming in Ten Years Caldera Ancient UNIX License GNU General Public License Index A unique resource that shows Linux kernel AND Linux application programmers how to write bug-free code that's optimized for performance. 1) Debugging techniques for Linux applications and the kernel 2) Performance tuning techniques for Linux application and the kernel. 3) Sample programs that show how debugging tools can find programming errors. 4) Sample programs that show how profiling tools can show which routines applications spend the most time in. 5) Key features that can be set up in the kernel to capture data once a problem occurs. "This is the definitive guide to Linux software debugging and performance optimization at both the kernel and application levels. Using extensive Linux code examples, Steve Best systematically introduces open source tools and best-practice techniques for delivering bug-free, well-tuned code."-- Quatrième de couverture "This is the definitive guide to Linux software debugging and performance optimization at both the kernel and application levels. Using extensive Linux code examples, Steve Best systematically introduces open source tools and best-practice techniques for delivering bug-free, well-tuned code."--Jacket Foreword Acknowledgments About the Authors Preface Overview Exploration Toolkit Processes: The Principal Model of Execution Memory Management Input/Output Filesystems Scheduling and Kernel Synchronization Booting the Kernel Building the Linux Kernel Adding Your Code to the Kernel Bibliography Index