Computer Vision Using Local Binary Patterns (Computational Imaging and Vision, 40)

The recent emergence of Local Binary Patterns (LBP) has led to significant progress in applying texture methods to various computer vision problems and applications. The focus of this research has broadened from 2D textures to 3D textures and spatiotemporal (dynamic) textures. Also, where texture was once utilized for applications such as remote sensing, industrial inspection and biomedical image analysis, the introduction of LBP-based approaches have provided outstanding results in problems relating to face and activity analysis, with future scope for face and facial expression recognition, biometrics, visual surveillance and video analysis. Computer Vision Using Local Binary Patterns provides a detailed description of the LBP methods and their variants both in spatial and spatiotemporal domains. This comprehensive reference also provides an excellent overview as to how texture methods can be utilized for solving different kinds of computer vision and image analysis problems. Source codes of the basic LBP algorithms, demonstrations, some databases and a comprehensive LBP bibliography can be found from an accompanying web site. Topics include: local binary patterns and their variants in spatial and spatiotemporal domains, texture classification and segmentation, description of interest regions, applications in image retrieval and 3D recognition - Recognition and segmentation of dynamic textures, background subtraction, recognition of actions, face analysis using still images and image sequences, visual speech recognition and LBP in various applications. Written by pioneers of LBP, this book is an essential resource for researchers, professional engineers and graduate students in computer vision, image analysis and pattern recognition. The book will also be of interest to all those who work with specific applications of machine vision. Computer Vision Using Local Binary Patterns 4 Preface 6 Contents 9 Abbreviations 13 Part I: Local Binary Pattern Operators 16 Chapter 1: Background 17 1.1 The Role of Texture in Computer Vision 17 1.2 Motivation and Background for LBP 18 1.3 A Brief History of LBP 20 1.4 Overview of the Book 21 References 24 Chapter 2: Local Binary Patterns for Still Images 27 2.1 Basic LBP 27 2.2 Derivation of the Generic LBP Operator 27 2.3 Mappings of the LBP Labels: Uniform Patterns 30 2.4 Rotational Invariance 32 2.4.1 Rotation Invariant LBP 33 2.4.2 Rotation Invariance Using Histogram Transformations 34 2.5 Complementary Contrast Measure 35 2.6 Non-parametric Classification Principle 37 2.7 Multiscale LBP 38 2.8 Center-Symmetric LBP 39 2.9 Other LBP Variants 40 2.9.1 Preprocessing 40 2.9.2 Neighborhood Topology 45 2.9.3 Thresholding and Encoding 46 2.9.4 Multiscale Analysis 49 2.9.5 Handling Rotation 51 2.9.6 Handling Color 52 2.9.7 Feature Selection and Learning 53 2.9.8 Complementary Descriptors 56 2.9.9 Other Methods Inspired by LBP 56 References 57 Chapter 3: Spatiotemporal LBP 62 3.1 Basic VLBP 62 3.2 Rotation Invariant VLBP 65 3.3 Local Binary Patterns from Three Orthogonal Planes 66 3.4 Rotation Invariant LBP-TOP 70 3.4.1 Problem Description 70 3.4.2 One Dimensional Histogram Fourier LBP-TOP (1DHFLBP-TOP) 72 3.5 Other Variants of Spatiotemporal LBP 74 References 77 Part II: Analysis of Still Images 79 Chapter 4: Texture Classification and Segmentation 80 4.1 Texture Classification 80 4.1.1 Texture Image Datasets 81 4.1.2 Texture Classification Experiments 83 4.2 Unsupervised Texture Segmentation 84 4.2.1 Overview of the Segmentation Algorithm 85 4.2.2 Splitting 86 4.2.3 Agglomerative Merging 86 4.2.4 Pixelwise Classification 87 4.2.5 Experiments 88 4.3 Discussion 88 References 89 Chapter 5: Description of Interest Regions 91 5.1 Related Work 91 5.2 CS-LBP Descriptor 92 5.3 Image Matching Experiments 94 5.3.1 Matching Results 96 5.4 Discussion 97 References 98 Chapter 6: Applications in Image Retrieval and 3D Recognition 99 6.1 Block-Based Methods for Image Retrieval 99 6.1.1 Description of the Method 100 6.1.1.1 Nonparametric Dissimilarity Measure 100 6.1.1.2 The Block Division Method 100 6.1.1.3 The Primitive Blocks Method 101 6.1.2 Experiments 102 6.1.2.1 Test Database and Configurations 102 6.1.2.2 Results 104 6.1.3 Discussion 105 6.2 Recognition of 3D Textured Surfaces 106 6.2.1 Texture Description by LBP Histograms 107 6.2.2 Use of Multiple Histograms as Texture Models 108 6.2.3 Experiments with CUReT Textures 109 6.2.4 Experiments with Scene Images 111 6.2.5 Discussion 112 References 114 Part III: Motion Analysis 116 Chapter 7: Recognition and Segmentation of Dynamic Textures 117 7.1 Dynamic Texture Recognition 117 7.1.1 Related Work 117 7.1.2 Measures 118 7.1.3 Multi-resolution Analysis 119 7.1.4 Experimental Setup 119 7.1.5 Results for VLBP 120 7.1.6 Results for LBP-TOP 121 7.1.7 Experiments of Rotation Invariant LBP-TOP to View Variations 123 7.2 Dynamic Texture Segmentation 124 7.2.1 Related Work 124 7.2.2 Features for Segmentation 126 7.2.2.1 LBP-TOP/Contrast 126 7.2.2.2 LBP-TOP/Contrast for Segmentation 127 7.2.3 Segmentation Procedure 128 7.2.3.1 Splitting 128 7.2.3.2 Merging 129 7.2.3.3 Pixelwise Classification 130 7.2.4 Experiments 130 7.3 Discussion 131 References 132 Chapter 8: Background Subtraction 134 8.1 Related Work 134 8.2 An LBP-based Approach 135 8.2.1 Modifications of the LBP Operator 135 8.2.2 Background Modeling 136 8.2.3 Foreground Detection 137 8.3 Experiments 137 8.4 Discussion 140 References 141 Chapter 9: Recognition of Actions 142 9.1 Related Work 142 9.2 Static Texture Based Description of Movements 143 9.3 Dynamic Texture Method for Motion Description 145 9.3.1 Human Detection with Background Subtraction 145 9.3.2 Action Description 146 9.3.3 Modeling Temporal Information with Hidden Markov Models 148 9.4 Experiments 149 9.5 Discussion 152 References 153 Part IV: Face Analysis 156 Chapter 10: Face Analysis Using Still Images 157 10.1 Face Description Using LBP 157 10.2 Eye Detection 159 10.3 Face Detection 160 10.4 Face Recognition 165 10.5 Facial Expression Recognition 170 10.6 LBP in Other Face Related Tasks 171 10.7 Conclusion 171 References 171 Chapter 11: Face Analysis Using Image Sequences 175 11.1 Facial Expression Recognition Using Spatiotemporal LBP 175 11.2 Face Recognition from Videos 179 11.3 Gender Classification from Videos 182 11.4 Discussion 184 References 185 Chapter 12: Visual Recognition of Spoken Phrases 187 12.1 Related Work 187 12.2 System Overview 188 12.3 Local Spatiotemporal Descriptors for Visual Information 188 12.4 Experiments 191 12.4.1 Dataset Description 191 12.4.2 Experimental Results 191 12.4.3 Boosting Slice Features 193 12.5 Discussion 194 References 195 Part V: LBP in Various Computer Vision Applications 196 Chapter 13: LBP in Different Applications 197 13.1 Detection and Tracking of Objects 197 13.2 Biometrics 198 13.3 Eye Localization and Gaze Tracking 199 13.4 Face Recognition in Unconstrained Environments 199 13.5 Visual Inspection 200 13.6 Biomedical Applications 201 13.7 Texture and Video Texture Synthesis 202 13.8 Steganography and Image Forensics 203 13.9 Video Analysis 203 13.10 Systems for Photo Management and Interactive TV 204 13.11 Embedded Vision Systems and Smart Cameras 205 References 206 Computer Vision Using Local Binary Patterns 209 Index 211 Written by the pioneers of local binary patterns, and including a wealth of illustrations, this book gives those working with LBPs a single-source, comprehensive resource on the uses of LBP methodology, in both spatial and spatiotemporal domains.