Visual saliency : from pixel-level to object-level analysis

This book will provide an introduction to recent advances in theory, algorithms and application of Boolean map distance for image processing. Applications include modeling what humans find salient or prominent in an image, and then using this for guiding smart image cropping, selective image filtering, image segmentation, image matting, etc. In this book, the authors present methods for both traditional and emerging saliency computation tasks, ranging from classical low-level tasks like pixel-level saliency detection to object-level tasks such as subitizing and salient object detection. For low-level tasks, the authors focus on pixel-level image processing approaches based on efficient distance transform. For object-level tasks, the authors propose data-driven methods using deep convolutional neural networks. The book includes both empirical and theoretical studies, together with implementation details of the proposed methods. Below are the key features for different types of readers. For computer vision and image processing practitioners:* Efficient algorithms based on image distance transforms for two pixel-level saliency tasks; * Promising deep learning techniques for two novel object-level saliency tasks; * Deep neural network model pre-training with synthetic data; * Thorough deep model analysis including useful visualization techniques and generalization tests; * Fully reproducible with code, models and datasets available. For researchers interested in the intersection between digital topological theories and computer vision problems: * Summary of theoretic findings and analysis of Boolean map distance; * Theoretic algorithmic analysis; * Applications in salient object detection and eye fixation prediction. Students majoring in image processing, machine learning and computer vision: This book provides up-to-date supplementary reading material for course topics like connectivity based image processing, deep learning for image processing; Some easy-to-implement algorithms for course projects with data provided (as links in the book); Hands-on programming exercises in digital topology and deep learning. Contents 5 1 Overview 8 1.1 Pixel-Level Saliency Detection 8 1.2 Object-Level Saliency Computation 10 1.3 Book Outline 11 1.3.1 Boolean Map Saliency for Eye Fixation Prediction 11 1.3.2 A Distance Transform Perspective 12 1.3.3 Efficient Distance Transform for Salient Region Detection 12 1.3.4 Salient Object Subitizing 13 1.3.5 Unconstrained Salient Object Detection 14 Part I Pixel-Level Saliency 15 2 Boolean Map Saliency: A Surprisingly Simple Method 16 2.1 Related Work 17 2.2 Boolean Map Based Saliency 18 2.2.1 Boolean Map Generation 19 2.2.2 Attention Map Computation 20 Activation 20 Normalization 21 2.3 Experiments 22 2.3.1 Datasets 23 2.3.2 Compared Models 24 2.3.3 Evaluation Methods 25 2.3.4 Results 26 2.3.5 Speed Accuracy Tradeoff 33 2.3.6 Component Analysis 34 2.4 Conclusion 35 3 A Distance Transform Perspective 37 3.1 The Boolean Map Distance 37 3.2 BMS and the Boolean Map Distance Transform 39 3.3 BMS and the Minimum Barrier Distance Transform 40 3.3.1 Preliminaries 40 3.3.2 BMS and BMS and varphi 41 3.3.3 BMS Approximates the MBD Transform 44 3.4 Distance Transform Algorithms 47 3.5 Conclusion 48 4 Efficient Distance Transform for Salient Region Detection 49 4.1 Fast Approximate MBD Transform 50 4.1.1 Background: Distance Transform 51 4.1.2 Fast MBD Transform by Raster Scan 51 4.1.3 Approximation Error Analysis 53 4.2 Minimum Barrier Salient Region Detection 55 4.2.1 MBD Transform for Salient Region Detection 55 4.2.2 Combination with Backgroundness Cue 56 4.2.3 Post-processing 57 4.3 Experiments 58 4.3.1 Speed Performance 59 4.3.2 Evaluation Using PR Curve 59 4.3.3 Evaluation Using Weighted-FF 61 4.3.4 Limitations 62 4.4 Conclusion 64 Part II Object-Level Saliency 66 5 Salient Object Subitizing 67 5.1 Related Work 69 5.2 The SOS Dataset 70 5.2.1 Image Source 70 5.2.2 Annotation Collection 71 5.2.3 Annotation Consistency Analysis 73 5.3 Salient Object Subitizing by Convolutional Neural Network 75 5.3.1 Leveraging Synthetic Images for CNN Training 75 5.4 Experiments 78 5.4.1 Experimental Setting 78 5.4.2 Results 80 5.4.3 Analysis 81 5.5 Applications 87 5.5.1 Salient Object Detection 87 5.5.2 Image Retrieval 90 5.5.3 Other Applications 94 5.6 Conclusion 94 6 Unconstrained Salient Object Detection 96 6.1 Related Work 97 6.2 A Salient Object Detection Framework 98 6.2.1 MAP-Based Proposal Subset Optimization 99 6.2.2 Formulation Details 100 6.2.3 Optimization 101 6.2.4 Salient Object Proposal Generation by CNN 103 6.3 Experiments 105 6.3.1 Results 107 6.3.2 Component Analysis 110 6.4 Conclusion 111 7 Conclusion and Future Work 113 A Proof of Theorem 3.6 115 A.1 Preliminaries 115 A.1.1 Alexander's Lemma 115 A.1.2 Hyxel and Supercover 118 A.2 Proof of Theorem 3.6 119 B Proof of Lemma 4.2 121 B.1 Distance Transform on Graph 121 B.2 Proof of Lemma 4.2 123 C Proof of the Submodularity of Function 6.11 127 References 129 This book provides an introduction to recent advances in theory, algorithms and application of Boolean map distance for image processing. Applications include modeling what humans find salient or prominent in an image, and then using this for guiding smart image cropping, selective image filtering, image segmentation, image matting, etc. In this book, the authors present methods for both traditional and emerging saliency computation tasks, ranging from classical low-level tasks like pixel-level saliency detection to object-level tasks such as subitizing and salient object detection. For low-level tasks, the authors focus on pixel-level image processing approaches based on efficient distance transform. For object-level tasks, the authors propose data-driven methods using deep convolutional neural networks. The book includes both empirical and theoretical studies, together with implementation details of the proposed methods. Below are the key features for different types of readers. For computer vision and image processing practitioners: Efficient algorithms based on image distance transforms for two pixel-level saliency tasks; Promising deep learning techniques for two novel object-level saliency tasks; Deep neural network model pre-training with synthetic data; Thorough deep model analysis including useful visualization techniques and generalization tests; Fully reproducible with code, models and datasets available. For researchers interested in the intersection between digital topological theories and computer vision problems: Summary of theoretic findings and analysis of Boolean map distance; Theoretic algorithmic analysis; Applications in salient object detection and eye fixation prediction. Students majoring in image processing, machine learning and computer vision: This book provides up-to-date supplementary reading material for course topics like connectivity based image processing, deep learning for image processing; Some easy-to-implement algorithms for course projects with data provided (as links in the book); Hands-on programming exercises in digital topology and deep learning. Front Matter ....Pages i-vii Overview (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 1-7 Front Matter ....Pages 9-9 Boolean Map Saliency: A Surprisingly Simple Method (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 11-31 A Distance Transform Perspective (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 33-44 Efficient Distance Transform for Salient Region Detection (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 45-61 Front Matter ....Pages 63-63 Salient Object Subitizing (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 65-93 Unconstrained Salient Object Detection (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 95-111 Conclusion and Future Work (Jianming Zhang, Filip Malmberg, Stan Sclaroff)....Pages 113-114 Back Matter ....Pages 115-138