Deep Learning-Based Face Analytics (Advances in Computer Vision and Pattern Recognition)

This book provides an overview of different deep learning-based methods for face recognition and related problems. Specifically, the authors present methods based on autoencoders, restricted Boltzmann machines, and deep convolutional neural networks for face detection, localization, tracking, recognition, etc. The authors also discuss merits and drawbacks of available approaches and identifies promising avenues of research in this rapidly evolving field. Even though there have been a number of different approaches proposed in the literature for face recognition based on deep learning methods, there is not a single book available in the literature that gives a complete overview of these methods. The proposed book captures the state of the art in face recognition using various deep learning methods, and it covers a variety of different topics related to face recognition. This book is aimed at graduate students studying electrical engineering and/or computer science. Biometrics is a course that is widely offered at both undergraduate and graduate levels at many institutions around the world: This book can be used as a textbook for teaching topics related to face recognition. In addition, the work is beneficial to practitioners in industry who are working on biometrics-related problems. The prerequisites for optimal use are the basic knowledge of pattern recognition, machine learning, probability theory, and linear algebra. Contents 1 Deep CNN Face Recognition: Looking at the Past and the Future 1.1 Synonyms 1.2 Introduction 1.3 Datasets 1.4 Face Detection 1.5 Loss Functions 1.6 Face Verification and Identification Using CNNs 1.7 Open Problems References 2 Face Segmentation, Face Swapping, and How They Impact Face Recognition 2.1 Introduction 2.2 Related Work 2.2.1 Face Segmentation 2.2.2 Face Swapping 2.3 Swapping Faces in Unconstrained Images 2.3.1 Fitting 3D Face Shapes 2.3.2 Deep Face Segmentation 2.3.3 Face Swapping and Blending 2.4 Experiments 2.4.1 Face Segmentation Evaluations 2.4.2 Qualitative Face Swapping Results 2.4.3 Qualitative Ablation Study 2.4.4 Limitations of Our System 2.5 The effects of Swapping on Recognition 2.5.1 Face Verification System 2.5.2 Inter-Subject Swapping Verification Protocols 2.5.3 Inter-Subject Swapping Results 2.5.4 Intra-Subject Swapping Verification Protocols and Results 2.6 Conclusions References 3 Disentangled Representation Learning and Its Application to Face Analytics 3.1 Introduction 3.2 Application 1: Facial Landmark Tracking 3.2.1 Related Works 3.2.2 Our Approach: RED-Net 3.2.3 Experiments 3.3 Application 2: Learning Facial Representations for Inference and Generation 3.3.1 Related Works 3.3.2 Our Approach: CR-GAN 3.3.3 Results: Multi-view Facial Image Generation 3.3.4 Results: Conditional Facial Attribute Manipulation 3.4 Conclusion References 4 Learning 3D Face Morphable Model from In-the-Wild Images 4.1 Introduction 4.2 Prior Work 4.2.1 Linear 3DMM 4.2.2 Improving Linear 3DMM 4.2.3 2D Face Alignment 4.2.4 3D Face Reconstruction 4.2.5 Unsupervised Learning in 3DMM 4.3 The Proposed Nonlinear 3DMM 4.3.1 Conventional Linear 3DMM 4.3.2 Nonlinear 3DMM 4.4 Improving Model Fidelity 4.4.1 Nonlinear 3DMM with Proxy and Residual 4.4.2 Global–Local-Based Network Architecture 4.5 Experimental Results 4.5.1 Ablation Study 4.5.2 Expressiveness 4.5.3 Representation Power 4.5.4 Applications 4.6 Conclusions References 5 Deblurring Face Images Using Deep Networks 5.1 Deep Semantic Face Deblurring 5.2 Deblurring Via Structure Generation and Detail Enhancement 5.3 Uncertainty Guided Multi-stream Semantic Networks 5.3.1 Image Deblurring Network 5.3.2 Semantic Segmentation Network (SN) 5.3.3 Base Network (BN) 5.3.4 UMSN Network 5.3.5 Loss for UMSN 5.3.6 Uncertainty Guidance 5.3.7 Experimental Results 5.4 Conclusion References 6 Blind Super-resolution of Faces for Surveillance 6.1 Introduction 6.2 Related Works 6.3 Learning Invariant Features for Faces 6.4 Network Architecture 6.4.1 Encoder-Decoder 6.4.2 GAN for Feature Mapping 6.4.3 Loss Function 6.4.4 Training 6.5 Experiments 6.6 Conclusions References 7 Hashing A Face 7.1 Introduction 7.2 Unique Challenges of Hashing A Face 7.3 Strategies for Face Hashing 7.3.1 Data-Dependent Versus Data-Independent 7.3.2 Linear Versus Pivots-Based Hashing 7.3.3 Unsupervised Versus Supervised Hashing 7.3.4 Image Versus Set/Video Hashing 7.4 Face Recognition Tasks and Evaluation 7.4.1 Face Verification 7.4.2 Face Search 7.4.3 Evaluation Metrics 7.5 Face Datasets 7.5.1 IJB-A: IARPA Janus Benchmark A 7.5.2 IJB-B: IARPA Janus Benchmark B 7.5.3 IJB-C: IARPA Janus Benchmark C 7.5.4 UMD Faces 7.5.5 CASIA WebFace Dataset 7.6 Face Features 7.6.1 UMD Features: First Generation 7.6.2 UMD Features: Second Generation 7.6.3 UMD Features: Third Generation 7.7 Face Hashing Experiments 7.7.1 Experimental Settings 7.7.2 IJB-A 7.7.3 IJB-B 7.7.4 IJB-C 7.8 Open Issues 7.9 Conclusion References 8 Evolution of Newborn Face Recognition 8.1 Introduction 8.1.1 Biometric Modalities for Newborns 8.1.2 Characteristics and Challenges of Newborn Face Recognition 8.2 Datasets for Newborn Face Recognition 8.2.1 Newborns Face Database 8.2.2 Newborns, Infants, and Toddler Longitudinal Face Database 8.2.3 Children Multimodal Biometric Database (CMDB) 8.3 Existing Techniques for Newborn Face Recognition 8.3.1 Handcrafted Feature Extraction Methods 8.3.2 Autoencoder Learning-Based Method 8.3.3 Class-Based Penalty in CNN Filter Learning 8.3.4 Learning Structure and Strength of CNN Filters 8.4 Results and Analysis of Existing Newborn Face Recognition Techniques 8.5 Conclusion References 9 Deep Feature Fusion for Face Analytics 9.1 Introduction 9.2 Feature Aggregation for Face Recognition 9.2.1 Metadata-Based Feature Aggregator Network (M-FAN) 9.2.2 Architecture 9.2.3 Gradient Backpropagation 9.2.4 Batch Training 9.2.5 Experiment Setup 9.2.6 Results on IJB-A 9.2.7 Results on Janus CS4 9.3 Feature Enhancement for Facial Action Unit Recognition 9.3.1 Multi-modal Conditional Feature Enhancement (MCFE) 9.3.2 Feature Extraction 9.3.3 Deep Feature Enhancement 9.3.4 Training MCFE for AU Recognition 9.3.5 Datasets for Experimental Analysis 9.3.6 Experiment Settings 9.3.7 Results 9.4 Conclusion References 10 Deep Learning for Video Face Recognition 10.1 Introduction 10.2 Traditional Methods 10.3 Existing Deep Learning-based Approaches 10.3.1 Pairwise Distance-Based Methods 10.3.2 Pooling-Based Methods 10.4 Neural Aggregation Network 10.4.1 Feature Embedding Module 10.4.2 Aggregation Module 10.4.3 Network Training 10.5 Experiments 10.5.1 Training Details 10.5.2 Methods for Evaluation 10.5.3 Results on IJB-A Dataset 10.5.4 Results on YouTube Face dataset 10.5.5 Results on Celebrity-1000 Dataset 10.6 Conclusions References 11 Thermal-to-Visible Face Synthesis and Recognition 11.1 The Infrared Spectrum 11.2 GAN-Based Synthesis of Visible Faces From Thermal Faces 11.2.1 Generative Adversarial Networks (GANs) 11.2.2 GAN-Based Synthesis Network 11.3 Synthesis of High-Quality Visible Faces From Polarimetric Thermal Faces Using GANs 11.4 Thermal-to-Visible Face Verification Via Attribute-Preserved Synthesis 11.5 Self-attention Guided Synthesis 11.6 Conclusion References 12 Obstructing DeepFakes by Disrupting Face Detection and Facial Landmarks Extraction 12.1 Introduction 12.2 Background and Related Works 12.3 Attacking Face Detectors 12.3.1 White-Box Adversarial Perturbation Generation 12.3.2 Gray-Box Adversarial Perturbation Generation 12.3.3 Black-Box Adversarial Perturbation Generation 12.4 Attacking Facial Landmark Extractors 12.5 Experiments 12.5.1 Attacking Face Detection 12.5.2 Attacking Landmark Extractors 12.6 Conclusion References 13 Multi-channel Face Presentation Attack Detection Using Deep Learning 13.1 Introduction 13.2 Related Work 13.2.1 Feature-Based Approaches for Face PAD 13.2.2 CNN-Based Approaches for Face PAD 13.2.3 One-Class Models for Face PAD 13.2.4 Multi-channel-Based Approaches for Face PAD 13.2.5 Challenges in PAD 13.3 Proposed Method 13.3.1 Preprocessing 13.3.2 Network Architecture 13.3.3 One-Class Contrastive Loss (OCCL) 13.3.4 Implementation Details 13.4 Experiments 13.4.1 WMCA Dataset 13.4.2 MLFP Dataset 13.4.3 SiW-M Dataset 13.4.4 Evaluation Metrics 13.4.5 Baselines 13.4.6 Experiments and Results in WMCA Dataset 13.4.7 Experiments and Results in MLFP Dataset 13.4.8 Experiments and Results in SiW-M Dataset 13.4.9 Cross-Database Evaluations 13.5 Discussions 13.6 Conclusions References 14 Scalable Person Re-identification: Beyond Supervised Approaches 14.1 Introduction 14.2 Related Work 14.3 Optimal Subset Selection for Labeling 14.3.1 Problem Statement 14.3.2 Solution Overview 14.3.3 Sample Experimental Results 14.4 On-Boarding New Cameras through Transfer Learning 14.4.1 Problem Statement 14.4.2 Solution Overview 14.4.3 Sample Experimental Results 14.5 Conclusions References 15 Towards Causal Benchmarking of Bias in Face Analysis Algorithms 15.1 Introduction 15.2 Related Work 15.3 Face Attribute Annotation in Synthetic Images 15.4 Method 15.4.1 Transects: A Walk in Face Space 15.4.2 Analyses Using Transects 15.4.3 Human Annotation 15.5 Experiments 15.5.1 Gender Classifiers 15.5.2 Transect Data 15.5.3 Comparison of Transects to Real Face Datasets 15.5.4 Analysis of Bias 15.5.5 Regression Analysis 15.6 Discussion and Conclusions 15.6.1 Summary 15.6.2 Limitations and Future Work References 16 Strategies of Face Recognition by Humans and Machines 16.1 Introduction 16.2 Identification Accuracy: Human Face Recognition 16.2.1 Face Identification Performance of Untrained Humans 16.2.2 Performance of Trained Versus Untrained Humans 16.3 Identification Accuracy: Machines Versus Humans 16.4 Fusion 16.4.1 Crowd-Sourcing Methods 16.4.2 Fusion: Human Participants 16.4.3 Fusion: Humans and Machines 16.5 Strategic Differences: Forensic Facial Examiners Versus Untrained Humans 16.6 Other-Race Effects in Humans and Machines 16.6.1 Theories of the Other-Race Effect 16.6.2 Other-Race Effect in Machines 16.7 Closing Thoughts References 17 Evaluation of Face Recognition Systems 17.1 Introduction 17.1.1 Objectives of Evaluation 17.2 Verification 17.2.1 Metrics 17.2.2 Error Tradeoff Characteristics 17.2.3 Population-Specific Error Rates 17.2.4 Image-Specific Error Rates 17.2.5 Summary Statistics 17.3 Identification 17.3.1 Closed Versus Open-Universe Identification 17.3.2 Enrollment Gallery Composition 17.3.3 Database Segmentation 17.4 Computational Efficiency 17.5 Summary and Recommendations References Index