Versatile Video Coding (VVC) : Machine Learning and Heuristics

translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Foreword 6 Acknowledgements 8 Contents 9 1 Introduction 12 References 15 2 Versatile Video Coding (VVC) 18 2.1 Basic Video Coding Concepts 18 2.2 VVC: A Hybrid Video Encoder 19 2.3 VVC Frames Organization and Block Partitioning 21 2.4 VVC Encoding Tools 24 2.4.1 VVC Prediction Tools 25 2.4.2 VVC Residual Coding and Entropy Coding 27 2.4.3 VVC In-Loop Filters 28 2.5 VVC Common Test Conditions 29 References 31 3 VVC Intra-frame Prediction 34 3.1 Angular Intra-prediction 36 3.2 Multiple Reference Line Prediction 37 3.3 Matrix-Based Intra-prediction 38 3.4 Intra-sub-partition 39 3.5 Encoding of Chrominance CBs 40 3.6 Transform Coding 41 References 43 4 State-of-the-Art Overview 45 References 50 5 Performance Analysis of VVC Intra-frame Prediction 53 5.1 Methodology 53 5.2 VVC Versus HEVC: Intra-frame Compression Performance and Computational Effort Evaluation 54 5.3 VVC Intra-frame Computational Effort Distribution of Luminance and Chrominance Channels 55 5.4 VVC Intra-frame Block Size Analysis 56 5.5 VVC Intra-frame Encoding Mode Analysis 58 5.6 VVC Intra-frame Encoding Transform Analysis 61 5.7 Rate-Distortion and Computational Effort of VVC Intra-frame Coding Tools 66 5.8 General Discussion 70 References 71 6 Heuristic-Based Fast Multi-type Tree Decision Scheme for Luminance 72 6.1 Initial Analysis 72 6.2 Designed Scheme 74 6.3 Results and Discussion 75 References 78 7 Light Gradient Boosting Machine Configurable Fast Block Partitioning for Luminance 79 7.1 Background on LGBM Classifiers 79 7.2 Methodology 80 7.3 Features Analysis and Selection 82 7.4 Classifiers Training and Performance 83 7.5 Classifiers Integration 88 7.6 Results and Discussion 90 References 95 8 Learning-Based Fast Decision for Intra-frame Prediction Mode Selection for Luminance 97 8.1 Fast Planar/DC Decision Based on Decision Tree Classifier 98 8.2 Fast MIP Decision based on Decision Tree Classifier 99 8.3 Fast ISP Decision Based on the Block Variance 100 8.4 Learning-Based Fast Decision Design 102 8.5 Results and Discussion 102 References 104 9 Fast Intra-frame Prediction Transform for Luminance Using Decision Trees 106 9.1 Fast MTS Decision Based on Decision Tree Classifier 107 9.2 Fast LFNST Decision Based on Decision Tree Classifier 108 9.3 Fast Transform Decision Design 108 9.4 Results and Discussion 110 References 112 10 Heuristic-Based Fast Block Partitioning Scheme for Chrominance 113 10.1 Chrominance CB Splitting Early Termination Based on Luminance QTMT 114 10.2 Fast Chrominance Split Decision Based on Variance of Sub-blocks 115 10.3 Fast Block Partitioning Scheme for Chrominance Coding Design 118 10.4 Results and Discussion 121 References 124 11 Conclusions and Open Research Possibilities 125 Index 128 This book discusses the Versatile Video Coding (VVC), the ISO and ITU state-of-the-art video coding standard. VVC reaches a compression efficiency significantly higher than its predecessor standard (HEVC) and it has a high versatility for efficient use in a broad range of applications and different types of video content, including Ultra-High Definition (UHD), High-Dynamic Range (HDR), screen content, 360 videos, and resolution adaptivity. The authors introduce the novel VVC tools for block partitioning, intra-frame and inter-frames predictions, transforms, quantization, entropy coding, and in-loop filtering. The authors also present some solutions exploring VVC encoding behavior at different levels to accelerate the intra-frame prediction, applying statistical-based heuristics and machine learning (ML) techniques. This book includes: A high-level description of the VVC novel encoding tools; A detailed description of the VVC intra-frame prediction; A deep statistical assessment of the VVC intra-frame prediction behavior; Five algorithms to reduce the VVC intra-frame prediction encoding effort This book discusses the Versatile Video Coding (VVC), the ISO and ITU state-of-the-art video coding standard. VVC reaches a compression efficiency significantly higher than its predecessor standard (HEVC) and it has a high versatility for efficient use in a broad range of applications and different types of video content, including Ultra-High Definition (UHD), High-Dynamic Range (HDR), screen content, 360o videos, and resolution adaptivity. The authors introduce the novel VVC tools for block partitioning, intra-frame and inter-frames predictions, transforms, quantization, entropy coding, and in-loop filtering. The authors also present some solutions exploring VVC encoding behavior at different levels to accelerate the intra-frame prediction, applying statistical-based heuristics and machine learning (ML) techniques.