Understanding FFmpeg with source code: FFMPEG Fundementals

In this book, we will review the ffmpeg source code and learn about ffmpeg's capabilities. In the first part, we will consider ffmpeg in general terms. In the second part, we will discuss the ffmpeg methods related to video. In this section, we will also examine the ffmpeg filters closely. In the third part, we will learn the details and methods of audio. In the fourth and fifth part, we will learn about Interlaced video and I-B-P frames. In the sixth part, we will learn what is codec and how does ffmpeg handle it. In the seventh part, we will try to explain how video encoding and audio encoding take place in ffmpeg, with sample codes. In the eighth part, we will try to explain how decoding works in fmmpeg and how to make a sample player with the help of SDL. In parts 9, 10 and 11, we will show the video and audio multiplexing methods. In the twelfth part, how to convert a video file into another format. We will learn all the transcoding steps. In the thirteenth part, we will learn all streaming protocols. In the last part, we will learn how to manage video and audio devices with ffmpeg. UNDERSTANDING FFMPEG WITH SOURCE CODE 2 Acknowledgements 3 Table of Contents 4 Introduction 15 1. FFMPEG Fundementals 17 FFMPEG Data Structure 19 AVCodecContext 21 AVStream 26 AVFormatContext 30 AVPacket 37 AVIOContext 43 URLContext 47 URLProtocol 48 Time information 49 There are two-way prediction frames: 51 Time information acquisition 52 APIs 54 Reading series 55 Codec series 62 Write series 66 Other 69 Several libavutil tools 70 AVOption (AVClass) 73 AVDictionary 78 ParseUtil 80 2. Video and Image basics 82 Raw Format 84 FOURCC 86 FOURCC Identifier 87 YUV Pixel Formats 88 Packed YUV Formats 90 Planar YUV Formats 94 RGB Pixel Formats 96 Bayer Data Formats 98 Create Sample Video 99 Convert Packet Format to Planar Format 104 Image Scale 109 Filter Chain 117 AVFilter 119 AVFilterPad 120 Picture Buffers 122 Permissions 123 Filter Links 125 Usage of Avfilter 128 Color Change Rectangular Area 141 Output: 143 Watermark 144 input -----------> deltapts0 --> overlay --> output 146 movie --> scale--> deltapts1 -------+ 147 Output: 148 Color Effects 149 Boxblur 150 Hue 152 Output: 154 lut, lutrgb, lutyuv 155 ‘E, PI, PHI’ 157 Crop Video 159 Draw Box on Video 162 Output: 163 Remove Logo From Video 164 Output: 166 Draw Grid on Video 167 Output: 170 Fade Effect on Video 171 Output: 172 Horizontal Flip Video 173 Output: 174 Add Noise on Video 175 Rotate Video 177 Output: 180 Trim Video 181 All Other Filters In Sample 183 Write Your Own Filter (WYOF) 188 WYOF Filter Structure 197 WYOF filter_frame () call flow 202 WYOF decode_video //ffmpeg.c 203 WYOF av_buffersrc_add_frame_flags//buffersrc.c 205 WYOF av_buffersrc_add_frame_internal //buffersrc.c 206 WYOF request_frame //buffersrc.c 207 WYOF ff_filter_frame // avfilter.c 208 WYOF ff_filter_frame_framed //avfilter.c 209 WYOF Extract the Core Code. 210 WYOF filter_frame //vf_transform.c 211 WYOF Walk into ff_filter_frame again // avfilter.c 213 WYOF Third time into ff_filter_frame // avfilter.c 215 WYOF How to encode ffmpeg after filter 217 WYOF do_video_out //ffmpeg.c 220 WYOF Function Flowchart 221 Metadata 223 Read Metadata Sample 225 Write Metadata Sample 226 3. Audio Basics 229 Audio introduction 230 Mono 231 Stereo 234 3D Surround 236 Number of Samples 237 Frequency of Sampling 238 Bit Speed 239 VBR (Variable Bitrate) 241 CBR (Constant Bitrate) 243 MP3 244 Description of Audio Formats 246 PCM data format 249 PCM stream 251 PCM audio processing 253 1. Separate left and right channel audio 254 The audio waveform of the right channel is as follows: 257 2. Audio pcm resampling 48000 to 44100 258 3. Using ffmpeg to construct silent frames 265 4. Halve the volume of the left channel in the PCM16LE two-channel audio sample data 267 5. Double the sound speed of PCM16LE two-channel audio sampling data 270 6. Convert PCM16LE two-channel audio sampling data to PCM8 audio sampling data 273 7. A part of data will be intercepted from PCM16LE mono audio sampling data 277 8. Convert PCM16LE two-channel audio sampling data to WAVE format audio data 281 Basic Audio Effects-Volume Control 286 How to change PCM Sample Rate 290 How to change the bit depth 291 4. Interlaced Video 296 Why TV Interlaced ? 297 Development and Evolution of Display Technology 302 2x Field Frequency Display 303 2x Horizontal Frequency Display 304 Progressive Display 305 Multi-format Receiving Single Format Display 307 Flat Display Device 308 Psf-Progressive Segmentation 310 Progressive Format for Broadcast 313 Progressive Scan Format for Production and Exchange 314 Progressive Scan Format for Distribution 319 Digital Video Interface 321 Digital Display Interface 324 DVI 325 HDMI 328 Analog Interface 329 Progressive Scan DVD 331 Interconversion of Progressive and Interlaced Signals 336 Digital TV Trends-Progressive, Interlaced or Progressive 340 HD Production Choice: 50i or 25P? 342 Can I Use 50i Interlaced Scanning Equipment to Copy and Make 25Psf Progressive Scanning Programs? 344 Deinterlacing 348 5. I-B-P Frame and Timebase 353 I-Frame 354 P-Frame 355 B-frame 356 DTS and PTS 357 Time Base and Timestamp in FFmpeg 359 1. The Concept of Time Base and Timestamp 360 2. Three Time Bases tbr, tbn and tbc 361 3. Internal Time Base AV_TIME_BASE 363 4. Time Value Form Conversion 364 5 Time Base Conversion Function 365 6 Time-Based Conversion During the Encapsulation Process 366 7 Time Base Conversion During Transcoding 369 7.1 Video Stream 371 7.2 Audio Stream 373 6. Codecs 375 Video Codec 376 List of Open-source Codecs 378 Video Codec List 379 Audio Codec List 381 FFMPEG libavcodec 384 libavcodec/cook.c 385 libavcodec/avcodec.h 386 libavcodec/allcodecs.c 387 libavcodec/cook.c Init 388 libavcodec/cook.c Close 389 libavcodec/cook.c Decode 390 7. Encoding 392 Video Encoding 393 1. FFMpeg Required Structure for Video Encoding 394 2. The Main Steps of FFMpeg Encoding 396 (2) Initialize the Required FFMpeg Structure as Required 397 (3), Encoding Loop Body 402 (4) Finishing Treatment 406 H.264 Encoding 409 H265 Encoding 415 VP8 Encoding 422 Audio Encoding 429 AAC Encoding 430 Picture Encoding 450 8. Decoding 455 1. Connect and Open the Video Stream 458 2. Locate Video Stream Data 460 3. Prepare the Decoder Codec 462 4. Decode 464 Simple Decoder Based on FFmpeg. 466 Simple Video Player Based on FFMPEG + SDL 475 SDL2.0 shows the flow chart of YUV 477 Simple Audio Player Based on FFMPEG + SDL 494 9. Multiplexing 501 10. Demultiplexing 514 Demultiplexing to H264 and Mp3 527 11. Remultiplexing 539 12. Transcoding 549 Whole process 550 Demultiplexing 551 Decoding 552 Filter 553 Encoding 554 Reuse 555 Time Stamp Processing During Transcoding 558 13. Streaming 580 1. RTMP 583 1.1. General Introduction 584 1.2. Handshake 586 1.3 RTMP Chunk Stream 588 1.3.1 Message 589 1.3.2 Chunking 590 1.4. Different Types of RTMP Message 593 1.5 Push RTMP Streamer Sample 596 1.5.1 Introduction 598 1.5.2 Need to Pay Attention to Package Format 599 1.5.3 Delay 600 1.5.4 PTS / DTS Issue 602 1.5.5 Flowchart 603 1.6 Save RTMP Streaming Media as a Local FLV File 610 2. UDP (User Data Protocol) 616 2.1. Multicasting (Multicasting) 618 2.2. UDP Broadcast 620 2.3. UDP Multicast 621 2.4. UDP Broadcast and Unicast 622 2.5. Pull RTMP Stream to UDP Output Playback 625 2.6. UDP Receiving TS Stream 632 3. RTP (Real-Time Transport Protocol) 637 3.1. RTP Working Mechanism 638 4. RTCP (RTP Control Protocol) 641 4.1. RTCP Working Mechanism 643 5. RTSP (RealTime Streaming Potocol) 645 Analysis of MPEG2 Transport Stream (MPEG2 TS) 652 14. Devices 685 Read camera 687 Screen capture 698 2. The dshow 701 References 711 About The Author 713