<!-- go/cmark --> <!--* freshness: {owner: 'brandtr' reviewed: '2021-04-15'} *-->

Video coding in WebRTC

Introduction to layered video coding

[Video coding][video-coding-wiki] is the process of encoding a stream of uncompressed video frames into a compressed bitstream, whose bitrate is lower than that of the original stream.

Block-based hybrid video coding

All video codecs in WebRTC are based on the block-based hybrid video coding paradigm, which entails prediction of the original video frame using either [information from previously encoded frames][motion-compensation-wiki] or information from previously encoded portions of the current frame, subtraction of the prediction from the original video, and [transform][transform-coding-wiki] and [quantization][quantization-wiki] of the resulting difference. The output of the quantization process, quantized transform coefficients, is losslessly [entropy coded][entropy-coding-wiki] along with other encoder parameters (e.g., those related to the prediction process) and then a reconstruction is constructed by inverse quantizing and inverse transforming the quantized transform coefficients and adding the result to the prediction. Finally, in-loop filtering is applied and the resulting reconstruction is stored as a reference frame to be used to develop predictions for future frames.

Frame types

When an encoded frame depends on previously encoded frames (i.e., it has one or more inter-frame dependencies), the prior frames must be available at the receiver before the current frame can be decoded. In order for a receiver to start decoding an encoded bitstream, a frame which has no prior dependencies is required. Such a frame is called a "key frame". For real-time-communications encoding, key frames typically compress less efficiently than "delta frames" (i.e., frames whose predictions are derived from previously encoded frames).

Single-layer coding

In 1:1 calls, the encoded bitstream has a single recipient. Using end-to-end bandwidth estimation, the target bitrate can thus be well tailored for the intended recipient. The number of key frames can be kept to a minimum and the compressability of the stream can be maximized. One way of achiving this is by using "single-layer coding", where each delta frame only depends on the frame that was most recently encoded.

Scalable video coding

In multiway conferences, on the other hand, the encoded bitstream has multiple recipients each of whom may have different downlink bandwidths. In order to tailor the encoded bitstreams to a heterogeneous network of receivers, [scalable video coding][svc-wiki] can be used. The idea is to introduce structure into the dependency graph of the encoded bitstream, such that layers of the full stream can be decoded using only available lower layers. This structure allows for a [selective forwarding unit][sfu-webrtc-glossary] to discard upper layers of the of the bitstream in order to achieve the intended downlink bandwidth.

There are multiple types of scalability:

• Temporal scalability are layers whose framerate (and bitrate) is lower than that of the upper layer(s)
• Spatial scalability are layers whose resolution (and bitrate) is lower than that of the upper layer(s)
• Quality scalability are layers whose bitrate is lower than that of the upper layer(s)

WebRTC supports temporal scalability for VP8, VP9 and AV1, and spatial scalability for VP9 and AV1.

Simulcast

Simulcast is another approach for multiway conferencing, where multiple independent bitstreams are produced by the encoder.

In cases where multiple encodings of the same source are required (e.g., uplink transmission in a multiway call), spatial scalability with inter-layer prediction generally offers superior coding efficiency compared with simulcast. When a single encoding is required (e.g., downlink transmission in any call), simulcast generally provides better coding efficiency for the upper spatial layers. The K-SVC concept, where spatial inter-layer dependencies are only used to encode key frames, for which inter-layer prediction is typically significantly more effective than it is for delta frames, can be seen as a compromise between full spatial scalability and simulcast.

Overview of implementation in `modules/video_coding`

Given the general introduction to video coding above, we now describe some specifics of the [modules/video_coding][modules-video-coding] folder in WebRTC.

Built-in software codecs in [`modules/video_coding/codecs`][modules-video-coding-codecs]

This folder contains WebRTC-specific classes that wrap software codec implementations for different video coding standards:

• [libaom][libaom-src] for [AV1][av1-spec]
• [libvpx][libvpx-src] for [VP8][vp8-spec] and [VP9][vp9-spec]
• [OpenH264][openh264-src] for [H.264 constrained baseline profile][h264-spec]

Users of the library can also inject their own codecs, using the [VideoEncoderFactory][video-encoder-factory-interface] and [VideoDecoderFactory][video-decoder-factory-interface] interfaces. This is how platform-supported codecs, such as hardware backed codecs, are implemented.

Video codec test framework in [`modules/video_coding/codecs/test`][modules-video-coding-codecs-test]

This folder contains a test framework that can be used to evaluate video quality performance of different video codec implementations.

SVC helper classes in [`modules/video_coding/svc`][modules-video-coding-svc]

• [ScalabilityStructure*][scalabilitystructure] - different

[standardized scalability structures][scalability-structure-spec]

• [ScalableVideoController][scalablevideocontroller] - provides instructions to the video encoder how

to create a scalable stream

• [SvcRateAllocator][svcrateallocator] - bitrate allocation to different spatial and temporal

layers

Utility classes in [`modules/video_coding/utility`][modules-video-coding-utility]

• [FrameDropper][framedropper] - drops incoming frames when encoder systematically

overshoots its target bitrate

• [FramerateController][frameratecontroller] - drops incoming frames to achieve a target framerate
• [QpParser][qpparser] - parses the quantization parameter from a bitstream
• [QualityScaler][qualityscaler] - signals when an encoder generates encoded frames whose

quantization parameter is outside the window of acceptable values

• [SimulcastRateAllocator][simulcastrateallocator] - bitrate allocation to simulcast layers

General helper classes in [`modules/video_coding`][modules-video-coding]

• [FecControllerDefault][feccontrollerdefault] - provides a default implementation for rate

allocation to [forward error correction][fec-wiki]

• [VideoCodecInitializer][videocodecinitializer] - converts between different encoder configuration

structs

Receiver buffer classes in [`modules/video_coding`][modules-video-coding]

• [PacketBuffer][packetbuffer] - (re-)combines RTP packets into frames
• [RtpFrameReferenceFinder][rtpframereferencefinder] - determines dependencies between frames based on information in the RTP header, payload header and RTP extensions
• [FrameBuffer][framebuffer] - order frames based on their dependencies to be fed to the decoder

[video-coding-wiki]: https://en.wikipedia.org/wiki/Videocodingformat [motion-compensation-wiki]: https://en.wikipedia.org/wiki/Motion_compensation [transform-coding-wiki]: https://en.wikipedia.org/wiki/Transform_coding [motion-vector-wiki]: https://en.wikipedia.org/wiki/Motion_vector [mpeg-wiki]: https://en.wikipedia.org/wiki/MovingPictureExperts_Group [svc-wiki]: https://en.wikipedia.org/wiki/ScalableVideoCoding [sfu-webrtc-glossary]: https://webrtcglossary.com/sfu/ [libvpx-src]: https://chromium.googlesource.com/webm/libvpx/ [libaom-src]: https://aomedia.googlesource.com/aom/ [openh264-src]: https://github.com/cisco/openh264 [vp8-spec]: https://tools.ietf.org/html/rfc6386 [vp9-spec]: https://storage.googleapis.com/downloads.webmproject.org/docs/vp9/vp9-bitstream-specification-v0.6-20160331-draft.pdf [av1-spec]: https://aomediacodec.github.io/av1-spec/ [h264-spec]: https://www.itu.int/rec/T-REC-H.264-201906-I/en [video-encoder-factory-interface]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/api/videocodecs/videoencoderfactory.h;l=27;drc=afadfb24a5e608da6ae102b20b0add53a083dcf3 [video-decoder-factory-interface]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/api/videocodecs/videodecoderfactory.h;l=27;drc=49c293f03d8f593aa3aca282577fcb14daa63207 [scalability-structure-spec]: https://w3c.github.io/webrtc-svc/#scalabilitymodes* [fec-wiki]: https://en.wikipedia.org/wiki/Errorcorrectioncode#Forwarderrorcorrection [entropy-coding-wiki]: https://en.wikipedia.org/wiki/Entropy_encoding [modules-video-coding]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/ [modules-video-coding-codecs]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/codecs/ [modules-video-coding-codecs-test]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/codecs/test/ [modules-video-coding-svc]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/svc/ [modules-video-coding-utility]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/utility/ [scalabilitystructure]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/svc/createscalabilitystructure.h?q=CreateScalabilityStructure [scalablevideocontroller]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/svc/scalablevideocontroller.h?q=ScalableVideoController [svcrateallocator]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/svc/svcrateallocator.h?q=SvcRateAllocator [framedropper]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/utility/frame_dropper.h?q=FrameDropper [frameratecontroller]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/utility/framerate_controller.h?q=FramerateController [qpparser]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/utility/qp_parser.h?q=QpParser [qualityscaler]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/utility/quality_scaler.h?q=QualityScaler [simulcastrateallocator]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/utility/simulcastrateallocator.h?q=SimulcastRateAllocator [feccontrollerdefault]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/feccontrollerdefault.h?q=FecControllerDefault [videocodecinitializer]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/include/videocodecinitializer.h?q=VideoCodecInitializer [packetbuffer]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/packet_buffer.h?q=PacketBuffer [rtpframereferencefinder]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/modules/videocoding/rtpframereference_finder.h?q=RtpFrameReferenceFinder [framebuffer]: https://source.chromium.org/chromium/chromium/src/+/main:thirdparty/webrtc/api/video/framebuffer.h [quantization-wiki]: https://en.wikipedia.org/wiki/Quantization(signalprocessing)