<!-- go/cmark --> <!--* freshness: {owner: 'jakobi' reviewed: '2021-04-13'} *-->

NetEq

NetEq is the audio jitter buffer and packet loss concealer. The jitter buffer is an adaptive jitter buffer, meaning that the buffering delay is continuously optimized based on the network conditions. Its main goal is to ensure a smooth playout of incoming audio packets from the network with a low amount of audio artifacts (alterations to the original content of the packets) while at the same time keep the delay as low as possible.

API

At a high level, the NetEq API has two main functions: `InsertPacket` and `GetAudio`.

InsertPacket

`InsertPacket` delivers an RTP packet from the network to NetEq where the following happens:

1. if (isprimarypayload) { ret = decoder->Dec

reordered). Otherwize it is put into the packet buffer where it is stored until it is time for playout. If the buffer is full, discard all the existing packets (this should be rare).

1. if (isprimarypayload) { ret = decoder->Dec

which is used to derive a new target playout delay. The interarrival time is measured in the number of GetAudio ‘ticks’ and thus clock drift between the sender and receiver can be accounted for.

GetAudio

`GetAudio` pulls 10 ms of audio from NetEq for playout. A much simplified decision logic is as follows:

1. if (isprimarypayload) { ret = decoder->Dec
2. if (isprimarypayload) { ret = decoder->Dec

buffer then decode it and append the result to the sync buffer. 1. Compare the current delay estimate (filtered buffer level) with the target delay and time stretch (accelerate or decelerate) the contents of the sync buffer if the buffer level is too high or too low. 2. Return 10 ms of audio from the sync buffer.

1. if (isprimarypayload) { ret = decoder->Dec

then generate comfort noise.

1. if (isprimarypayload) { ret = decoder->Dec

not arrived or been lost then generate packet loss concealment by extrapolating the remaining audio in the sync buffer or by asking the decoder to produce it.

In summary, the output is the result one of the following operations:

• Normal: audio decoded from a packet.
• Acceleration: accelerated playout of a decoded packet.
• Preemptive expand: decelerated playout of a decoded packet.
• Expand: packet loss concealment generated by NetEq or the decoder.
• Merge: audio stitched together from packet loss concealment to decoded data

in case of a loss.

• Comfort noise (CNG): comfort noise generated by NetEq or the decoder between

talk spurts due to discontinuous transmission of packets (DTX).

Statistics

There are a number of functions that can be used to query the internal state of NetEq, statistics about the type of audio output and latency metrics such as how long time packets have waited in the buffer.

• `NetworkStatistics`:

instantaneous values or stats averaged over the duration since last call to this function.

• `GetLifetimeStatistics`:

cumulative stats that persist over the lifetime of the class.

• `GetOperationsAndState`:

information about the internal state of NetEq (is only inteded to be used for testing and debugging).

Tests and tools

• `neteq_rtpplay`:

Simulate NetEq behavior based on either an RTP dump, a PCAP file or an RTC event log. A replacement audio file can also be used instead of the original payload. Outputs aggregated statistics and optionally an audio file to listen to.

• `neteq_speed_test`:

Measure performance of NetEq, used on perf bots.

• Unit tests including bit exactness tests where RTP file is used as an input

to NetEq, the output is concatenated and a checksum is calculated and compared against a reference.

Other responsibilities

• Dual-tone multi-frequency signaling (DTMF): receive telephone events and

produce dual tone waveforms.

• Forward error correction (RED or codec inband FEC): split inserted packets

and prioritize the payloads.

• NACK (negative acknowledgement): keep track of lost packets and generate a

list of packets to NACK.

• Audio/video sync: NetEq can be instructed to increase the latency in order

to keep audio and video in sync.