2014-04-08

1
U N I V E R S I T Y O F B R I T I S H C O L U M B I A
High Efficiency Video Coding
(HEVC)
Dr. Panos Nasiopoulos
Dr. Mahsa Pourazad

Introduction
! Motivations for developing of new video coding standard:
" Availability of advanced technologies for capturing displaying of ultra-
high definition (UHD) resolution video material.
Insufficient capacity to transmit large amounts of high definition (HD)
content - let alone UHD
" The need for coding efficiency superior to H.264/AVC
Increased use of parallel processing
! ISO/IEC Moving Picture Experts Group (MPEG) and
ITU-T Video Coding Experts Group (VCEG) established
a Joint Collaborative Team on Video Coding (JCT-VC)
for working on development of a new high-performance
video coding standard (50% bit rate reduction compared
to H.264/AVC with the same visual quality)

2
2014-04-08
2
Introduction (2)
! A formal Call for Proposals (CfP) on video compression
technology was issued in January 2010, and in response
27 proposals were received:
" Key features of seven top high-performing proposals into a
single Test Model under Consideration (TMuC), which became
the basis for the first HEVC reference encoder software (HM)[1].
! JCT-VC received hundreds of contributions from industry
and academia for further improvement of the HEVC test
model:
" Evaluation of submissions and including the best ones in the
HEVC standard.
! After 12 JCT-VC meetings, the first version of the
standard was completed and published in January 2013.
3
Overview
! Similar to H.264/AVC and most previous video coding
standards utilizes hybrid prediction and is Block-based.
! Several novel features:
" larger block sizes
" new quad-tree based block structure
" new tools for supporting parallel encoding and decoding
" improved motion information encoding,
" extended intra prediction modes,
" flexible transforms (different block sizes)
" extensive in-loop processing
4
2014-04-08
3
HEVC Encoder Block Diagram
5
Loop Filtering
Picture partitioning
6
! The basic block of HEVC is the Largest Coding Unit (LCU)
which is 64x64 (the same concept of macroblock).
! LCU can recursively split into 4 equally-sized Coding Units
(CUs)
" Content-adaptive coding tree structure comprised of CUs
" Luma coding blocks : 8x8 to 64x64
" Chroma coding blocks: 4x4 to 32x32
HEVC supports only 4:2:0
Partitioning of a LCU to different
sizes of CU (example)

2014-04-08
4
Picture partitioning(2)
7
Depth 0
Depth 1
Depth 2
Depth 3
! Sample on Quad-tree partitioning structure:
Prediction Units (PU)
8
! In the process of Inter and Intra prediction CUs can further split to smaller
units called Prediction Units (PUs):
! PUs can be as large as their CU root or as small as 4x4 block
! Splitting CUs to PUs is not recursive and it can only be done once
N/4 x N (R)
N x N N/2 x N N x N/2 N/2 x N/2
N/4 x N (L)
N x N/4 (U) N x N/4 (D)
N x N N/2 x N/2
! For inter prediction
N X N: the size of CU
! For intra prediction a CU
with the size of N X N can
split PUs:
2014-04-08
5
Transform Units
9
! Similar to H.264/AVC HEVC applies a DCT-like transform to
the residuals to de-correlate and compact the data.
! Transform Unit (TU) is the basic
unit for the transform and
quantization processes.
! Each coding block can recursively
split to quadrant
! Each CU may contain one or more
TUs.
! Except for 4x4 TUs the Chroma
TU size is always half of Luma TU
size (for 4x4 they have equal size)

CU Partitions
TU Partitions
Demonstration of HEVC Partitioning:
10
2014-04-08
6
Slices in H.264/AVC
11
! Silces in H.264/AVC support parallel encoding/
decoding and also error resiliency (independently
decoding without using information from any other
slices).
! Flexible Macroblock Ordering (FMO) tool: dividing
macroblocks into slices in a highly flexible manner
(not necessarily adjacent).
! Without FMO, slices consist of continuous
macroblocks in raster scan order.
! FMO improves error resiliency in high-loss
situations, at the cost of high complexity and lower
coding efficiency (prediction across slice boundaries
is disabled).
! Majority of H.264/AVC encoders do not support
FMO.
Without FMO
With FMO
Slices and Tile Structure in HEVC
12
! HEVC also supports slices, similar to slices in H.264/AVC without FMO.
! HEVC introduced tiles to support parallel processing, with more flexibility
than normal slices in H.264/AVC (lower complexity than FMO)
! Tiles are always rectangular.
Slice 3
Slice 1
Slice 2
Slice 4
! Each tile may contain more than
one slice.
! Each slice may contain more than
one tile (each tile must be entirely
within the slice not partially).
! In each tile, LCUs are processed in
a raster scan order.
! Tiles are processed in a raster scan
order within a picture.
Tile Boundaries
2014-04-08
7
Entropy Slices
13
! HEVC supports parallelization of data processing using multi-core CPU/
GPU architecture (for faster performance).
! Each slice can be divided into additional slices called entropy slices each
of which can be entropy decoded independently without reference to other
entropy slices.
! Each core of CPU handles the entropy decoding process (less decoding
time).
! Note that prior to reconstruction of entropy slices, decoded data needs to
be buffered, to support raster scan.
! Each picture contains both regular and entropy slices,
! Each regular slice may contain one or more entropy slices while an entropy
slice is contained in only one slice.
Wavefront Parallel Processing
14
! Slices and tiles provide support parallel encoding and decoding at the cost
of breaking prediction dependencies across boundaries
! Statistics used in entropy coding have to be initialized for every slice/tile.
Several LCUs have to be encoded for the entropy coder to learn the
statistics of the image content for efficient coding (causes inefficient
encoding in the first several LCUs)
! Solution: Wavefront parallel processing in HEVC (higher efficiency, less
error resilient)
2014-04-08
8
Wavefront Parallel Processing
15
! Wavefront processing is a way to achieve parallel encoding and
decoding without breaking prediction dependencies, and using as much
context as possible in entropy encoding.
! Start processing a new row of LCUs with a new parallel process (usually
a new thread) as soon as two LCUs have been processed in the row
above.
! For example, when thread 4 starts encoding its initial CABAC
parameters will depend upon the content of six previous LCUs, since
CABAC dependencies propagate down the first two columns.
1
2 2
3
4
1
Already encoded LCU
LCU currently being encoded by thread 1
CABAC dependency
and
Intra Prediction
! Intra-coded pictures are those without reference to other
pictures in the sequence.
! Intra prediction mode takes advantage of spatial
similarities between PUs.
! Compared to H.264/AVC, more intra prediction modes
are supported in HEVC.
16
2014-04-08
9
Intra Prediction(Contd)
HEVC Intra Prediction
Modes (35 Luma & 6
Chroma)
H.264/AVC Intra Prediction
Modes (9 Luma & 4
Chroma)
17
Intra Prediction: Intra Planar Mode
18
! It is used to predict and reconstruct smooth regions.
! Average of two linear interpolations(horizontal and
vertical).
! The right-column and the bottom-row are generated by
replication process from the right-top pixel and the left-
bottom pixel inside the current unit
2014-04-08
10
Intra Prediction: Chroma Modes (Contd)
! DM (Direct mode)
# it is selected if the texture directionality of luma and chroma
components looks similar. In this case it uses the exact same
mode as luma component.

! LM (Linear mode)
# It is selected if the sample intensity of luma and chroma
components is highly correlated. In this case, chroma
components are predicted using reconstructed luma components
by linear model relationship.
19
Inter Prediction
20
! A CU of size 2Nx2N can contain one PU of the same size, or it can be
split into two PUs of size Nx2N or 2NxN, or four PUs of size NxN
! Furthermore, inter-coded CUs can use Asymmetric Motion Partitions
(AMP), which allow for asymmetrical splitting of a CU into smaller PUs.
! AMP can be used on CUs of size 64x64 down to 16x16.
! AMP improves the coding efficiency, since it allows PUs to more
accurately conform to the shape of objects in the picture without requiring
further splitting.
2014-04-08
11
Interpolation filter for Quarter-sample-accurate motion
compensation in H.264/AVC
21
! Compensation for motion with fractional pel-accuracy requires
interpolation of the reference signal.
! In H.264/AVC first a 6-tap filter is applied for half-pel generation and then
a bi-linear interpolation of the integer and half pel values is used to
provide 1/4 pel accuracy interpolation.
Quarter-Samples Interpolation:
e=(A+a+1)/2
a3 = round( (5E - F + 20A + 20B - G + 5H)/32 )
b3 = round( (5M - N + 20A + 20C - O + 5P)/32 )
c1 = round( (5b1 - b2 + 20b3 + 20b4 - b5 + 5b6)/32 )
Interpolation filter for Quarter-sample-accurate motion
compensation in HEVC (Luma samples)
22
! Similar to H.264/AVC, the accuracy of motion compensation in HECV is
1/4 pel for luma samples.
! To obtain the non-integer luma samples, separable one-dimensional 8-
tap and 7-tap interpolation filters are applied horizontally and vertically to
generate luma half pel and quarter pel samples respectively .

2014-04-08
12
Interpolation filter for Quarter-sample-accurate motion
compensation in HEVC (Chroma samples)
23
! The prediction values for chroma components are similarly generated by
applying a onedimensional 4-tap DCT-based interpolation filter.
! Accuracy for chroma sample prediction is 1/8 of chroma samples.
Position Filter coefficients
1/8 {-2, 58, 10, -2}
2/8 {-4, 54, 16, -2}
3/8 {-6, 46, 28, -4}
4/8 {-4, 36, 36, -4}
5/8 {-4, 28, 46, -6}
6/8 {-2, 16, 54, -4}
7/8 {-2, 10, 58, -2}

Motion Parameter Encoding and Improved Skip Mode
(H.264/AVC)
24
! In H.264/AVC, motion vector (MV) coding is performed Spatially
" encoding the difference between the estimated MV and the predicted one,
where the predicted MV is the median of three surrounding MVs (to the left,
above and above-right).
! H.264/AVC has a SKIP mode, where no motion parameters or quantized
residuals are encoded in the bitstream, instead the motion parameters
are inferred from a co-located MB in the previous frame.
! A motion parameter set consists of a motion vector, reference picture
index, and reference picture usage flag.

2014-04-08
13
Motion Parameter Encoding and Improved Skip Mode
(HEVC)
25
! In HEVC, MVs can be predicted either spatially or temporally (using MVs
from previously coded pictures).
! Motion merge: A number of candidate motion-parameter sets are derived
from spatially and temporally close PUs and one is chosen by the
encoder and signalled in the bit-stream.
Motion Parameter Encoding and Improved Skip Mode
(HEVC)
26
! In HEVC, MVs can be predicted either spatially or temporally (using MVs
from previously coded pictures).
! Motion vector coding options:
" Explicit encoding of motion parameters (i.e. using motion vector prediction
and encoding the MV difference and reference picture index), same concept
as H.264/AVC , but with spatial & temporal candidates
" Motion merge mode (new feature)
" SKIP mode (which is closely related to motion merge).
2014-04-08
14
Motion Merge:
27
! Motion merge: A number of candidate motion-parameter sets are derived
from spatially and temporally close PUs . The chosen one by the encoder
is signalled in the bit-stream.
! Candidates for motion merge mode:
" Four spatial candidates (A1, B1, A0 and B0). If one or more of those PUs is
unavailable or intra coded B2 is also considered).
" One temporal candidate (co-located PU in the
picture closest temporally to the current one
that has already been encoded).
" zero-motion candidate
! Encoder signals which candidate is used, and the
motion parameter set for the current PU is copied
from the selected candidate.
! Note that motion merge avoids the need to encode a motion vector for
the PU, instead only the index of a candidate in the motion merge list is
encoded.
Skip Mode:
28
! HEVC SKIP mode: similar to motion merge mode in HEVC, the
encoder signals the index of a motion merge candidate and the
motion parameters for the current CU are copied from the selected
candidate
! Difference with Motion Merge: in SKIP mode no residual data is
encoded
" This allows areas of the picture that change very little between frames or
have constant motion to be encoded using very few bits.
2014-04-08
15
Transform (DCT-like):
29
! Similar to H.264/AVC, HEVC applies a DCT-like transform on the
prediction residual.
! HEVC includes transforms that can be applied to blocks of size 4x4 to
32x32 pixels .
basis vectors of the 4x4: {64, 64, 64, 64}
{83, 36,-36,-83}
{64,-64,-64, 64}
{36,-83, 83,-36}

8x8 transform: {64, 64, 64, 64, 64, 64, 64, 64}
{89, 75, 50, 18,-18,-50,-75,-89}
{83, 36,-36,-83,-83,-36, 36, 83}
{75,-18,-89,-50, 50, 89, 18,-75}
{64,-64,-64, 64, 64,-64,-64, 64}
{50,-89, 18, 75,-75,-18, 89,-50}
{36,-83, 83,-36,-36, 83,-83, 36}
{18,-50, 75,-89, 89,-75, 50,-18}

Transform (DST):
30
! HEVC also incorporates a discrete sine transform (DST) which is used
for blocks coded with some directional intra prediction modes (4x4
CUs).
! When using intra prediction, the pixels close to the ones used for
prediction (i.e. near the top or left boundaries) will usually be predicted
more accurately than the pixels further away.
! Residuals tend to be larger for pixels away from the boundaries.
! The DST will usually be better at encoding these kinds of residuals,
because the DST basis functions start low and increase, compared to
the DCT basis functions that start high and decrease .
2014-04-08
16
Transform:
31
! The transform design of HEVC has the following properties:
! 16 bit data representation before and after each transform stage
(independent of the internal bit depth)
! 16 bit multipliers for all internal multiplications, no need for correction
of different norms of basis vectors during quantization/dequantization
(unlike the integer transform in H.264/AVC)
! Coefficients and basis vectors of the smaller transforms are a subset
of coefficients and basis vectors of larger transforms (significant
savings in memory for storing transform matrix entries).
Loop Filtering
! Loop Fileting in HEVC consists of the following tools:
# Deblocking Filter
# Sample Adaptive Offset
! These techniques are intended to undo the distortion
introduced in the main steps of the encoding process
(prediction, transform and quantization).
! As it is part of the prediction loop, the pictures will serve
as better references for motion-compensated prediction,
since they have less encoding distortion.
32
2014-04-08
17
Deblocking Filter
! An in-loop deblocking filter is used similar to the one
used in H.264/AVC to reduce blocking artifact.

! block boundaries in HEVC
# Coding Units (CU) boundaries
# Prediction Units (PU) boundaries
# Transform Units (TU) boundaries
33
Deblocking Filter( Contd)
34
! based on the pixel gradients across the boundary and
thresholds derived based on the quantization parameter
in the blocks:

# whether the filter should be applied or not ,
# strength of the filter

is decided.

! Improvement in both the subjective and
objective quality of the video

2014-04-08
18
Sample Adaptive Offset (SAO)
! New coding tool introduced in HEVC
! It classifies reconstructed pixels into different categories
based on:
# intensity or
# edge properties

! It will then add an offset either of the following:
# Band Offset (BO)
# Edge Offset (EO)
to the pixels in each category to reduce distortion.
35
Band Offset (BO)
! Band offset (BO) classifies all pixels of a region into
multiple bands with pixels that are in the same intensity
interval.

! The intensity range is divided into 32 equal intervals
from zero to the maximum intensity. Therefore for 8-bit
data, bands will be 8 pixels wide.
36
2014-04-08
19
Band Offset (BO) (Contd)
! The 32 bands are divided into two groups.
# A group consisting the central 16 bands.
# A group consisting the rest 16 bands.
! The encoder decides which group of bands to apply
SAO to, so 16 offsets will be encoded in the bit stream.

! Example:
37
Edge Offset (EO)
! Edge Offset (EO) uses one of four one-dimensional 3-
pixel following patterns to classify pixels based on their
edge direction



! The encoder selects the pattern to be used in each
region by comparing each pixel with its two neighboring
pixels.
38
2014-04-08
20
Edge Offset (EO) (Contd)
! Each pixel can be classified as on of the following:

# peak (greater than two neighbors, category 4)
# valley (less than the two neighbors, category 1)
# edge (equal to one neighbor, categories 2 and 3)
# none of above.


! Four offset values will be calculated for these four
categories.

39
Category Condition
1 (valley)
P < two neighbors
2 (edge)
P < one neighbor && P == one neighbor
3 (edge)
P > one neighbor && P == one neighbor
4 (peak)
P > two neighbors
0
None of the above
Entropy coding (H.264/AVC):
40
! H.264/AVC Entropy Coding:
" Context-Adaptive Variable-Length Coding (CAVLC) is the base
" Context-Adaptive Binary Arithmetic Coding (CABAC) is optional (Main
and High profiles)
! CABAC can provide better coding efficiency than CAVLC, due to its
arithmetic coding engine and more sophisticated context modeling
(complicated probability model for bins of binarized symbol)
! CABAC increases the coding complexity specially at higher bitrates
where transform coefficient data has a dominant role in encoded
bitstreams).
2014-04-08
21
Entropy coding (HEVC):
41
! HEVC entropy coding utilizes the best features of both CABAC and
CAVLC coding (i.e., high efficiency and low complexity respectively)
through a harmonized design.
! HEVC s entropy coding engine is mainly based on CABAC, but with
improvement for high bit rates.
! HEVC entropy coding has 2 modes:
" High efficiency binarization (HEB): entirely CABAC
" High throughput binarization (HTB): all the syntax elements except
residual coefficients are coded using CABAC, and residual coefficients
are coded via CAVLC.

! The complexity of the HEVC entropy coding with HTB is very close to
CAVLC complexity since significant portion of the bitstream is coded
with bypassing CABAC.
Experimental Results:
42
! Comparing HEVC (HM5.1) with H.264/AVC standard (JM16.2
software).
2014-04-08
22
RD Curves:
43
Conclusion:
44
! Performance of the current design of HEVC is promising (29.14% to
46.73% bit rate saving)
! Subjective results confirmed the quality of HEVC encoded videos at
the same bit-rate is more than 50% better than the H.264-coded
videos (tests was perfomed by MPEG)
! MISSION ACCOMPLISHED$
2014-04-08
23
Conclusion:
45
! M. T. Pourazad, C. Doutre, M. Azimi, and P. Nasiopoulos, HEVC:
The New Gold Standard for Video Compression, IEEE Consumer
Electronic Magazine, vol.1, issue 3, pp. 36-46, July 2012.
! G.J. Sullivan, J. Ohm, J; H. Woo-Jin, and T. Wiegand, Overview of
the High Efficiency Video Coding (HEVC) Standard, IEEE Trans. on
Circuits and Systems for Video Technology, Vol. 22(12), pp. 1649
1668, Dec. 2012.