JVET HEVC: Ticket #1085: Insufficient bits for the result of luma quarter pel interpolation in HM code.
https://hevc.hhi.fraunhofer.de/trac/hevc/ticket/1085
<p>
The data type of the interpolated pixels in Short *dst is insufficient for a implementation of the filter conforming to JCTVC-L1003_v34
'short' can be 16 bits depending upon the platform and compiler used.
</p>
<p>
template<Int N, Bool isVertical, Bool isFirst, Bool isLast>
Void TComInterpolationFilter::filter(Int bitDepth, Short const *src, Int srcStride, Short *dst, Int dstStride, Int width, Int height, Short const *coeff)
</p>
<p>
Examples of case when 16 bits is insufficient is the quarter-sample location 'j' (8-209).
The maximum and minimum interpolated values of quarter-sample location 'j' depends on the range of 'b' (8-200).
</p>
<p>
Assume eight bit input. In equation (8-200)
The maximum of 'b' occurs when A(−3,0) = 0, A(−2,0) = 255, A(−1,0) = 0, A(0,0) = 255, A(1,0) = 255, A(2,0) =0, A(3,0) = 255, A(4,0) = 0. The maximum of 'b' is 88*255.
The minimum of 'b' occurs when A(−3,0) = 255, A(−2,0) = 0, A(−1,0) = 255, A(0,0) = 0, A(1,0) = 0, A(2,0) =255, A(3,0) = 0, A(4,0) = 255. The minimum of 'b' is -24*255.
</p>
<p>
For quarter-sample location 'j', in equation (8-209)
The maximum of 'j' occurs when b(0,-3) = -24*255, b(0,−2) = 88*255, b(0,−1) = -24*255, b(0,0) = 88*255, b(0,1) = 88*255, b(0,2) = -24*255, b(0,3) = 88*255, b(0,4) = -24*255. The maximum of 'j' is 33150
The minimum of 'j' occurs when b(0,-3) = 88*255, b(0,−2) = -24*255, b(0,−1) = 88*255, b(0,0) = -24*255, b(0,1) = -24*255, b(0,2) = 88*255, b(0,3) = -24*255, b(0,4) = 88*255. The minimum value is -16830.
</p>
<p>
The maximum of 'j' is greater than 32767(the maximum that can be represented by 16 bits). The current implementation of the filter does not confrom to the JCTVC-L1003_v34 draft of the standard when bi-prediction is used.
</p>
en-usTrac 1.0.18DefaultCC PluginFri, 03 May 2013 11:22:55 GMTcc set
https://hevc.hhi.fraunhofer.de/trac/hevc/ticket/1085#comment:1
https://hevc.hhi.fraunhofer.de/trac/hevc/ticket/1085#comment:1
<ul>
<li><strong>cc</strong>
<em>fbossen</em> <em>ksuehring</em> <em>davidf</em> <em>jct-vc@…</em> added
</li>
</ul>
TicketfbossenTue, 07 May 2013 14:02:25 GMTstatus changed; resolution set
https://hevc.hhi.fraunhofer.de/trac/hevc/ticket/1085#comment:2
https://hevc.hhi.fraunhofer.de/trac/hevc/ticket/1085#comment:2
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>invalid</em>
</li>
</ul>
<p>
Values are stored with an offset of -8192 such as to shift the -16380..33150 range to -24572..24958 which nicely fits into the signed 16-bit range.
</p>
Ticket