This reverts commit d25f87f517. This breaks decoding of some h264 files I have tested the original patch with fate but by mistake have forgotten to specify the fate samples so testing was limited to the internal regression tests.
d25f87f517