avcodec: Don't assume separate u and v planes in ff_alloc_picture

alloc_frame_buffer in ff_alloc_picture asserts that the linesize
of planes 1 and 2 are the same. If the pixfmt has a single uv
plane, like NV12, this won't be true.

So, let's only do this check if there are more than 2 planes.

We never hit this with previous hw formats because they don't set
linesize to meaningful values, but the cuda hw format sets the
values based on the underlying data layout.
This commit is contained in:
Philip Langdale 2017-11-13 21:11:27 -08:00
parent 54f8ac199f
commit bb4c9d0a8e
1 changed files with 3 additions and 1 deletions

View File

@ -22,6 +22,7 @@
#include "libavutil/avassert.h"
#include "libavutil/common.h"
#include "libavutil/pixdesc.h"
#include "avcodec.h"
#include "motion_est.h"
@ -155,7 +156,8 @@ static int alloc_frame_buffer(AVCodecContext *avctx, Picture *pic,
return -1;
}
if (pic->f->linesize[1] != pic->f->linesize[2]) {
if (av_pix_fmt_count_planes(pic->f->format) > 2 &&
pic->f->linesize[1] != pic->f->linesize[2]) {
av_log(avctx, AV_LOG_ERROR,
"get_buffer() failed (uv stride mismatch)\n");
ff_mpeg_unref_picture(avctx, pic);