avcodec/videotoolboxenc: fix vtctx reset condition

In vtenc_populate_extradata, the cleanup function vtenc_reset should not
be used when no error occurs, otherwise some color information is lost
(#11036).

This patch checks the status code and conducts the correct cleanup.

Signed-off-by: Hao Guan <hguandl@gmail.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
This commit is contained in:
Hao Guan 2024-06-21 14:08:20 +08:00 committed by Zhao Zhili
parent 24b3bad811
commit cd2f8a22e9
1 changed files with 8 additions and 1 deletions

View File

@ -2767,7 +2767,14 @@ static int vtenc_populate_extradata(AVCodecContext *avctx,
pe_cleanup:
CVPixelBufferRelease(pix_buf);
vtenc_reset(vtctx);
if (status) {
vtenc_reset(vtctx);
} else if (vtctx->session) {
CFRelease(vtctx->session);
vtctx->session = NULL;
}
vtctx->frame_ct_out = 0;
av_assert0(status != 0 || (avctx->extradata && avctx->extradata_size > 0));