From 47cb503bf57b00d0170072a2fbf196303d69f30e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Michaj=C5=82ow?= Date: Mon, 26 Aug 2024 02:07:00 +0200 Subject: [PATCH] f_hwtransfer: fix best upload format selection It is possible for a format to be supported by hardware but not by av_hwframe_transfer for uploading. This breaks the best upload format selection. We first select the hardware input format and then choose the best software format that can be uploaded. In some cases, this may result in a format that is not uploadable at all, leading to an error. To solve this, we should avoid adding non-uploadable formats to the fmts table. For example, for format d3d11/yuv420p, av_hwframe_transfer_get_formats() returns empty list of formats. --- filters/f_hwtransfer.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/filters/f_hwtransfer.c b/filters/f_hwtransfer.c index 77dfaffe0d..2c578ba1e9 100644 --- a/filters/f_hwtransfer.c +++ b/filters/f_hwtransfer.c @@ -486,7 +486,8 @@ static bool probe_formats(struct mp_filter *f, int hw_imgfmt, bool use_conversio enum AVPixelFormat *fmts; if (av_hwframe_transfer_get_formats(frames, - AV_HWFRAME_TRANSFER_DIRECTION_TO, &fmts, 0) >= 0) + AV_HWFRAME_TRANSFER_DIRECTION_TO, &fmts, 0) >= 0 && + fmts[0] != AV_PIX_FMT_NONE) { int index = p->num_fmts; MP_TARRAY_APPEND(p, p->fmts, p->num_fmts, imgfmt);