f_hwtransfer: fix best upload format selection

It is possible for a format to be supported by hardware but not by
av_hwframe_transfer for uploading. This breaks the best upload format
selection. We first select the hardware input format and then choose the
best software format that can be uploaded. In some cases, this may
result in a format that is not uploadable at all, leading to an error.
To solve this, we should avoid adding non-uploadable formats to the fmts
table.

For example, for format d3d11/yuv420p, av_hwframe_transfer_get_formats()
returns empty list of formats.
This commit is contained in:
Kacper Michajłow 2024-08-26 02:07:00 +02:00
parent bb0a852f56
commit 47cb503bf5
1 changed files with 2 additions and 1 deletions

View File

@ -486,7 +486,8 @@ static bool probe_formats(struct mp_filter *f, int hw_imgfmt, bool use_conversio
enum AVPixelFormat *fmts;
if (av_hwframe_transfer_get_formats(frames,
AV_HWFRAME_TRANSFER_DIRECTION_TO, &fmts, 0) >= 0)
AV_HWFRAME_TRANSFER_DIRECTION_TO, &fmts, 0) >= 0 &&
fmts[0] != AV_PIX_FMT_NONE)
{
int index = p->num_fmts;
MP_TARRAY_APPEND(p, p->fmts, p->num_fmts, imgfmt);