We generally try to avoid that due to the #ifdef mess.
The equivalent format is defined in ffmpeg 4.4 while our interop code
requires a higher version, but that doesn't cause any problems.
It's conceptually broken for an hw format to be identified as a
supported sw format for an hwdec, but in the case of the drm hwdec,
we have no choice but to do so for the weird rpi formats, as these
are declared, incorrectly, as hw formats in their forked ffmpegs.
This means we can't reject such formats as a matter of principle,
as we do today. Instead let's just assume that such formats can always
be accepted as-is, and will never require conversion. In practice, this
is either true or it will fail in the VO, which is the old behaviour
from before I introduced the conversion filter capability.
Some hwdecs (eg: dxva) have no frames constraints and no static
supported_formats, which ends up segfaulting. This change fixes the
segfault, but also includes additional changes to avoid concluding that
direct output of hardware decoded video is impossible.
In the case where there are no frame constraints and no
supported_formats, we basically have no idea what the hardware actually
supports. Previously, we just tried to display the frame, but all the
work I did to detect incompatible formats causes this scenario to now
conclude that no formats can be displayed, and forces a HW download to
system memory.
I have made a couple of small changes to assume that direct display is
possible. If it's not, the VO will error out down the line, which is
what happened previously.
Some ffmpeg hwcontexts do not implement frame constraints. That means
they cannot report which formats are support for given configurations.
My previous changes to make hwupload more capable relies on the
constraints to be provided to accurately describe what combinations are
supported, so we will currently see a bunch of errors failing to
configure the hwupload filter on platforms such as v4l2 SoCs with
drmprime. This doesn't break playback, but it's confusing.
To bridge the gap, this change uses the static format metadata that we
include for hwdecs to build a fake constraints struct to keep all the
other code working.
This may seem obvious in retrospect, but we need to explicitly account
for the case where the source format is supported by the VO, but not
a valid target format for the conversion filter. In that situation, we
would conclude that conversion was necessary, because we relied solely
on the conversion filter to identify acceptable target formats.
To avoid that, we should go through the VO's reported set of supported
formats and if the source format is on the list, ensure that format is
also on the target list.
This is mostly a no-op as most VOs do not report supported formats
(instead assuming that all formats decoders can produce are supported)
and in the case where it matters (vaapi), there is only one format that
the VO supports which the conversion filter does not (yuv444p).
The last piece in the puzzle for doing hardware conversions
automatically is ensuring we only consider valid target formats for the
conversion. Although it is unintuitive, some vaapi drivers can expose a
different set of formats for uploads vs for conversions, and that is
the case on the Intel hardware I have here.
Before this change, we would use the upload target list, and our
selection algorithm would pick a format that doesn't work for
conversions, causing everything to fail. Whoops.
Successfully obtaining the conversion target format list is a bit of a
convoluted process, with only parts of it encapsulated by ffmpeg.
Specifically, ffmpeg understands the concept of hardware configurations
that can affect the constraints of a device, but does not define what
configurations are - that is left up to the specific hwdevice type.
In the case of vaapi, we need to create a config for the video
processing endpoint, and use that when querying for constraints.
I decided to encapsulate creation of the config as part of the hwdec
init process, so that the constraint query can be down in the
hwtransfer code in an opaque way. I don't know if any other hardware
will need this capability, but if so, we'll be able to account for it.
Then, when we look at probing, instead of checking for what formats
are supported for transfers, we use the results of the constraint query
with the conversion config. And as that config doesn't depend on the
source format, we only need to do it once.
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
Although we can support vulkan multiplane images, cuda lacks any such
support, and so cannot natively import such images for interop. It's
possible that we can do separate exports for each plane in the image
and have it work, but for now, we can selectively disable multiplane
when we know that we'll be consuming cuda frames.
As a reminder, even though cuda is the frame source, interop is one way
so the vulkan images have to be imported to cuda before we copy the
frame contents over.
This logic here is slightly more complex than I'd like but you can't
just set the flag blindly, as it will cause hwframes ctx creation to
fail if the format is packed or if it's planar rgb. Oh well.
Vulkan Video Decoding has finally become a reality, as it's now
showing up in shipping drivers, and the ffmpeg support has been
merged.
With that in mind, this change introduces HW interop support for
ffmpeg Vulkan frames. The implementation is functionally complete - it
can display frames produced by hardware decoding, and it can work with
ffmpeg vulkan filters. There are still various caveats due to gaps and
bugs in drivers, so YMMV, as always.
Primary testing has been done on Intel, AMD, and nvidia hardware on
Linux with basic Windows testing on nvidia.
Notable caveats:
* Due to driver bugs, video decoding on nvidia does not work right now,
unless you use the Vulkan Beta driver. It can be worked around, but
requires ffmpeg changes that are not considered acceptable to merge.
* Even if those work-arounds are applied, Vulkan filters will not work
on video that was decoded by Vulkan, due to additional bugs in the
nvidia drivers. The filters do work correctly on content decoded some
other way, and then uploaded to Vulkan (eg: Decode with nvdec, upload
with --vf=format=vulkan)
* Vulkan filters can only be used with drivers that support
VK_EXT_descriptor_buffer which doesn't include Intel ANV as yet.
There is an MR outstanding for this.
* When dealing with 1080p content, there may be some visual distortion
in the bottom lines of frames due to chroma scaling incorporating the
extra hidden lines at the bottom of the frame (1080p content is
actually stored as 1088 lines), depending on the hardware/driver
combination and the scaling algorithm. This cannot be easily
addressed as the mechanical fix for it violates the Vulkan spec, and
probably requires a spec change to resolve properly.
All of these caveats will be fixed in either drivers or ffmpeg, and so
will not require mpv changes (unless something unexpected happens)
If you want to run on nvidia with the non-beta drivers, you can this
ffmpeg tree with the work-around patches:
* https://github.com/philipl/FFmpeg/tree/vulkan-nvidia-workarounds
This mapping isn't actually relevant until we have the Vulkan interop
merged, and it requires a newer version of libavutil than our minimum
requirement. So I'm going to remove it from master and put it in the
interop PR.
Fixes#10813
Wayland VO that can display images from either vaapi or drm hwdec
The PR adds the following changes:
1. a context_wldmabuf context with no gl dependencies
2. no-op ra_wldmabuf and dmabuf_interop_wldmabuf objects
no-op because there is no need to map/unmap the drmprime buffer,
and there is no need to manage any textures.
Tested on both x86_64 and rk3399 AArch64
vaapi allows for implicit conversion on upload, which has some
relevance as the set of supported source formats is larger than the
set of displayable formats. In theory, this allows for offloading the
conversion to the GPU - if you have any confidence in the hardware
and/or driver's ability to do the conversion.
Today, we actually track the 'input' and 'output' upload formats
separately all the way up until the point we do a check as to whether
the original source format is an accepted 'output' format and then
reject it if it is not.
This means that we're essentially ignoring all the work we did to track
those 'input' formats in the first place. But it also means that it's a
simple change to compare against the 'input' format instead. The logic
is already in place to do best format selection on both sides.
I imagine that if I read through the history here, wm4 tried to
implement all of this properly and then gave up in disgust after seeing
vaapi mangle various conversions.
This is particularly interesting for vo-dmabuf-wayland where it is only
possible to display the subset of valid vaapi formats that are
supported by the compositor, yet all playback has to go through vaapi.
Users will then be able to take advantage of all possible vaapi formats
to avoid having to do software format conversion.
A few years ago, wm4 got sufficiently annoyed with how vaapi image
format support was being discovered that he flipped the table and
introduced the shit list (which just included vaapi) to hard-code the
set of supported formats.
While that might have been necessary at the time, I haven't been able
to find a situation where the true list of supported formats was unsafe
to use. We filter down the list based on what the vo reports - and the
vo is already doing a thorough testing of formats, and if a format
makes it through that gauntlet, it does actually work.
Interestingly, as far as I can tell, the hwdec_vaapi probing code was
already good enough at the time (also written by wm4), so perhaps the
key difference here is that the driver side of things has improved.
I dug into this because of the support for the 422/444 high bit depth
vaapi formats I added to ffmpeg. These are obviously not in the hard
coded list today, but they work fine.
Finally, although it's positioned as a vaapi thing, it's really just
Intel specific, as the AMD vaapi driver has never exposed support for
anything except the formats used by the decoder/encoder profiles.
Certain combinations of hardware formats require the use of hwmap to
transfer frames between the formats, rather than hwupload, which will
fail if attempted.
To keep the usage of vf_format for HW -> HW transfers as intuitive as
possible, we should detect these cases and do the map operation instead
of uploading.
For now, the relevant cases are moving between VAAPI and Vulkan, and
VAAPI and DRM Prime, in both directions. I have introduced the IMGFMT
entry for Vulkan here so that I can put in the complete mapping table.
It's actually not useless, as you can map to Vulkan, use a Vulkan
filter and then map back to VAAPI for display output.
Historically, HW -> HW uploads did not exist, so the current code
assumes they will never happen. But as part of introducing Vulkan
support into ffmpeg, we added HW -> HW support to enable transfers
between Vulkan and CUDA.
Today, that means you can use the lavfi hwupload filter with the
correct configuration (and previous changes in this series) but it
would be more convenient to enable HW -> HW in the format filter so
that the transfers can be done more intuitively:
```
--vf=format=fmt=cuda
```
and
```
--vf=format=fmt=vulkan
```
Most of the work here is skipping logic that is specific to SW -> HW
uploads doing format conversion. There is no ability to do inline
conversion when moving between HW formats, so the format must be
mutually understood to begin with.
Additional work needs to be done to enable transfers between VAAPI
and Vulkan which uses mapping, rather than uploads. I'll tackle that
in the next change.
It turns out that it's generally more useful to look up hwdecs by image
format, rather than device type. In the situations where we need to
find one, we generally know the image format we're dealing with. Doing
this avoids us having to create mappings from image format to device
type.
The most significant part of this change is filling in the image format
for the various hw interops. There is a hw_imgfmt field today today, but
only a couple of the interops fill it in, and that seems to be because
we've never actually used this piece of metadata before. Well, now we
have a good use for it.
Notably, BGR0, which is the only additional format listed as supported
by the texture mapper, results in broken colors. This is most likely not
a mpv issue, so the whitelist fulfils its purpose.
I think the previous code didn't consider the situation if the input
format was not any of the upload formats. It then could have possibly
tried to upload the wrong format (and not sure what the underlying APIs
do with it).
Take care of this, also improve logging, and change it such that
mp_hwupload_find_upload_format() does not unnecessarily change the state
(although it doesn't really matter).
(Oh yes, we could have skipped all the complexity, and hardcoded the
cases that work in the first place. This wouldn't be an issue if FFmpeg
or libva exported correct information. Also possible that FFmpeg's
filter chain does not allow to do this correctly in the first place,
since filters expose next to no meta information about what hw formats
etc. they need.)
Note that uploading yuv420p to a nv12 vaapi surface actually works, but
the blacklist excludes it. So this might get a bit slower. I'm not
bothering with this case because it's rarely needed, and the blacklist
specification would become a bit more complex if you had to specify
sw/upload format pairs.
Fixes: #7350
Basically, instead of trusting the upload format, and picking the first
sw format that has a desired upload format, trust the sw format. So now
we pick the sw format first, and then select from the set of upload
formats supported by it.
This is probably more straightforward, and works around a crash with
vaapi:
mpv 8bit.mkv --vf=format=vaapi --gpu-hwdec-interop=all
(Forces vaapi upload if hw decoding is not enabled.)
Unfortunately, this still does not work, because vaapi, FFmpeg, the VO
interop code, or all of them are doing something stupid. In particular,
this picks the yuv420p sw format, which doesn't really exist despiter
advertised (???????????????????????????????????????), and simply breaks.
See: #7350
This just wraps the mp_image_hw_download() function as a filter and adds
some minor caching/error logging. (Shame that it needs to much
boilerplate, I guess.)
Will be used by the following commit. Wrapping it as filter seemed more
convenient than other choices.
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
sfan5
Philip Langdale
llyyr
Harri Nieminen
Thomas Weißschuh
Aaron Boxer
wm4