Refactor the image allocation code, and expose part of it as helper
code. This aims towards allowing callers to easily allocate mp_image
references from custom-allocated linear buffers. This is exposing only
as much as what should be actually required.
Slightly cleaner, possibly slightly more correct. (The last case should
be dead code now. In general, we can't know the implied colorspace from
a AV_PIX_FMT, at least not if FFmpeg adds a new one.)
Another legacy annoyance. The only place where packed YUV is still
important is slightly older Apple hardware or drivers, which require
it for efficient hardware decoding.
For HLG, due to the usage of a reference OOTF configured for 1000 cd/m²,
the default sig_peak of =nom_peak was suboptimal. We can go down to
1000/100 (=10.0), since that's the true dynamic range of the output
signal after it passes through the OOTF.
This introduces (yet another..) mp_colorspace members, an enum `light`
(for lack of a better name) which basically tells us whether we're
dealing with scene-referred or display-referred light, but also a bit
more metadata (in which way is the scene-referred light expected to be
mapped to the display?).
The addition of this parameter accomplishes two goals:
1. Allows us to actually support HLG more-or-less correctly[1]
2. Allows people playing back direct “camera” content (e.g. v-log or
s-log2) to treat it as scene-referred instead of display-referred
[1] Even better would be to use the display-referred OOTF instead of the
idealized OOTF, but this would require either native HLG support in
LittleCMS (unlikely) or more communication between lcms.c and
video_shaders.c than I'm remotely comfortable with
That being said, in principle we could switch our usage of the BT.1886
EOTF to the BT.709 OETF instead and treat BT.709 content as being
scene-referred under application of the 709+1886 OOTF; which moves that
particular conversion from the 3dlut to the shader code; but also allows
a) users like UliZappe to turn it off and b) supporting the full HLG
OOTF in the same framework. But I think I prefer things as they are
right now.
List of changes:
1. Kill nom_peak, since it's a pointless non-field that stores nothing
of value and is _always_ derived from ref_white anyway.
2. Kill ref_white/--target-brightness, because the only case it really
existed for (PQ) actually doesn't need to be this general: According
to ITU-R BT.2100, PQ *always* assumes a reference monitor with a
white point of 100 cd/m².
3. Improve documentation and comments surrounding this stuff.
4. Clean up some of the code in general. Move stuff where it belongs.
Since michael was somewhat involved in it, wait with the actual license
change until the core is relicensed. Thus mark it as "Almost LGPL.".
The worrisome part about mp_image.c is that it was created by cehoyos
(which disagreed with LGPL) in commit f2dee327b2. But it turns out it
was a patch by someone else (who agreed with LGPL).
For some reason, the patch was actually slightly modified by cehoyos for
no reason (messed with the include statements), so we mess them back,
just to be sure.
Other than this, there were some commits that added support for new
IMGFMTs over the years. Some of these were by people we didn't ask or we
didn't get permission from. But since the original mp_image code was
replaced by more generic code using FFmpeg pixdesc, none of these
changes are left anyway.
One additional change by cehoyos (115bfb9762) has been removed as well
(when "direct rendering" was dropped from the filter chain).
If imgfmt is a hwaccel format, hw_subfmt will contain the CPU equivalent
of the data stored in the hw frames.
Strictly speaking, not doing this was a bug, but since hwaccel formats
were tagged with MP_IMGFLAG_YUV, it didn't have much of an impact.
Preparation for enabling hw filters. mp_image_params can't have an
AVHWFramesContext reference (because it can't hold any allocations, and
isn't meant to hold "active" data in the first place.
So just use a mp_image. It has all real data removed, because that would
essentially leak 1 frame once the decoder or renderer don't need it
anymore.
Mostly affects conversion of the colorimetric parameters.
Not changing AV_FRAME_DATA_MASTERING_DISPLAY_METADATA handling - that's
too messy, as decoders typically output it for keyframes only, and would
require weird caching that can't even be done on the level of the frame
rewrapping functions.
This fixes direct rendering with hwdec_vaegl.c.
The code duplication between update_image_params() and
mp_image_copy_fields_from_av_frame() is quite annoying,
bit will have to be resolved in another commit.
Helps with gif, probably does unwanted things with other formats.
This doesn't handle --end quite correctly, but this could be added
later.
Fixes#3924.
The hw_subfmt field roughly corresponds to the field
AVHWFramesContext.sw_format in ffmpeg. The ffmpeg one is of the type
AVPixelFormat (instead of the underlying hardware format), so it's a
good idea to switch to this too for preparation.
Now the hw_subfmt field is an mp_imgfmt instead of an opaque/API-
specific number. VDPAU and Direct3D11 already used mp_imgfmt, but
Videotoolbox and VAAPI had to be switched.
One somewhat user-visible change is that the verbose log will now always
show the hw_subfmt as image format, instead of as nonsensical number.
(In the end it would be good if we could switch to AVHWFramesContext
completely, but the upstream API is incomplete and doesn't cover
Direct3D11 and Videotoolbox.)
This involves multiple changes:
1. Brightness metadata is split into nominal peak and signal peak.
For a quick and dirty explanation: nominal peak is the brightest value
that your color space can represent (i.e. the brightness of an encoded
1.0), and signal peak is the brightest value that actually occurs in
the video (i.e. the brightest thing that's displayed).
2. vo_opengl uses a new decision logic to figure out the right nom_peak
and sig_peak for all situations. It also does a better job of picking
the right target gamut/colorspace to use for the OSD. (Which still is
and still should be treated as sRGB). This change in logic also
fixes#3293 en passant.
3. Since it was growing rapidly, the logic for auto-guessing / inferring
the right colorimetry configuration (in pass_colormanage) was split from
the logic for actually performing the adaptation (now pass_color_map).
Right now, the new logic doesn't do a whole lot since HDR metadata is
still ignored (but not for long).
This has two reasons:
1. I tend to add new fields to this metadata, and every time I've done
so I've consistently forgotten to update all of the dozens of places in
which this colorimetry metadata might end up getting used. While most
usages don't really care about most of the metadata, sometimes the
intend was simply to “copy” the colorimetry metadata from one struct to
another. With this being inside a substruct, those lines of code can now
simply read a.color = b.color without having to care about added or
removed fields.
2. It makes the type definitions nicer for upcoming refactors.
In going through all of the usages, I also expanded a few where I felt
that omitting the “young” fields was a bug.
User request and not that hard. Closes#3157.
Note that FFmpeg doesn't support this and there's no signalling in HEVC
etc., so the only way users can access it is by using vf_format
manually.
Mind: This encoding uses full range values, not TV range.
Instead of doing HDR tone mapping on an ad-hoc basis inside
pass_colormanage, the reference peak of an image is now part of the
image params (alongside colorspace, gamma, etc.) and tone mapping is
done whenever peak_src != peak_dst.
To get sensible behavior when mixing HDR and SDR content and displays,
target-brightness is a generic filler for "the assumed brightness of SDR
content".
This gets rid of the weird display_scaled hack, sets the framework
for multiple HDR functions with difference reference peaks, and allows
us to (in a future commit) autodetect the right source peak from
the HDR metadata.
(Apart from metadata, the source peak can also be controlled via
vf_format. For HDR content this adjusts the overall image brightness,
for SDR content it's like simulating a different exposure)
No reason to do so. See also commit 240ba92b.
Since now many mp_images will never have a pixel aspect ratio set,
redefine a 0/0 aspect ratio to "undefined" instead invalid. This also
brings it more in line with how decoder vs. container aspect ratios are
handled.
Most callers seem to be fine with the new behavior.
mp_image_params_valid() in particular has to be adjusted, or some things
stop working due to mp_images not becoming valid after setting size and
format.
This is quite unexpected. It's caused by mp_image_set_size(), which is
used to update certain fields which can be format-dependent, but which
is actually also supposed to reset the pixel aspect ratio.
A minor simplification. Most callers don't need this, and there's no
good reason why the caller should provide an "initializer" like this.
(This function calls mp_image_new_dummy_ref(), which has no reason
for an initializer either.)
For hwaccel formats, mp_image will merely point to a hardware surface
handle. In these cases, the mp_image_params.imgfmt field describes the
format insufficiently, because it mostly only describes the type of the
hardware format, not its underlying format.
Introduce hw_subfmt to describe the underlying format. It makes sense to
use it with most hwaccels, though for now it will be used with the
following commit only.
Don't allow rounding to let it underflow to 0. 0 width or height is
simply not allowed and could cause problems otherwhere.
Indirectly fixes CID 1350057, which complains about not checking the
resulting output size values before using it in divisions.
MPlayer traditionally always used the display aspect ratio, e.g. 16:9,
while FFmpeg uses the sample (aka pixel) aspect ratio.
Both have a bunch of advantages and disadvantages. Actually, it seems
using sample aspect ratio is generally nicer. The main reason for the
change is making mpv closer to how FFmpeg works in order to make life
easier. It's also nice that everything uses integer fractions instead
of floats now (except --video-aspect option/property).
Note that there is at least 1 user-visible change: vf_dsize now does
not set the display size, only the display aspect ratio. This is
because the image_params d_w/d_h fields did not just set the display
aspect, but also the size (except in encoding mode).
The vf_format suboption is replaced with --video-output-levels (a global
option and property). In particular, the parameter is removed from
mp_image_params. The mechanism is moved to the "video equalizer", which
also handles common video output customization like brightness and
contrast controls.
The new code is slightly cleaner, and the top-level option is slightly
more user-friendly than as vf_format sub-option.
Make the GPU memcpy from the dxva2 code generally useful to other parts
of the player.
We need to check at configure time whether SSE intrinsics work at all.
(At least in this form, they won't work on clang, for example. It also
won't work on non-x86.)
Introduce a mp_image_copy_gpu(), and make the dxva2 code use it. Do some
awkward stuff to share the existing code used by mp_image_copy(). I'm
hoping that FFmpeg will sooner or later provide a function like this, so
we can remove most of this again. (There is a patch, bit it's stuck in
limbo since forever.)
All this is used by the following commit.
Some code called by vf_vdpaupp.c calls mp_image_new_custom_ref(), but
out of convenience doesn't reset the buffers. Make this behavior ok.
(The assert() was there to catch usage errors, but the same error could
already happen before the refcount changes were made, so the check is
not overly helpful.)
Fixes#2115.
mpv had refcounted frames before libav*, so we were not using
libavutil's facilities. Change this and drop our own code.
Since AVFrames are not actually refcounted, and only the image data
they reference, the semantics change a bit. This affects mainly
mp_image_pool, which was operating on whole images instead of buffers.
While we could work on AVBufferRefs instead (and use AVBufferPool),
this doesn't work for use with hardware decoding, which doesn't
map cleanly to FFmpeg's reference counting. But it worked out. One
weird consequence is that we still need our custom image data
allocation function (for normal image data), because AVFrame's uses
multiple buffers.
There also seems to be a timing-dependent problem with vaapi (the
pool appears to be "leaking" surfaces). I don't know if this is a new
problem, or whether the code changes just happened to cause it more
often. Raising the number of reserved surfaces seemed to fix it, but
since it appears to be timing dependent, and I couldn't find anything
wrong with the code, I'm just going to assume it's not a new bug.
MP_IMGFIELD_TOP/MP_IMGFIELD_BOTTOM were completely unused, and
MP_IMGFIELD_ORDERED was always set (even though vf_vdpaupp.c strangely
checked for the latter).
Because gcc (and clang) is a goddamn PITA and unnecessarily warns if
the universal initializer for structs is used (like mp_image x = {})
and the first member of the struct is also a struct, move the w/h
fields to the top.
Remove the colorspace-related top-level options, add them to vf_format.
They are rather obscure and not needed often, so it's better to get them
out of the way. In particular, this gets rid of the semi-complicated
logic in command.c (most of which was needed for OSD display and the
direct feedback from the VO). It removes the duplicated color-related
name mappings.
This removes the ability to write the colormatrix and related
properties. Since filters can be changed at runtime, there's no loss of
functionality, except that you can't cycle automatically through the
color constants anymore (but who needs to do this).
This also changes the type of the mp_csp_names and related variables, so
they can directly be used with OPT_CHOICE. This probably ended up a bit
awkward, for the sake of not adding a new option type which would have
used the previous format.
Like FFmpeg/Libav do. It seems not all code can actually deal with this
situation, so it's better to shift the special-cases to code which needs
it (possibly OSD code; screenshots of 0x0 windows would just fail).
Normally, the size of an mage plane is assumed to be stride*height. But
in theory, if stride is larger than width*bpp, the last line might not
be padded, simply because it's not necessary. FFmpeg's or mpv's image
allocators always guarantee that this padding exists (it wastes some
insignificant memory for avoiding such subtle issues), but some other
libraries might not.
I suspect one such case might be Xv via vo_xv (see #1698), although my X
server appears to provide full padding. In any case, it can't harm.
There's literally no reason why these functions have to be inline (they
might be performance critical, but then the function call overhead isn't
going to matter at all).
Uninline them and move them to mp_image.c. Drop the header file and fix
all uses of it.
wm4
Niklas Haas
Niklas Haas
Dmitrij D. Czarkoff
Marcin Kurczewski