2018-01-18 13:44:20 +00:00
|
|
|
#include "audio/aframe.h"
|
|
|
|
#include "audio/chmap_sel.h"
|
|
|
|
#include "audio/format.h"
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
#include "common/common.h"
|
|
|
|
#include "common/msg.h"
|
|
|
|
#include "video/hwdec.h"
|
|
|
|
#include "video/mp_image.h"
|
2019-10-02 20:30:25 +00:00
|
|
|
#include "video/mp_image_pool.h"
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
|
|
|
#include "f_autoconvert.h"
|
|
|
|
#include "f_hwtransfer.h"
|
2018-01-18 13:44:20 +00:00
|
|
|
#include "f_swresample.h"
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
#include "f_swscale.h"
|
|
|
|
#include "f_utils.h"
|
|
|
|
#include "filter.h"
|
|
|
|
#include "filter_internal.h"
|
|
|
|
|
|
|
|
struct priv {
|
|
|
|
struct mp_log *log;
|
|
|
|
|
|
|
|
struct mp_subfilter sub;
|
|
|
|
|
|
|
|
bool force_update;
|
|
|
|
|
|
|
|
int *imgfmts;
|
|
|
|
int *subfmts;
|
|
|
|
int num_imgfmts;
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
struct mp_image_params imgparams;
|
|
|
|
bool imgparams_set;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
|
|
|
// Enable special conversion for the final stage before the VO.
|
|
|
|
bool vo_convert;
|
|
|
|
|
|
|
|
// sws state
|
|
|
|
int in_imgfmt, in_subfmt;
|
|
|
|
|
2018-01-18 13:44:20 +00:00
|
|
|
int *afmts;
|
|
|
|
int num_afmts;
|
|
|
|
int *srates;
|
|
|
|
int num_srates;
|
|
|
|
struct mp_chmap_sel chmaps;
|
|
|
|
|
|
|
|
int in_afmt, in_srate;
|
|
|
|
struct mp_chmap in_chmap;
|
|
|
|
|
|
|
|
double audio_speed;
|
|
|
|
bool resampling_forced;
|
|
|
|
|
audio: change format negotiation, remove channel remix fudging
The audio format neogitation code was pretty complicated, although the
idea was simple: when the format changes (or on the first audio frame),
filter only the new frame through the entire filter chain, discard the
resulting frame, but use the format to initialize the AO.
This was useful for "fudging" the channel remix behavior (upmix or
downmix), and moving it before other filters. Apparently this was useful
for things like DRC filters, which might work better in stereo, and
which also can only achieve the desired volume levels by doing it before
a downmix, which would modify the volume. This mechanism was introduced
in commit 60048b7eb957b (which the commit message also describes as
"idiotic heuristic"). Knowing the output format is inherently necessary
for this, because otherwise we can't know what the hell the user defined
filters will do.
There were problems with robustness. Some filters needed more than one
frame. Resampling in particular would discard initial audio at high
resampling ratios. Some filters might drop audio intentionally (like
clipping data on timestamp ranges). There were also allegations that
some decoders output 0 length frames (although that is invalid in
libavcodec). The state machine was excessively complex and hard to
understand too.
There are 3 things that could have been done:
1. Fix robustness problems by doing more heuristics, like repeating
audio frames or simply decoding several frames. Since filters can
behave differently, this would have added lots of complexity.
2. Make use of libavfilter's format negotiation, and add the same to
mpv builtin filters. This is sort of annoying, because the format
negotiation in libavfilter changes the state of the filters. It also
reports only some parameters (mostly all for audio, but a lot of
holes for video). It would remove some of the state machine, but not
all.
3. Drop the channel remix fudging, and do the same as the video chain.
This would not require format negotiation, but instead you can just
filter the audio frames, and look what comes out of it. If nothing
comes out, simply never create an AO.
This commit selects option 3. It removes the remix fudging, which means
the loss of a feature. Users can instead add "--af=format=channels=2"
before their DRC filter, or something. I'm also considering changing the
default for --audio-channels back to stereo, and downmix in the decoder
or at the start of the filter chain, which would give the same results,
except requiring more configuration.
Implementation-wise, this is still a bit different from the video path.
The VO always remains the same instance, while the AO might have to be
recreated on configuration changes. This still requires explicit format
change handling + draining old data, but by putting it into
f_autoconvert, not much new code is needed.
2018-04-07 12:38:40 +00:00
|
|
|
bool format_change_blocked;
|
|
|
|
bool format_change_cont;
|
|
|
|
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
struct mp_autoconvert public;
|
|
|
|
};
|
|
|
|
|
|
|
|
// Dummy filter for bundling sub-conversion filters.
|
|
|
|
static const struct mp_filter_info convert_filter = {
|
|
|
|
.name = "convert",
|
|
|
|
};
|
|
|
|
|
|
|
|
void mp_autoconvert_clear(struct mp_autoconvert *c)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
p->num_imgfmts = 0;
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
p->imgparams_set = false;
|
2018-01-18 13:44:20 +00:00
|
|
|
p->num_afmts = 0;
|
|
|
|
p->num_srates = 0;
|
|
|
|
p->chmaps = (struct mp_chmap_sel){0};
|
|
|
|
p->force_update = true;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void mp_autoconvert_add_imgfmt(struct mp_autoconvert *c, int imgfmt, int subfmt)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
MP_TARRAY_GROW(p, p->imgfmts, p->num_imgfmts);
|
|
|
|
MP_TARRAY_GROW(p, p->subfmts, p->num_imgfmts);
|
|
|
|
|
|
|
|
p->imgfmts[p->num_imgfmts] = imgfmt;
|
|
|
|
p->subfmts[p->num_imgfmts] = subfmt;
|
|
|
|
|
|
|
|
p->num_imgfmts += 1;
|
|
|
|
p->force_update = true;
|
|
|
|
}
|
|
|
|
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
void mp_autoconvert_set_target_image_params(struct mp_autoconvert *c,
|
|
|
|
struct mp_image_params *par)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
if (p->imgparams_set && mp_image_params_equal(&p->imgparams, par) &&
|
|
|
|
p->num_imgfmts == 1 && p->imgfmts[0] == par->imgfmt &&
|
|
|
|
p->subfmts[0] == par->hw_subfmt)
|
|
|
|
return;
|
|
|
|
|
|
|
|
p->imgparams = *par;
|
|
|
|
p->imgparams_set = true;
|
|
|
|
|
|
|
|
p->num_imgfmts = 0;
|
|
|
|
mp_autoconvert_add_imgfmt(c, par->imgfmt, par->hw_subfmt);
|
|
|
|
}
|
|
|
|
|
2019-10-02 20:30:25 +00:00
|
|
|
void mp_autoconvert_add_all_sw_imgfmts(struct mp_autoconvert *c)
|
|
|
|
{
|
|
|
|
for (int n = IMGFMT_START; n < IMGFMT_END; n++) {
|
|
|
|
if (!IMGFMT_IS_HWACCEL(n))
|
|
|
|
mp_autoconvert_add_imgfmt(c, n, 0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-01-18 13:44:20 +00:00
|
|
|
void mp_autoconvert_add_afmt(struct mp_autoconvert *c, int afmt)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
MP_TARRAY_APPEND(p, p->afmts, p->num_afmts, afmt);
|
|
|
|
p->force_update = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
void mp_autoconvert_add_chmap(struct mp_autoconvert *c, struct mp_chmap *chmap)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
mp_chmap_sel_add_map(&p->chmaps, chmap);
|
|
|
|
p->force_update = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
void mp_autoconvert_add_srate(struct mp_autoconvert *c, int rate)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
MP_TARRAY_APPEND(p, p->srates, p->num_srates, rate);
|
|
|
|
// Some other API we call expects a 0-terminated sample rates array.
|
|
|
|
MP_TARRAY_GROW(p, p->srates, p->num_srates);
|
|
|
|
p->srates[p->num_srates] = 0;
|
|
|
|
p->force_update = true;
|
|
|
|
}
|
|
|
|
|
2019-10-02 21:00:28 +00:00
|
|
|
// If this returns true, and *out==NULL, no conversion is necessary.
|
|
|
|
static bool build_image_converter(struct mp_autoconvert *c, struct mp_log *log,
|
|
|
|
struct mp_image *img, struct mp_filter **f_out)
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
{
|
2019-10-02 21:00:28 +00:00
|
|
|
struct mp_filter *f = c->f;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
2019-10-02 21:00:28 +00:00
|
|
|
*f_out = NULL;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
2019-10-02 21:00:28 +00:00
|
|
|
if (!p->num_imgfmts)
|
|
|
|
return true;
|
2018-04-07 11:05:25 +00:00
|
|
|
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
for (int n = 0; n < p->num_imgfmts; n++) {
|
|
|
|
bool samefmt = img->params.imgfmt == p->imgfmts[n];
|
|
|
|
bool samesubffmt = img->params.hw_subfmt == p->subfmts[n];
|
hwtransfer: implement support for hw->hw format conversion
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
2023-07-29 14:43:58 +00:00
|
|
|
/*
|
|
|
|
* In practice, `p->subfmts` is not usually populated today, in which
|
|
|
|
* case we must actively probe formats below to establish if the VO can
|
|
|
|
* accept the subfmt being used by the hwdec.
|
|
|
|
*/
|
|
|
|
if (samefmt && samesubffmt) {
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
if (p->imgparams_set) {
|
2024-02-10 22:46:04 +00:00
|
|
|
if (!mp_image_params_static_equal(&p->imgparams, &img->params))
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
break;
|
|
|
|
}
|
2019-10-02 21:00:28 +00:00
|
|
|
return true;
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
}
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
struct mp_filter *conv = mp_filter_create(f, &convert_filter);
|
2023-11-18 22:30:42 +00:00
|
|
|
if (!conv)
|
|
|
|
return false;
|
|
|
|
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
mp_filter_add_pin(conv, MP_PIN_IN, "in");
|
|
|
|
mp_filter_add_pin(conv, MP_PIN_OUT, "out");
|
|
|
|
|
2019-10-02 20:30:25 +00:00
|
|
|
// 0: hw->sw download
|
|
|
|
// 1: swscale
|
|
|
|
// 2: sw->hw upload
|
|
|
|
struct mp_filter *filters[3] = {0};
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
bool need_sws = true;
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
bool force_sws_params = false;
|
|
|
|
struct mp_image_params imgpar = img->params;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
|
|
|
int *fmts = p->imgfmts;
|
|
|
|
int num_fmts = p->num_imgfmts;
|
2020-01-12 00:33:02 +00:00
|
|
|
int hwupload_fmt = 0;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
2019-10-02 20:30:25 +00:00
|
|
|
bool imgfmt_is_sw = !IMGFMT_IS_HWACCEL(img->imgfmt);
|
|
|
|
|
|
|
|
// This should not happen. But not enough guarantee to make it an assert().
|
|
|
|
if (imgfmt_is_sw != !img->hwctx)
|
2019-10-02 21:00:28 +00:00
|
|
|
mp_warn(log, "Unexpected AVFrame/imgfmt hardware context mismatch.\n");
|
2019-10-02 20:30:25 +00:00
|
|
|
|
|
|
|
bool dst_all_hw = true;
|
|
|
|
bool dst_have_sw = false;
|
hwtransfer: implement support for hw->hw format conversion
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
2023-07-29 14:43:58 +00:00
|
|
|
bool has_src_hw_fmt = false;
|
2019-10-02 20:30:25 +00:00
|
|
|
for (int n = 0; n < num_fmts; n++) {
|
|
|
|
bool is_hw = IMGFMT_IS_HWACCEL(fmts[n]);
|
|
|
|
dst_all_hw &= is_hw;
|
|
|
|
dst_have_sw |= !is_hw;
|
hwtransfer: implement support for hw->hw format conversion
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
2023-07-29 14:43:58 +00:00
|
|
|
has_src_hw_fmt |= is_hw && fmts[n] == imgpar.imgfmt;
|
2019-10-02 20:30:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Source is hw, some targets are sw -> try to download.
|
|
|
|
bool hw_to_sw = !imgfmt_is_sw && dst_have_sw;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
hwtransfer: implement support for hw->hw format conversion
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
2023-07-29 14:43:58 +00:00
|
|
|
if (has_src_hw_fmt) {
|
|
|
|
int src_fmt = img->params.hw_subfmt;
|
|
|
|
/*
|
|
|
|
* If the source format is a hardware format, and our output supports
|
|
|
|
* that hardware format, we prioritize preserving the use of that
|
|
|
|
* hardware format. In most cases, the sub format will also be supported
|
|
|
|
* and no conversion will be required, but in some cases, the hwdec
|
|
|
|
* may be able to output formats that the VO cannot display, and
|
|
|
|
* hardware format conversion becomes necessary.
|
|
|
|
*/
|
|
|
|
struct mp_hwupload upload = mp_hwupload_create(conv, imgpar.imgfmt,
|
|
|
|
src_fmt,
|
|
|
|
true);
|
|
|
|
if (upload.successful_init) {
|
|
|
|
if (upload.f) {
|
|
|
|
mp_info(log, "Converting %s[%s] -> %s[%s]\n",
|
|
|
|
mp_imgfmt_to_name(imgpar.imgfmt),
|
|
|
|
mp_imgfmt_to_name(src_fmt),
|
|
|
|
mp_imgfmt_to_name(imgpar.imgfmt),
|
|
|
|
mp_imgfmt_to_name(upload.selected_sw_imgfmt));
|
|
|
|
filters[2] = upload.f;
|
|
|
|
}
|
|
|
|
hw_to_sw = false;
|
|
|
|
need_sws = false;
|
|
|
|
} else {
|
|
|
|
mp_err(log, "Failed to create HW uploader for format %s\n",
|
|
|
|
mp_imgfmt_to_name(src_fmt));
|
|
|
|
}
|
|
|
|
} else if (dst_all_hw && num_fmts > 0) {
|
2022-09-27 16:12:54 +00:00
|
|
|
bool upload_created = false;
|
|
|
|
int sw_fmt = imgfmt_is_sw ? img->imgfmt : img->params.hw_subfmt;
|
|
|
|
|
2022-09-27 15:35:14 +00:00
|
|
|
for (int i = 0; i < num_fmts; i++) {
|
|
|
|
// We can probably use this! Very lazy and very approximate.
|
hwtransfer: implement support for hw->hw format conversion
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
2023-07-29 14:43:58 +00:00
|
|
|
struct mp_hwupload upload = mp_hwupload_create(conv, fmts[i],
|
|
|
|
sw_fmt, false);
|
|
|
|
if (upload.successful_init) {
|
2022-09-27 15:35:14 +00:00
|
|
|
mp_info(log, "HW-uploading to %s\n", mp_imgfmt_to_name(fmts[i]));
|
hwtransfer: implement support for hw->hw format conversion
Historically, we have not attempted to support hw->hw format conversion
in the autoconvert logic. If a user needed to do these kinds of
conversions, they needed to manually insert the hw format's conversion
filter manually (eg: scale_vaapi).
This was usually fine because the general rule is that any format
supported by the hardware can be used as well as any other. ie: You
would only need to do conversion if you have a specific goal in mind.
However, we now have two situations where we can find ourselves with a
hardware format being produced by a decoder that cannot be accepted by
a VO via hwdec-interop:
* dmabuf-wayland can only accept formats that the Wayland compositor
accepts. In the case of GNOME, it can only accept a handful of RGB
formats.
* When decoding via VAAPI on Intel hardware, some of the more unusual
video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats
which gpu-next cannot handle, causing rendering to fail.
In both these cases (at least when using VAAPI with dmabuf-wayland), if
we could detect the failure case and insert a `scale_vaapi` filter, we
could get successful output automatically. For `hwdec=drm`, there is
currently no conversion filter, so dmabuf-wayland is still out of luck
there.
The basic approach to implementing this is to detect the case where we
are evaluating a hardware format where the VO can accept the hardware
format itself, but may not accept the underlying sw format. In the
current code, we bypass autoconvert as soon as we see the hardware
format is compatible.
My first observation was that we actually have logic in autoconvert to
detect when the input sw format is not in the list of allowed sw
formats passed into the autoconverter. Unfortunately, we never populate
this list, and the way you would expect to do that is vo-query-format
returning sw format information, which it does not. We could define an
extended vo-query-format-2, but we'd still need to implement the
probing logic to fill it in.
On the other hand, we already have the probing logic in the hwupload
filter - and most recently I used that logic to implement conversion
on upload. So if we could leverage that, we could both detect when
hw->hw conversion is required, and pick the best target format.
This exercise is then primarily one of detecting when we are in this
case and letting that code run in a useful way. The hwupload filter is
a bit awkward to work with today, and so I refactored a bunch of the
set up code to actually make it more encapsulated. Now, instead of the
caller instantiating it and then triggering the probe, we probe on
creation and instantiate the correct underlying filter (hwupload vs
conversion) automatically.
2023-07-29 14:43:58 +00:00
|
|
|
filters[2] = upload.f;
|
|
|
|
hwupload_fmt = upload.selected_sw_imgfmt;
|
2022-09-27 15:35:14 +00:00
|
|
|
fmts = &hwupload_fmt;
|
|
|
|
num_fmts = hwupload_fmt ? 1 : 0;
|
|
|
|
hw_to_sw = false;
|
|
|
|
|
|
|
|
// We cannot do format conversions when transferring between
|
|
|
|
// two hardware devices, so reject this format if that would be
|
|
|
|
// required.
|
|
|
|
if (!imgfmt_is_sw && hwupload_fmt != sw_fmt) {
|
|
|
|
mp_err(log, "Format %s is not supported by %s\n",
|
|
|
|
mp_imgfmt_to_name(sw_fmt),
|
|
|
|
mp_imgfmt_to_name(p->imgfmts[i]));
|
2022-09-27 16:12:54 +00:00
|
|
|
continue;
|
2022-09-27 15:35:14 +00:00
|
|
|
}
|
2022-09-27 16:12:54 +00:00
|
|
|
upload_created = true;
|
|
|
|
break;
|
2022-03-25 18:16:21 +00:00
|
|
|
}
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
2022-09-27 16:12:54 +00:00
|
|
|
if (!upload_created) {
|
|
|
|
mp_err(log, "Failed to create HW uploader for format %s\n",
|
|
|
|
mp_imgfmt_to_name(sw_fmt));
|
|
|
|
}
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
2019-10-02 20:30:25 +00:00
|
|
|
int src_fmt = img->imgfmt;
|
|
|
|
if (hw_to_sw) {
|
2019-10-02 21:00:28 +00:00
|
|
|
mp_info(log, "HW-downloading from %s\n", mp_imgfmt_to_name(img->imgfmt));
|
2019-10-02 20:30:25 +00:00
|
|
|
int res_fmt = mp_image_hw_download_get_sw_format(img);
|
|
|
|
if (!res_fmt) {
|
2019-10-02 21:00:28 +00:00
|
|
|
mp_err(log, "cannot copy surface of this format to CPU memory\n");
|
2019-10-02 20:30:25 +00:00
|
|
|
goto fail;
|
|
|
|
}
|
|
|
|
struct mp_hwdownload *hwd = mp_hwdownload_create(conv);
|
|
|
|
if (hwd) {
|
|
|
|
filters[0] = hwd->f;
|
|
|
|
src_fmt = res_fmt;
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
// Downloading from hw will obviously change the parameters. We
|
|
|
|
// stupidly don't know the result parameters, but if it's
|
|
|
|
// sufficiently sane, it will only do the following.
|
|
|
|
imgpar.imgfmt = src_fmt;
|
|
|
|
imgpar.hw_subfmt = 0;
|
|
|
|
// Try to compensate for in-sane cases.
|
|
|
|
mp_image_params_guess_csp(&imgpar);
|
2019-10-02 20:30:25 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
if (p->imgparams_set) {
|
|
|
|
force_sws_params |= !mp_image_params_equal(&imgpar, &p->imgparams);
|
|
|
|
need_sws |= force_sws_params;
|
|
|
|
}
|
2022-03-25 18:16:21 +00:00
|
|
|
if (!imgfmt_is_sw && dst_all_hw) {
|
|
|
|
// This is a hw -> hw upload, so the sw format must already be
|
|
|
|
// mutually understood. No conversion can be done.
|
|
|
|
need_sws = false;
|
|
|
|
}
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
if (need_sws) {
|
|
|
|
// Create a new conversion filter.
|
|
|
|
struct mp_sws_filter *sws = mp_sws_filter_create(conv);
|
|
|
|
if (!sws) {
|
2019-10-02 21:00:28 +00:00
|
|
|
mp_err(log, "error creating conversion filter\n");
|
|
|
|
goto fail;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
2020-04-12 19:55:35 +00:00
|
|
|
sws->force_scaler = c->force_scaler;
|
|
|
|
|
2019-11-02 16:50:24 +00:00
|
|
|
int out = mp_sws_find_best_out_format(sws, src_fmt, fmts, num_fmts);
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
if (!out) {
|
2019-10-02 21:00:28 +00:00
|
|
|
mp_err(log, "can't find video conversion for %s\n",
|
2019-10-02 20:30:25 +00:00
|
|
|
mp_imgfmt_to_name(src_fmt));
|
|
|
|
goto fail;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
if (out == src_fmt && !force_sws_params) {
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
// Can happen if hwupload goes to same format.
|
|
|
|
talloc_free(sws->f);
|
|
|
|
} else {
|
|
|
|
sws->out_format = out;
|
filters: extend vf_format so that it can convert color parameters
Form some reason (and because of my fault), vf_format converts image
formats, but nothing else. For example, setting the "colormatrix"
sub-parameter would not convert it to the new value, but instead
overwrite the metadata (basically "reinterpreting" the image data
without changing it).
Make the historical mistake worse, and go all the way and extend it such
that it can perform a conversion. For compatibility reasons, this needs
to be requested explicitly. (Maybe this would deserve a separate filter
to begin with, but things are messed up anyway. Feel free to suggest an
elegant and simple solution.)
This demonstrates how zimg can properly perform some conversions which
swscale cannot (see examples added to vf.rst).
Stupidly this requires 2 code paths, one for conversion, and one for
overriding the parameters.
Due to the filter bullshit (what was I thinking), this requires quite
some acrobatics that would not be necessary without these abstractions.
On the other hand, it'd definitely be more of a mess without it. Oh
whatever.
2019-10-20 23:29:48 +00:00
|
|
|
sws->out_params = p->imgparams;
|
|
|
|
sws->use_out_params = force_sws_params;
|
2019-10-02 21:00:28 +00:00
|
|
|
mp_info(log, "Converting %s -> %s\n", mp_imgfmt_to_name(src_fmt),
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
mp_imgfmt_to_name(sws->out_format));
|
2019-10-02 20:30:25 +00:00
|
|
|
filters[1] = sws->f;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-10-02 20:30:25 +00:00
|
|
|
mp_chain_filters(conv->ppins[0], conv->ppins[1], filters, 3);
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
2019-10-02 21:00:28 +00:00
|
|
|
*f_out = conv;
|
|
|
|
return true;
|
2019-10-02 20:30:25 +00:00
|
|
|
|
|
|
|
fail:
|
|
|
|
talloc_free(conv);
|
2019-10-02 21:00:28 +00:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
bool mp_autoconvert_probe_input_video(struct mp_autoconvert *c,
|
|
|
|
struct mp_image *img)
|
|
|
|
{
|
|
|
|
struct mp_filter *conv = NULL;
|
|
|
|
bool res = build_image_converter(c, mp_null_log, img, &conv);
|
|
|
|
talloc_free(conv);
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void handle_video_frame(struct mp_filter *f)
|
|
|
|
{
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
|
|
|
struct mp_image *img = p->sub.frame.data;
|
|
|
|
|
|
|
|
if (p->force_update)
|
|
|
|
p->in_imgfmt = p->in_subfmt = 0;
|
|
|
|
|
|
|
|
if (img->imgfmt == p->in_imgfmt && img->params.hw_subfmt == p->in_subfmt) {
|
|
|
|
mp_subfilter_continue(&p->sub);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!mp_subfilter_drain_destroy(&p->sub)) {
|
autoconvert: destroy sub filter immediately if reconfiguration is needed
I'm currently not convinced that the way the output_chain is handled as
part of reconfiguration is correct. If there is an event requiring
reconfiguration, such as toggling the use of hwdec, we currently do not
ensure that the filter chain is fully drained first. This creates a
situation where the filter chain is invalidated while the autoconvert's
sub filter (that does the real work) still has a frame to process and
pass on.
As the autoconvert code calls mp_subfilter_drain_destroy(), it returns
early to allow for draining before destroying the subfilter, but that
means the subfilter is still present in its original configuration, and
no actually draining is done before the existing filters reinitialise
themselves.
This leads to a situation where, if a hardware scaling filter is being
used by autoconvert, that filter is still present and responds when
told to reinitialise. But it cannot successfully reinitialise if the
triggering event is disabling hw decoding, as the input frame to the
filter will now be a software frame, so reinit fails, leading to total
failure of the filter chain, which is a fatal error, and we exit.
I think this was never noticed before, because I think it's not
possible for the hwtransfer filter to be active in a situation where
you can dynamically change the state such that the input or output
formats of the output chain are invalidated.
eg: If the autoconverter is activated because of `--vf=format=vaapi`,
it is actually not possible to toggle hwdec off, as the explicit user
filter ensure the hwdec is always present and active.
So, my solution here is to destroy the sub filter, regardless of
whether it needs draining or not. We simply have no opportunity to
drain and reconfigure in the correct order, and we must consider the
remaining frame in the filter as a casualty of the toggling process.
I'm sure there is a more substantial rework of the output_chain
reconfiguration process that could ensure draining before
reconfiguration begins, but my ambitions do not currently extend that
far.
2023-07-31 10:24:32 +00:00
|
|
|
MP_VERBOSE(f, "Sub-filter requires draining but we must destroy it now.\n");
|
|
|
|
mp_subfilter_destroy(&p->sub);
|
2019-10-02 21:00:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
p->in_imgfmt = img->params.imgfmt;
|
|
|
|
p->in_subfmt = img->params.hw_subfmt;
|
|
|
|
p->force_update = false;
|
|
|
|
|
|
|
|
struct mp_filter *conv = NULL;
|
|
|
|
if (build_image_converter(&p->public, p->log, img, &conv)) {
|
|
|
|
p->sub.filter = conv;
|
|
|
|
mp_subfilter_continue(&p->sub);
|
|
|
|
} else {
|
|
|
|
mp_filter_internal_mark_failed(f);
|
|
|
|
}
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
2018-01-18 13:44:20 +00:00
|
|
|
static void handle_audio_frame(struct mp_filter *f)
|
|
|
|
{
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
2018-04-07 11:05:25 +00:00
|
|
|
struct mp_aframe *aframe = p->sub.frame.data;
|
2018-01-18 13:44:20 +00:00
|
|
|
|
|
|
|
int afmt = mp_aframe_get_format(aframe);
|
|
|
|
int srate = mp_aframe_get_rate(aframe);
|
|
|
|
struct mp_chmap chmap = {0};
|
|
|
|
mp_aframe_get_chmap(aframe, &chmap);
|
|
|
|
|
2018-04-12 16:48:41 +00:00
|
|
|
if (p->resampling_forced && !af_fmt_is_pcm(afmt)) {
|
|
|
|
MP_WARN(p, "ignoring request to resample non-PCM audio for speed change\n");
|
|
|
|
p->resampling_forced = false;
|
|
|
|
}
|
|
|
|
|
audio: change format negotiation, remove channel remix fudging
The audio format neogitation code was pretty complicated, although the
idea was simple: when the format changes (or on the first audio frame),
filter only the new frame through the entire filter chain, discard the
resulting frame, but use the format to initialize the AO.
This was useful for "fudging" the channel remix behavior (upmix or
downmix), and moving it before other filters. Apparently this was useful
for things like DRC filters, which might work better in stereo, and
which also can only achieve the desired volume levels by doing it before
a downmix, which would modify the volume. This mechanism was introduced
in commit 60048b7eb957b (which the commit message also describes as
"idiotic heuristic"). Knowing the output format is inherently necessary
for this, because otherwise we can't know what the hell the user defined
filters will do.
There were problems with robustness. Some filters needed more than one
frame. Resampling in particular would discard initial audio at high
resampling ratios. Some filters might drop audio intentionally (like
clipping data on timestamp ranges). There were also allegations that
some decoders output 0 length frames (although that is invalid in
libavcodec). The state machine was excessively complex and hard to
understand too.
There are 3 things that could have been done:
1. Fix robustness problems by doing more heuristics, like repeating
audio frames or simply decoding several frames. Since filters can
behave differently, this would have added lots of complexity.
2. Make use of libavfilter's format negotiation, and add the same to
mpv builtin filters. This is sort of annoying, because the format
negotiation in libavfilter changes the state of the filters. It also
reports only some parameters (mostly all for audio, but a lot of
holes for video). It would remove some of the state machine, but not
all.
3. Drop the channel remix fudging, and do the same as the video chain.
This would not require format negotiation, but instead you can just
filter the audio frames, and look what comes out of it. If nothing
comes out, simply never create an AO.
This commit selects option 3. It removes the remix fudging, which means
the loss of a feature. Users can instead add "--af=format=channels=2"
before their DRC filter, or something. I'm also considering changing the
default for --audio-channels back to stereo, and downmix in the decoder
or at the start of the filter chain, which would give the same results,
except requiring more configuration.
Implementation-wise, this is still a bit different from the video path.
The VO always remains the same instance, while the AO might have to be
recreated on configuration changes. This still requires explicit format
change handling + draining old data, but by putting it into
f_autoconvert, not much new code is needed.
2018-04-07 12:38:40 +00:00
|
|
|
bool format_change = afmt != p->in_afmt ||
|
|
|
|
srate != p->in_srate ||
|
|
|
|
!mp_chmap_equals(&chmap, &p->in_chmap) ||
|
|
|
|
p->force_update;
|
|
|
|
|
|
|
|
if (!format_change && (!p->resampling_forced || p->sub.filter))
|
2018-01-18 13:44:20 +00:00
|
|
|
goto cont;
|
|
|
|
|
|
|
|
if (!mp_subfilter_drain_destroy(&p->sub))
|
|
|
|
return;
|
|
|
|
|
audio: change format negotiation, remove channel remix fudging
The audio format neogitation code was pretty complicated, although the
idea was simple: when the format changes (or on the first audio frame),
filter only the new frame through the entire filter chain, discard the
resulting frame, but use the format to initialize the AO.
This was useful for "fudging" the channel remix behavior (upmix or
downmix), and moving it before other filters. Apparently this was useful
for things like DRC filters, which might work better in stereo, and
which also can only achieve the desired volume levels by doing it before
a downmix, which would modify the volume. This mechanism was introduced
in commit 60048b7eb957b (which the commit message also describes as
"idiotic heuristic"). Knowing the output format is inherently necessary
for this, because otherwise we can't know what the hell the user defined
filters will do.
There were problems with robustness. Some filters needed more than one
frame. Resampling in particular would discard initial audio at high
resampling ratios. Some filters might drop audio intentionally (like
clipping data on timestamp ranges). There were also allegations that
some decoders output 0 length frames (although that is invalid in
libavcodec). The state machine was excessively complex and hard to
understand too.
There are 3 things that could have been done:
1. Fix robustness problems by doing more heuristics, like repeating
audio frames or simply decoding several frames. Since filters can
behave differently, this would have added lots of complexity.
2. Make use of libavfilter's format negotiation, and add the same to
mpv builtin filters. This is sort of annoying, because the format
negotiation in libavfilter changes the state of the filters. It also
reports only some parameters (mostly all for audio, but a lot of
holes for video). It would remove some of the state machine, but not
all.
3. Drop the channel remix fudging, and do the same as the video chain.
This would not require format negotiation, but instead you can just
filter the audio frames, and look what comes out of it. If nothing
comes out, simply never create an AO.
This commit selects option 3. It removes the remix fudging, which means
the loss of a feature. Users can instead add "--af=format=channels=2"
before their DRC filter, or something. I'm also considering changing the
default for --audio-channels back to stereo, and downmix in the decoder
or at the start of the filter chain, which would give the same results,
except requiring more configuration.
Implementation-wise, this is still a bit different from the video path.
The VO always remains the same instance, while the AO might have to be
recreated on configuration changes. This still requires explicit format
change handling + draining old data, but by putting it into
f_autoconvert, not much new code is needed.
2018-04-07 12:38:40 +00:00
|
|
|
if (format_change && p->public.on_audio_format_change) {
|
|
|
|
if (p->format_change_blocked)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (!p->format_change_cont) {
|
|
|
|
p->format_change_blocked = true;
|
|
|
|
p->public.
|
|
|
|
on_audio_format_change(p->public.on_audio_format_change_opaque);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
p->format_change_cont = false;
|
|
|
|
}
|
|
|
|
|
2018-01-18 13:44:20 +00:00
|
|
|
p->in_afmt = afmt;
|
|
|
|
p->in_srate = srate;
|
|
|
|
p->in_chmap = chmap;
|
|
|
|
p->force_update = false;
|
|
|
|
|
|
|
|
int out_afmt = 0;
|
|
|
|
int best_score = 0;
|
|
|
|
for (int n = 0; n < p->num_afmts; n++) {
|
|
|
|
int score = af_format_conversion_score(p->afmts[n], afmt);
|
|
|
|
if (!out_afmt || score > best_score) {
|
|
|
|
best_score = score;
|
|
|
|
out_afmt = p->afmts[n];
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!out_afmt)
|
|
|
|
out_afmt = afmt;
|
|
|
|
|
|
|
|
// (The p->srates array is 0-terminated already.)
|
|
|
|
int out_srate = af_select_best_samplerate(srate, p->srates);
|
|
|
|
if (out_srate <= 0)
|
|
|
|
out_srate = p->num_srates ? p->srates[0] : srate;
|
|
|
|
|
|
|
|
struct mp_chmap out_chmap = chmap;
|
|
|
|
if (p->chmaps.num_chmaps) {
|
|
|
|
if (!mp_chmap_sel_adjust(&p->chmaps, &out_chmap))
|
|
|
|
out_chmap = p->chmaps.chmaps[0]; // violently force fallback
|
|
|
|
}
|
|
|
|
|
|
|
|
if (out_afmt == p->in_afmt && out_srate == p->in_srate &&
|
|
|
|
mp_chmap_equals(&out_chmap, &p->in_chmap) && !p->resampling_forced)
|
|
|
|
{
|
|
|
|
goto cont;
|
|
|
|
}
|
|
|
|
|
|
|
|
MP_VERBOSE(p, "inserting resampler\n");
|
|
|
|
|
|
|
|
struct mp_swresample *s = mp_swresample_create(f, NULL);
|
|
|
|
if (!s)
|
|
|
|
abort();
|
|
|
|
|
|
|
|
s->out_format = out_afmt;
|
|
|
|
s->out_rate = out_srate;
|
|
|
|
s->out_channels = out_chmap;
|
|
|
|
|
|
|
|
p->sub.filter = s->f;
|
|
|
|
|
|
|
|
cont:
|
|
|
|
|
|
|
|
if (p->sub.filter) {
|
|
|
|
struct mp_filter_command cmd = {
|
|
|
|
.type = MP_FILTER_COMMAND_SET_SPEED_RESAMPLE,
|
|
|
|
.speed = p->audio_speed,
|
|
|
|
};
|
|
|
|
mp_filter_command(p->sub.filter, &cmd);
|
|
|
|
}
|
|
|
|
|
|
|
|
mp_subfilter_continue(&p->sub);
|
|
|
|
}
|
|
|
|
|
2024-04-10 05:47:05 +00:00
|
|
|
static void autoconvert_process(struct mp_filter *f)
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
{
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
|
|
|
if (!mp_subfilter_read(&p->sub))
|
|
|
|
return;
|
|
|
|
|
2018-04-07 11:05:25 +00:00
|
|
|
if (p->sub.frame.type == MP_FRAME_VIDEO) {
|
|
|
|
handle_video_frame(f);
|
|
|
|
return;
|
|
|
|
}
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
|
2018-04-07 11:05:25 +00:00
|
|
|
if (p->sub.frame.type == MP_FRAME_AUDIO) {
|
|
|
|
handle_audio_frame(f);
|
|
|
|
return;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
mp_subfilter_continue(&p->sub);
|
|
|
|
}
|
|
|
|
|
audio: change format negotiation, remove channel remix fudging
The audio format neogitation code was pretty complicated, although the
idea was simple: when the format changes (or on the first audio frame),
filter only the new frame through the entire filter chain, discard the
resulting frame, but use the format to initialize the AO.
This was useful for "fudging" the channel remix behavior (upmix or
downmix), and moving it before other filters. Apparently this was useful
for things like DRC filters, which might work better in stereo, and
which also can only achieve the desired volume levels by doing it before
a downmix, which would modify the volume. This mechanism was introduced
in commit 60048b7eb957b (which the commit message also describes as
"idiotic heuristic"). Knowing the output format is inherently necessary
for this, because otherwise we can't know what the hell the user defined
filters will do.
There were problems with robustness. Some filters needed more than one
frame. Resampling in particular would discard initial audio at high
resampling ratios. Some filters might drop audio intentionally (like
clipping data on timestamp ranges). There were also allegations that
some decoders output 0 length frames (although that is invalid in
libavcodec). The state machine was excessively complex and hard to
understand too.
There are 3 things that could have been done:
1. Fix robustness problems by doing more heuristics, like repeating
audio frames or simply decoding several frames. Since filters can
behave differently, this would have added lots of complexity.
2. Make use of libavfilter's format negotiation, and add the same to
mpv builtin filters. This is sort of annoying, because the format
negotiation in libavfilter changes the state of the filters. It also
reports only some parameters (mostly all for audio, but a lot of
holes for video). It would remove some of the state machine, but not
all.
3. Drop the channel remix fudging, and do the same as the video chain.
This would not require format negotiation, but instead you can just
filter the audio frames, and look what comes out of it. If nothing
comes out, simply never create an AO.
This commit selects option 3. It removes the remix fudging, which means
the loss of a feature. Users can instead add "--af=format=channels=2"
before their DRC filter, or something. I'm also considering changing the
default for --audio-channels back to stereo, and downmix in the decoder
or at the start of the filter chain, which would give the same results,
except requiring more configuration.
Implementation-wise, this is still a bit different from the video path.
The VO always remains the same instance, while the AO might have to be
recreated on configuration changes. This still requires explicit format
change handling + draining old data, but by putting it into
f_autoconvert, not much new code is needed.
2018-04-07 12:38:40 +00:00
|
|
|
void mp_autoconvert_format_change_continue(struct mp_autoconvert *c)
|
|
|
|
{
|
|
|
|
struct priv *p = c->f->priv;
|
|
|
|
|
|
|
|
if (p->format_change_blocked) {
|
|
|
|
p->format_change_cont = true;
|
|
|
|
p->format_change_blocked = false;
|
|
|
|
mp_filter_wakeup(c->f);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2024-04-10 05:47:05 +00:00
|
|
|
static bool autoconvert_command(struct mp_filter *f, struct mp_filter_command *cmd)
|
2018-01-18 13:44:20 +00:00
|
|
|
{
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
|
|
|
if (cmd->type == MP_FILTER_COMMAND_SET_SPEED_RESAMPLE) {
|
|
|
|
p->audio_speed = cmd->speed;
|
|
|
|
// If we needed resampling once, keep forcing resampling, as it might be
|
|
|
|
// quickly changing between 1.0 and other values for A/V compensation.
|
|
|
|
if (p->audio_speed != 1.0)
|
|
|
|
p->resampling_forced = true;
|
|
|
|
return true;
|
2018-04-21 11:18:03 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (cmd->type == MP_FILTER_COMMAND_IS_ACTIVE) {
|
|
|
|
cmd->is_active = !!p->sub.filter;
|
|
|
|
return true;
|
2018-01-18 13:44:20 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2024-04-10 05:47:05 +00:00
|
|
|
static void autoconvert_reset(struct mp_filter *f)
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
{
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
|
|
|
mp_subfilter_reset(&p->sub);
|
audio: change format negotiation, remove channel remix fudging
The audio format neogitation code was pretty complicated, although the
idea was simple: when the format changes (or on the first audio frame),
filter only the new frame through the entire filter chain, discard the
resulting frame, but use the format to initialize the AO.
This was useful for "fudging" the channel remix behavior (upmix or
downmix), and moving it before other filters. Apparently this was useful
for things like DRC filters, which might work better in stereo, and
which also can only achieve the desired volume levels by doing it before
a downmix, which would modify the volume. This mechanism was introduced
in commit 60048b7eb957b (which the commit message also describes as
"idiotic heuristic"). Knowing the output format is inherently necessary
for this, because otherwise we can't know what the hell the user defined
filters will do.
There were problems with robustness. Some filters needed more than one
frame. Resampling in particular would discard initial audio at high
resampling ratios. Some filters might drop audio intentionally (like
clipping data on timestamp ranges). There were also allegations that
some decoders output 0 length frames (although that is invalid in
libavcodec). The state machine was excessively complex and hard to
understand too.
There are 3 things that could have been done:
1. Fix robustness problems by doing more heuristics, like repeating
audio frames or simply decoding several frames. Since filters can
behave differently, this would have added lots of complexity.
2. Make use of libavfilter's format negotiation, and add the same to
mpv builtin filters. This is sort of annoying, because the format
negotiation in libavfilter changes the state of the filters. It also
reports only some parameters (mostly all for audio, but a lot of
holes for video). It would remove some of the state machine, but not
all.
3. Drop the channel remix fudging, and do the same as the video chain.
This would not require format negotiation, but instead you can just
filter the audio frames, and look what comes out of it. If nothing
comes out, simply never create an AO.
This commit selects option 3. It removes the remix fudging, which means
the loss of a feature. Users can instead add "--af=format=channels=2"
before their DRC filter, or something. I'm also considering changing the
default for --audio-channels back to stereo, and downmix in the decoder
or at the start of the filter chain, which would give the same results,
except requiring more configuration.
Implementation-wise, this is still a bit different from the video path.
The VO always remains the same instance, while the AO might have to be
recreated on configuration changes. This still requires explicit format
change handling + draining old data, but by putting it into
f_autoconvert, not much new code is needed.
2018-04-07 12:38:40 +00:00
|
|
|
|
|
|
|
p->format_change_cont = false;
|
|
|
|
p->format_change_blocked = false;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
}
|
|
|
|
|
2024-04-10 05:47:05 +00:00
|
|
|
static void autoconvert_destroy(struct mp_filter *f)
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
{
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
|
|
|
|
mp_subfilter_reset(&p->sub);
|
|
|
|
TA_FREEP(&p->sub.filter);
|
|
|
|
}
|
|
|
|
|
|
|
|
static const struct mp_filter_info autoconvert_filter = {
|
|
|
|
.name = "autoconvert",
|
|
|
|
.priv_size = sizeof(struct priv),
|
2024-04-10 05:47:05 +00:00
|
|
|
.process = autoconvert_process,
|
|
|
|
.command = autoconvert_command,
|
|
|
|
.reset = autoconvert_reset,
|
|
|
|
.destroy = autoconvert_destroy,
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
struct mp_autoconvert *mp_autoconvert_create(struct mp_filter *parent)
|
|
|
|
{
|
|
|
|
struct mp_filter *f = mp_filter_create(parent, &autoconvert_filter);
|
|
|
|
if (!f)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
mp_filter_add_pin(f, MP_PIN_IN, "in");
|
|
|
|
mp_filter_add_pin(f, MP_PIN_OUT, "out");
|
|
|
|
|
|
|
|
struct priv *p = f->priv;
|
|
|
|
p->public.f = f;
|
|
|
|
p->log = f->log;
|
2018-01-18 13:44:20 +00:00
|
|
|
p->audio_speed = 1.0;
|
video: rewrite filtering glue code
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
2018-01-16 10:53:44 +00:00
|
|
|
p->sub.in = f->ppins[0];
|
|
|
|
p->sub.out = f->ppins[1];
|
|
|
|
|
|
|
|
return &p->public;
|
|
|
|
}
|