stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
/*
|
|
|
|
* This file is part of mpv.
|
|
|
|
*
|
|
|
|
* mpv is free software; you can redistribute it and/or
|
|
|
|
* modify it under the terms of the GNU Lesser General Public
|
|
|
|
* License as published by the Free Software Foundation; either
|
|
|
|
* version 2.1 of the License, or (at your option) any later version.
|
|
|
|
*
|
|
|
|
* mpv is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU Lesser General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU Lesser General Public
|
|
|
|
* License along with mpv. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <archive.h>
|
|
|
|
#include <archive_entry.h>
|
|
|
|
|
|
|
|
#include "common/common.h"
|
|
|
|
#include "common/playlist.h"
|
2020-01-04 18:53:57 +00:00
|
|
|
#include "options/m_config.h"
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
#include "stream/stream.h"
|
2018-07-27 03:01:35 +00:00
|
|
|
#include "misc/natural_sort.h"
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
#include "demux.h"
|
|
|
|
|
|
|
|
#include "stream/stream_libarchive.h"
|
|
|
|
|
2020-01-04 18:53:57 +00:00
|
|
|
struct demux_libarchive_opts {
|
2023-02-20 03:32:50 +00:00
|
|
|
bool rar_list_all_volumes;
|
2020-01-04 18:53:57 +00:00
|
|
|
};
|
|
|
|
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
static int cmp_filename(const void *a, const void *b)
|
|
|
|
{
|
2018-07-27 03:01:35 +00:00
|
|
|
return mp_natural_sort_cmp(*(char **)a, *(char **)b);
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static int open_file(struct demuxer *demuxer, enum demux_check check)
|
|
|
|
{
|
2016-12-04 22:15:31 +00:00
|
|
|
if (!demuxer->access_references)
|
|
|
|
return -1;
|
|
|
|
|
2015-08-17 21:59:44 +00:00
|
|
|
int flags = 0;
|
2015-08-24 20:21:36 +00:00
|
|
|
int probe_size = STREAM_BUFFER_SIZE;
|
|
|
|
if (check <= DEMUX_CHECK_REQUEST) {
|
2015-08-17 21:59:44 +00:00
|
|
|
flags |= MP_ARCHIVE_FLAG_UNSAFE;
|
2015-08-24 20:21:36 +00:00
|
|
|
probe_size *= 100;
|
|
|
|
}
|
|
|
|
|
stream: turn into a ring buffer, make size configurable
In some corner cases (see #6802), it can be beneficial to use a larger
stream buffer size. Use this as argument to rewrite everything for no
reason.
Turn stream.c itself into a ring buffer, with configurable size. The
latter would have been easily achievable with minimal changes, and the
ring buffer is the hard part. There is no reason to have a ring buffer
at all, except possibly if ffmpeg don't fix their awful mp4 demuxer, and
some subtle issues with demux_mkv.c wanting to seek back by small
offsets (the latter was handled with small stream_peek() calls, which
are unneeded now).
In addition, this turns small forward seeks into reads (where data is
simply skipped). Before this commit, only stream_skip() did this (which
also mean that stream_skip() simply calls stream_seek() now).
Replace all stream_peek() calls with something else (usually
stream_read_peek()). The function was a problem, because it returned a
pointer to the internal buffer, which is now a ring buffer with
wrapping. The new function just copies the data into a buffer, and in
some cases requires callers to dynamically allocate memory. (The most
common case, demux_lavf.c, required a separate buffer allocation anyway
due to FFmpeg "idiosyncrasies".) This is the bulk of the demuxer_*
changes.
I'm not happy with this. There still isn't a good reason why there
should be a ring buffer, that is complex, and most of the time just
wastes half of the available memory. Maybe another rewrite soon.
It also contains bugs; you're an alpha tester now.
2019-11-06 20:36:02 +00:00
|
|
|
void *probe = ta_alloc_size(NULL, probe_size);
|
|
|
|
if (!probe)
|
2015-08-24 20:21:36 +00:00
|
|
|
return -1;
|
stream: turn into a ring buffer, make size configurable
In some corner cases (see #6802), it can be beneficial to use a larger
stream buffer size. Use this as argument to rewrite everything for no
reason.
Turn stream.c itself into a ring buffer, with configurable size. The
latter would have been easily achievable with minimal changes, and the
ring buffer is the hard part. There is no reason to have a ring buffer
at all, except possibly if ffmpeg don't fix their awful mp4 demuxer, and
some subtle issues with demux_mkv.c wanting to seek back by small
offsets (the latter was handled with small stream_peek() calls, which
are unneeded now).
In addition, this turns small forward seeks into reads (where data is
simply skipped). Before this commit, only stream_skip() did this (which
also mean that stream_skip() simply calls stream_seek() now).
Replace all stream_peek() calls with something else (usually
stream_read_peek()). The function was a problem, because it returned a
pointer to the internal buffer, which is now a ring buffer with
wrapping. The new function just copies the data into a buffer, and in
some cases requires callers to dynamically allocate memory. (The most
common case, demux_lavf.c, required a separate buffer allocation anyway
due to FFmpeg "idiosyncrasies".) This is the bulk of the demuxer_*
changes.
I'm not happy with this. There still isn't a good reason why there
should be a ring buffer, that is complex, and most of the time just
wastes half of the available memory. Maybe another rewrite soon.
It also contains bugs; you're an alpha tester now.
2019-11-06 20:36:02 +00:00
|
|
|
int probe_got = stream_read_peek(demuxer->stream, probe, probe_size);
|
2019-06-19 14:48:46 +00:00
|
|
|
struct stream *probe_stream =
|
stream: turn into a ring buffer, make size configurable
In some corner cases (see #6802), it can be beneficial to use a larger
stream buffer size. Use this as argument to rewrite everything for no
reason.
Turn stream.c itself into a ring buffer, with configurable size. The
latter would have been easily achievable with minimal changes, and the
ring buffer is the hard part. There is no reason to have a ring buffer
at all, except possibly if ffmpeg don't fix their awful mp4 demuxer, and
some subtle issues with demux_mkv.c wanting to seek back by small
offsets (the latter was handled with small stream_peek() calls, which
are unneeded now).
In addition, this turns small forward seeks into reads (where data is
simply skipped). Before this commit, only stream_skip() did this (which
also mean that stream_skip() simply calls stream_seek() now).
Replace all stream_peek() calls with something else (usually
stream_read_peek()). The function was a problem, because it returned a
pointer to the internal buffer, which is now a ring buffer with
wrapping. The new function just copies the data into a buffer, and in
some cases requires callers to dynamically allocate memory. (The most
common case, demux_lavf.c, required a separate buffer allocation anyway
due to FFmpeg "idiosyncrasies".) This is the bulk of the demuxer_*
changes.
I'm not happy with this. There still isn't a good reason why there
should be a ring buffer, that is complex, and most of the time just
wastes half of the available memory. Maybe another rewrite soon.
It also contains bugs; you're an alpha tester now.
2019-11-06 20:36:02 +00:00
|
|
|
stream_memory_open(demuxer->global, probe, probe_got);
|
2020-01-04 17:59:23 +00:00
|
|
|
struct mp_archive *mpa = mp_archive_new(mp_null_log, probe_stream, flags, 0);
|
2015-08-24 20:21:36 +00:00
|
|
|
bool ok = !!mpa;
|
|
|
|
free_stream(probe_stream);
|
|
|
|
mp_archive_free(mpa);
|
stream: turn into a ring buffer, make size configurable
In some corner cases (see #6802), it can be beneficial to use a larger
stream buffer size. Use this as argument to rewrite everything for no
reason.
Turn stream.c itself into a ring buffer, with configurable size. The
latter would have been easily achievable with minimal changes, and the
ring buffer is the hard part. There is no reason to have a ring buffer
at all, except possibly if ffmpeg don't fix their awful mp4 demuxer, and
some subtle issues with demux_mkv.c wanting to seek back by small
offsets (the latter was handled with small stream_peek() calls, which
are unneeded now).
In addition, this turns small forward seeks into reads (where data is
simply skipped). Before this commit, only stream_skip() did this (which
also mean that stream_skip() simply calls stream_seek() now).
Replace all stream_peek() calls with something else (usually
stream_read_peek()). The function was a problem, because it returned a
pointer to the internal buffer, which is now a ring buffer with
wrapping. The new function just copies the data into a buffer, and in
some cases requires callers to dynamically allocate memory. (The most
common case, demux_lavf.c, required a separate buffer allocation anyway
due to FFmpeg "idiosyncrasies".) This is the bulk of the demuxer_*
changes.
I'm not happy with this. There still isn't a good reason why there
should be a ring buffer, that is complex, and most of the time just
wastes half of the available memory. Maybe another rewrite soon.
It also contains bugs; you're an alpha tester now.
2019-11-06 20:36:02 +00:00
|
|
|
ta_free(probe);
|
2015-08-24 20:21:36 +00:00
|
|
|
if (!ok)
|
|
|
|
return -1;
|
|
|
|
|
2020-01-04 18:53:57 +00:00
|
|
|
struct demux_libarchive_opts *opts =
|
|
|
|
mp_get_config_group(demuxer, demuxer->global, demuxer->desc->options);
|
|
|
|
|
|
|
|
if (!opts->rar_list_all_volumes)
|
2020-04-27 08:13:59 +00:00
|
|
|
flags |= MP_ARCHIVE_FLAG_NO_VOLUMES;
|
2020-01-04 18:53:57 +00:00
|
|
|
|
2020-01-04 17:59:23 +00:00
|
|
|
mpa = mp_archive_new(demuxer->log, demuxer->stream, flags, 0);
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
if (!mpa)
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
struct playlist *pl = talloc_zero(demuxer, struct playlist);
|
|
|
|
demuxer->playlist = pl;
|
|
|
|
|
|
|
|
char *prefix = mp_url_escape(mpa, demuxer->stream->url, "~|");
|
|
|
|
|
|
|
|
char **files = NULL;
|
|
|
|
int num_files = 0;
|
|
|
|
|
2016-07-18 10:44:56 +00:00
|
|
|
while (mp_archive_next_entry(mpa)) {
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
// stream_libarchive.c does the real work
|
2019-12-20 07:35:08 +00:00
|
|
|
char *f = talloc_asprintf(mpa, "archive://%s|/%s", prefix,
|
2016-07-18 10:44:56 +00:00
|
|
|
mpa->entry_filename);
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
MP_TARRAY_APPEND(mpa, files, num_files, f);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (files)
|
|
|
|
qsort(files, num_files, sizeof(files[0]), cmp_filename);
|
|
|
|
|
|
|
|
for (int n = 0; n < num_files; n++)
|
|
|
|
playlist_add_file(pl, files[n]);
|
|
|
|
|
2019-12-28 20:12:02 +00:00
|
|
|
playlist_set_stream_flags(pl, demuxer->stream_origin);
|
stream, demux: redo origin policy thing
mpv has a very weak and very annoying policy that determines whether a
playlist should be used or not. For example, if you play a remote
playlist, you usually don't want it to be able to read local filesystem
entries. (Although for a media player the impact is small I guess.)
It's weak and annoying as in that it does not prevent certain cases
which could be interpreted as bad in some cases, such as allowing
playlists on the local filesystem to reference remote URLs. It probably
barely makes sense, but we just want to exclude some other "definitely
not a good idea" things, all while playlists generally just work, so
whatever.
The policy is:
- from the command line anything is played
- local playlists can reference anything except "unsafe" streams
("unsafe" means special stream inputs like libavfilter graphs)
- remote playlists can reference only remote URLs
- things like "memory://" and archives are "transparent" to this
This commit does... something. It replaces the weird stream flags with a
slightly clearer "origin" value, which is now consequently passed down
and used everywhere. It fixes some deviations from the described policy.
I wanted to force archives to reference only content within them, but
this would probably have been more complicated (or required different
abstractions), and I'm too lazy to figure it out, so archives are now
"transparent" (playlists within archives behave the same outside).
There may be a lot of bugs in this.
This is unfortunately a very noisy commit because:
- every stream open call now needs to pass the origin
- so does every demuxer open call (=> params param. gets mandatory)
- most stream were changed to provide the "origin" value
- the origin value needed to be passed along in a lot of places
- I was too lazy to split the commit
Fixes: #7274
2019-12-20 08:41:42 +00:00
|
|
|
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
demuxer->filetype = "archive";
|
|
|
|
demuxer->fully_read = true;
|
|
|
|
|
|
|
|
mp_archive_free(mpa);
|
demux: change hack for closing subtitle files early
Subtitles (and a few other file types, like playlists) are not streamed,
but fully read on opening. This means keeping the file handle or network
socket open is a waste of resources and could cause other weird
behavior. This is why there's a hack to close them after opening.
Change this hack to make the demuxer itself do this, which is less
weird. (Until recently, demuxer->stream ownership was more complex,
which is why it was done this way.)
There is some evil shit due to a huge ownership/lifetime mess of various
objects. Especially EDL (the currently only nested demuxer case)
requires being careful about mp_cancel and passing down stream pointers.
As one defensive programming measure, stop accessing the "stream"
variable in open_given_type(), even where it would still work. This
includes removing a redundant line of code, and removing the peak call,
which should not be needed anymore, as the remaining demuxers do this
mostly correctly.
2018-09-07 21:02:36 +00:00
|
|
|
demux_close_stream(demuxer);
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2020-01-04 18:53:57 +00:00
|
|
|
#define OPT_BASE_STRUCT struct demux_libarchive_opts
|
|
|
|
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
const struct demuxer_desc demuxer_desc_libarchive = {
|
|
|
|
.name = "libarchive",
|
|
|
|
.desc = "libarchive wrapper",
|
|
|
|
.open = open_file,
|
2020-01-04 18:53:57 +00:00
|
|
|
.options = &(const struct m_sub_options){
|
|
|
|
.opts = (const struct m_option[]) {
|
2023-02-20 03:32:50 +00:00
|
|
|
{"rar-list-all-volumes", OPT_BOOL(rar_list_all_volumes)},
|
2020-01-04 18:53:57 +00:00
|
|
|
{0}
|
|
|
|
},
|
|
|
|
.size = sizeof(OPT_BASE_STRUCT),
|
|
|
|
},
|
stream: libarchive wrapper for reading compressed archives
This works similar to the existing .rar support, but uses libarchive.
libarchive supports a number of formats, including zip and (most of)
rar.
Unfortunately, seeking does not work too well. Most libarchive readers
do not support seeking, so it's emulated by skipping data until the
target position. On backwards seek, the file is reopened. This works
fine on a local machine (and if the file is not too large), but will
perform not so well over network connection.
This is disabled by default for now. One reason is that we try
libarchive on every file we open, before trying libavformat, and I'm not
sure if I trust libarchive that much yet. Another reason is that this
breaks multivolume rar support. While libarchive supports seeking in
rar, and (probably) supports multivolume archive, our support of
libarchive (probably) does not. I don't care about multivolume rar, but
vocal users do.
2015-08-16 22:55:26 +00:00
|
|
|
};
|