avformat/rmdec: Check old_format len for overflow

Maybe such large values could be disallowed earlier and closer to where
they are set.

Fixes: signed integer overflow: 538976288 * 8224 cannot be represented in type 'int'
Fixes: 29102/clusterfuzz-testcase-minimized-ffmpeg_dem_RM_fuzzer-6704350354341888

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 06d174e289)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Michael Niedermayer 2021-04-26 22:35:37 +02:00
parent c7678193cf
commit fd21f6a0cc
1 changed files with 5 additions and 3 deletions

View File

@ -1011,8 +1011,8 @@ static int rm_read_packet(AVFormatContext *s, AVPacket *pkt)
{
RMDemuxContext *rm = s->priv_data;
AVStream *st = NULL; // init to silence compiler warning
int i, len, res, seq = 1;
int64_t timestamp, pos;
int i, res, seq = 1;
int64_t timestamp, pos, len;
int flags;
for (;;) {
@ -1031,7 +1031,9 @@ static int rm_read_packet(AVFormatContext *s, AVPacket *pkt)
ast = st->priv_data;
timestamp = AV_NOPTS_VALUE;
len = !ast->audio_framesize ? RAW_PACKET_SIZE :
ast->coded_framesize * ast->sub_packet_h / 2;
ast->coded_framesize * (int64_t)ast->sub_packet_h / 2;
if (len > INT_MAX)
return AVERROR_INVALIDDATA;
flags = (seq++ == 1) ? 2 : 0;
pos = avio_tell(s->pb);
} else {