swscale/output: Avoid undefined overflow in yuv2rgb_write_full()

Fixes: signed integer overflow: -140140 * 16525 cannot be represented in type 'int'
Fixes: 68859/clusterfuzz-testcase-minimized-ffmpeg_SWS_fuzzer-4516387130245120

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c221c7422f)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Michael Niedermayer 2024-06-16 01:59:23 +02:00
parent 709fae3a49
commit 77c7c10755
No known key found for this signature in database
GPG Key ID: B18E8928B3948D64
1 changed files with 3 additions and 3 deletions

View File

@ -1847,9 +1847,9 @@ static av_always_inline void yuv2rgb_write_full(SwsContext *c,
Y -= c->yuv2rgb_y_offset;
Y *= c->yuv2rgb_y_coeff;
Y += 1 << 21;
R = (unsigned)Y + V*c->yuv2rgb_v2r_coeff;
G = (unsigned)Y + V*c->yuv2rgb_v2g_coeff + U*c->yuv2rgb_u2g_coeff;
B = (unsigned)Y + U*c->yuv2rgb_u2b_coeff;
R = (unsigned)Y + V*(unsigned)c->yuv2rgb_v2r_coeff;
G = (unsigned)Y + V*(unsigned)c->yuv2rgb_v2g_coeff + U*(unsigned)c->yuv2rgb_u2g_coeff;
B = (unsigned)Y + U*(unsigned)c->yuv2rgb_u2b_coeff;
if ((R | G | B) & 0xC0000000) {
R = av_clip_uintp2(R, 30);
G = av_clip_uintp2(G, 30);