lavu/tx: clip when converting table values to fixed-point

INT32_MAX (2147483647) isn't exactly representable by a floating point
value, with the closest being 2147483648.0. So when rescaling a value
of 1.0, this could overflow when casting the 64-bit value returned from
lrintf() into 32 bits.
Unfortunately the properties of integer overflows don't match up well
with how a Fourier Transform operates. So clip the value before
casting to a 32-bit int.

Should be noted we don't have overflows with the table values we're
currently using. However, converting a Kaiser-Bessel window function
with a length of 256 and a parameter of 5.0 to fixed point did create
overflows. So this is more of insurance to save debugging time
in case something changes in the future.
The macro is only used during init, so it being a little slower is
not a problem.
This commit is contained in:
Lynne 2021-01-09 20:41:25 +01:00
parent 29993b2947
commit 91e1625db1
No known key found for this signature in database
GPG Key ID: A2FEA5F03F034464

View File

@ -85,7 +85,7 @@ typedef void FFTComplex;
(dim) = (int)(((accu) + 0x40000000) >> 31); \
} while (0)
#define RESCALE(x) (lrintf((x) * 2147483648.0))
#define RESCALE(x) (av_clip64(lrintf((x) * 2147483648.0), INT32_MIN, INT32_MAX))
#define FOLD(x, y) ((int)((x) + (unsigned)(y) + 32) >> 6)