Closes#20994.
I haven't considered how this affects performance of difficulty
calculation or otherwise. Seems like a sane initial fix which we can
iterate on going forward.
I've tested using the scenario in the linked issue.
I'm not going to add test coverage because
[BeatmapConversionTest](https://github.com/ppy/osu/blob/master/osu.Game/Tests/Beatmaps/BeatmapConversionTest.cs)
should correctly cover this scenario once we are serialising control
points as well.
