Let me explain you the issue in more details because I think that the mistake lays somewhere else.
I have validation on whether I will allow the user to swipe to the next page. To do that I use the event handler OnSelectedIndexChanged that you have and if the validation fails I call the following method:
rotator.SetValue(SfRotator.SelectedIndexProperty, rotator.SelectedIndex - 1);
This works perfectly when I just scroll once to the next page. It fails though if I scroll fast many times.