Masking a video

I don’t think I’ve ever tried masking video before and I’m having trouble. It starts off unmasked, which is fine. Soon as my mask kicks in on a keyframe the video restarts. The video runs from the timeline and only has the one keyframe at the start whereas the mask layer starts with an empty keyframe and then comes in with a tween. I tested another video with another mask in another file and got similar results. Am I doing something wrong or is this just the way it is?