“Fast-Forward” is a bit of an analog-world concept where RPMs are a real thing, maybe. Spin the tape faster.
If you’re running a stream, or playing from a local buffer, it could be that the user experience of “fast forward” depends on how quickly the device can buffer the signal from the server. Just guessing.
Too bad Apple TV doesn’t support directly advancing the stream to a specific hh:mm:ss point.