I usually leave mine on game mode for day-to-day stuff because the mouse latency is unbearable otherwise. However when watching a movie or TV show, I’ll swap it back over to cinema mode to smooth out frames and reduce artifacting.

As a fellow TV-as-a-monitor enthusiast, it might be your display. Most TVs do a lot of frame processing to give you smoother playback at the cost of latency, making games feel sluggish. My TV has a “Gaming” preset that actually turns off some of that processing, allowing for 60hz refresh rate and minimal latency. Might be worth checking your TV for a setting like that.