"Ooooh, I think we both know better than that, Captain," the Doctor said. "Look at you! You're fixed! Nothing would affect your memory. Certainly not a low-scale time alteration by a cheap Time Agent."

That had to be it. Sure, the low-scale time alteration probably wouldn't hit the Doctor at all, but this man must've been clever, to catch the Doctor so off-guard.

He pointed at Jack, unimpressed with his attitude. "You're trying to play with me, and I'm not the sort of man to be played with."