The Doctor remembered those things happening, but Jack wasn't in his memories. The gap where he was had been filled by Rose or Adam or Leo. Jack wasn't there, but how could he know?

Jack didn't come flooding back into his memory as the Doctor had hoped. There were still huge gaps that needed filling, and every thought about the gaps wracked the Doctor with more pain.

"So time hasn't been altered, just our recollection of it," he said, looking down at the cut on his thumb. "Removing me from your history and you from mine. But why? What would that accomplish, short of us mistrusting each other?"