You have to think of it in terms of "current time Desmond" and "past Desmond". By the time Daniel talks to "past Desmond", "current Desmond" has progressed past the point where Daniel & Desmond would have originally met on the island (Daniel hadn't time jumped back yet to meet him), so that's why he wouldn't have remembered. He essentially has a new memory created of the meeting, but since it is 3 years later in Desmond's time, that's when he is able to remember it.
At least, that's how I'm rationalizing it.