It is true that Jeremy was the only witness to that conversation, and I accept that this is an important thing to take into account because he could have made the whole thing up.
But, he can't have made it up entirely because there were other witnesses in the case who vouched for the general theme of such a discussion/kitchen table meeting. Again, though, that could be what persuaded Jeremy in his own mind to invent the discussion, knowing others would back up its plausibility.