This is an understandable argument, and I tangentially addressed it in a short paragraph. To explain in detail:

What makes you think that seeing it again somehow confirms that the memory was real? Most memories cannot be repeated, since they are of one-time events. If one consistently followed the argument given, they would not be able to confirm that most memories were real, and would have to deny those. More likely, of course, you know which memories can (sometimes) be confirmed by looking in a certain way, and which can't, and in what cases each applies, and what the exceptions are, etc. If another wallet were present in the place you expected yours, you would invent a story as to why you got confused. If someone told you they moved your wallet, you would still believe the memory was real, but specifically because you *don't* see it there. None of this is an automatic function of the brain, it involves complex learned reasoning about how certain things in the world work. Toddlers often fail at this reasoning, and must be taught its nuances. And there are too many exceptions for these to be hard rules. Hence "memory is learned, not natural."

But most importantly, why would you want to confirm that your though of the wallet was a "memory"? Surely all you care about is that you can find the wallet, no matter where it is, or how it got there. The directing thought need only be useful, not necessarily a reflection of reality. To confirm a memory was of something historical and real takes extra effort behind this, and that effort needs a justification - you don't do it for every memory. What is the justification? What situation would compel you to take extra action and extra time for no other reason than to confirm a thought reflected something real in the past? The answer is given - to confirm our shared reality. You may do it to confirm your own sanity, but consider then how you define "sanity". What is the only known way to confirm you are not having hallucinations? Hint, it's social confirmation. And at that point it doesn't matter if it's a shared delusion.

Again, I am not denying you can have thoughts that reflect experiences, I'm arguing that "memory" defined as an accurate thought-representation of reality only has meaning in a social context. And "memory" as someone noted, means explicit memory, that is, something you would name as a "memory".