This back and forth got me thinking about the notion of intentionality, and in an attempt to clarify my earlier observation, I ended up writing a post about it: https://ykulbashian.medium.com/why-is-it-so-hard-to-know-what-thoughts-are-about-5d02300fc3e9

In brief: although intentionality has an intuitive appeal, is encounters difficulties when trying to actually pin down what a thought is about.

Intentionality may be a path to improving current AI. But that is not my goal. My goal is to redefine AI from the ground up, replacing many of its most fundamental assumptions. Neither probabilistic thinking, nor intentionality are adequate to that work.