This is much worse [than Photos.app’s Enhanced Visual Search]: open an image containing a “landmark” in the Preview app, then open the Inspector. An icon will appear which, when clicked, will popup information about the landmark. Verified with a JPEG with no GPS metadata. Take a screenshot of the opened image, the same works in the screenshot. This is system level and turning it off in the Photos app has no effect.
I found that Preview will identify landmarks—i.e. it is sending a neural hash of my photo to Apple’s servers—even without my having the inspector open and even though I have Enhanced Visual Search unchecked in Photos.app’s settings. I think this is due to the Visual Look Up feature introduced in macOS 12.3. The way to turn this off is unintuitive. It’s not in the settings for Photos or Preview or even Privacy. Instead, you go to System Settings ‣ Spotlight and uncheck Siri Suggestions.
I don’t have iCloud Photos enabled. I don’t have Enhanced Visual Search enabled. And I wasn’t doing anything with photos. This just happened randomly in the background.