On the other hand, I think it’s an open question what “real” intelligence actually is, and when even people exhibit it. AI systems have a bit of a rhetorical disadvantage here because you can open them up and say, “oh, it’s just doing [foo], that’s not actually intelligent”, while the homo sapiens source code is much more obfuscated.

I’m sure there are also other people who make similar points in other domains, and that’s probably why you and I are having a hard time remembering our first encounters with these ideas. The important thing here, however, is that when I say “system” in the last sentence of the paragraph, that might be read as “computer system.” I mean it much more broadly — which I’d hoped to indicate with the non-computer example of Garfinkle’s experiment, which runs along the lines of what you suggest about some therapists. It’s probably a good idea for me to include a note, or some other explanatory text in the chapter, making it clear that I’m not just talking about software.

And it’s not only with machines where the projection can be wrong: the real therapist at such a visit might well be following a pretty mechanical learned script too, despite being incorrectly perceived as generating his responses via some complex cognitive process (that’s one explanation for why Eliza’s Doctor script worked as well as it did—that it’s actually a reasonably accurate simulation of this particular interaction). And of course all sorts of instinctual hardwire animal (and human) behaviors are misperceived as the result of complex cognition—even ant behavior tends to get perceived as intelligent.