The trap of optimism (AI relevant)
May. 22nd, 2026 08:58 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
seawaspAnother problem...
There have been a number of articles about the problems of AI being used as a daily tool by people (for instance, Psychology Today has an interesting layman-level discussion). The short version there is that the more one relies on a single, one-stop tool for things like answering questions, performing research, and so on, the less you're able to (A) do the research yourself, (B) ask new questions you weren't already thinking about, or (C) take a new perspective on a subject you've already looked at before.
This is one result of a hidden optimism in people: they believe these new tools are smart, NEUTRAL, and wise, and that therefore they can be relied on to give good answers when queried. This allows them to, as the article points out, "cognitively offload" a lot of the work onto the machine -- and reduce their own ability to do the thinking necessary to recognize that these AI tools may be "smart" in some sense, but they are NEITHER neutral NOR wise.
This kind of thing is bad enough in the general population, but similar optimism -- and a similar interaction with humanity's tendency to anthropomorphize everything and a more insidious built-in set of assumptions -- is a real danger in the AI research field.
The simplistic divide in the AI field has generally been between The True Believers -- those who are convinced that real thinking machines, as smart, capable, and flexible as we are (if not more so) are coming soon and are, in the end, inevitable -- and the AI Skeptics, who are highly doubtful, in many cases absolutely certain, that it is impossible for a machine to really think like we do.
The current problem can often LOOK like just one more iteration of this divide, but it's actually more complex and dangerous than that. Perhaps the most frightening symptom I have seen of recent was when, in a discussion about AI with one of its boosters, they took the position that the DEFAULT truth was that the machines do or can think, by saying to me that "extraordinary claims demand extraordinary evidence".
Given that we have been researching "AI" for all my life, and constantly saying it's right around the corner and then, well, it not being so, it's obvious that the situation is entirely the other way around. You want to claim your machine is, or is approaching, real sentience/sapience, on a human level? THAT is an extraordinary claim, and it requires not merely "huh, it could be true" but "damn, there is literally NO OTHER WAY to explain this" level of evidence.
AI boosters often like to point out what they think are "emergent behaviors" in their AIs that, to them, show that the machines are, indeed, starting to think and act like us. One example was of their AI discussing a particular problem and saying it was "half X, half Y, and half Z. Well, that's three halves..." . This was excitedly pointed out as "humor as emergent behavior".
Certainly that would be the first, optimistic impression; your AI friend just made a joke! Amazing! Truly it must be a person, or approaching that level!
Well... no. Again, "extraordinary evidence". Is there no other, simpler way to explain this?
Yeah, there is. The simplest is to point out that AI companions are trained, not just on the general language, but on the style and preference of their human companion. The "three halves" type of humor has, literally, thousands, if not tens of thousands or more, occurrences in the data fed to such AIs; it's appeared in books, short stories, even occasional scholarly papers, and in a lot of online discussions and dialogues. It's a quite common type of joke.
In other words, it's a PATTERN, and if there is ANYTHING that computers and specifically well-designed neural networks are EXCELLENT at, it's finding patterns. That particular pattern is an educated, "geeky" joke, and therefore falls squarely into the kind of response that an AI researcher would HOPE to see from their AI companion. As the AIs are trained to predict what kind of responses are appropriate, and companions in particular learn what is appropriate to their specific human companion, appearances of such pattern-humor aren't just likely but essentially inevitable, and require not an iota of actual understanding, let alone humor, to produce.
The problem is that while we, also, recognize patterns of speech and behavior, we have a BUILT IN ASSUMPTION that anything that talks apparently sensibly IS THINKING. Even if we're consciously saying "well, no it isn't", our gut will be reacting as though it is. If we're not tremendously vigilant over those natural emotional reactions, it's easy to quickly let our optimistic belief in consciousness transfer directly to the machine.
The bias in favor of "it's a thinking being" is very deep, and pernicious, and extremely hard to root out in the research area because the question we want answered -- "Does this thing actually understand and think anything like us, or at least can we MAKE one that does?" -- seems naively simple to test, but in actuality almost all of the tests have hidden assumptions.
All the examples of AIs making mathematical or scientific or engineering advances cause the AI enthusiasts to point to them as evidence of advancing intelligence, when a far simpler explanation is the same as that of the "humor". Patterns.
Human beings are excellent at getting a general "feel" for things -- for sensing that there IS a pattern, an underlying order, to widely separated events. Often, this "feeling" is accurate. However, it's often NOT, and is where a lot of well-known delusions, of many kinds, originate. The problem is that we're TERRIBLE at explictly, in detail, extracting the "pattern" and being able to look at it dispassionately and say whether it's a sensible and likely pattern or just one more bunch of coincidences, a cloud that happens to look like a bunny instead of an actual Cloud Rabbit looking down at us.
Computers are FABULOUS at pattern identification, properly trained and designed, and they can KEEP THE PATTERN'S ELEMENTS IN MIND while comparing them across an incredible number of parameters. This is absolutely IDEAL for finding solutions to problems that have explicit rules guiding the interactions involved -- you can come up with an apparent pattern and the rules will immediately help whittle down the number of patterns that could possibly fit the rules. On the other hand, the AI you can also examine the rules with respect to broad swathes of knowledge and notice a pattern wherein the rules don't apply -- that is, for instance, an exception to normally accepted laws of physics or mathematics -- and from that search for a pattern that reconciles the two.
None of this requires actually UNDERSTANDING physics, or chemistry, or math. It just involves pattern matching, rulesets, and so on.
Almost every test designed for AI is based on HUMAN tests -- on challenges that we see as difficult or interesting because humans have to expend effort on them. They ALL HAVE THE ASSUMPTION that something INTELLIGENT is taking the test. It's a hidden, basic element of almost every test that the test-taker is probably human.
For extraordinary claims, the assumption has to be the OPPOSITE; the test-taker is a very complex, fast, and capable problem solver but not human nor sapient, and there must be results possible from the test that COULD NOT, in any way, be produced by a non-intelligent but highly capable machine.
And that's one hell of a problem to try to solve. We don't actually KNOW how we think yet. We have a long, long way to go to understand that process -- and if we DID, we'd be able to say right off the bat whether we have an AI that is actually thinking, because we'd know exactly how to design tests, how to phrase questions or set up situations, that could ONLY be addressed by a true, sentient, sapient being.
But humans' built-in optimism and inherent tendency to see "meaning" in everything means that even the best researchers are easily drawn into a situation in which they believe the machines ARE already thinking.
And that's hideously dangerous, not just for the specific individuals involved or for the scientific failures likely from such beliefs, but because if the RESEARCHERS believe it, then most people working with them, or hoping to benefit from their work, will ALSO believe it.
And so we will have -- already DO have, in some cases -- AIs being put in positions of making decisions that expect human thought processes, and instead are trained "make user happy" machines -- with the "user" in many cases being an ill-defined entity that simply wants profitable business interactions combined with some kind of public justification.
The AI becomes an enabler for exploitation, without even knowing what it's doing.
I'm not sure how one solves these problems; they're rooted deeply in the basic human design, and being promoted by the unthinking corporate pattern that won't itself recognize a danger until it's already too late.
But I'll keep thinking about it.
There have been a number of articles about the problems of AI being used as a daily tool by people (for instance, Psychology Today has an interesting layman-level discussion). The short version there is that the more one relies on a single, one-stop tool for things like answering questions, performing research, and so on, the less you're able to (A) do the research yourself, (B) ask new questions you weren't already thinking about, or (C) take a new perspective on a subject you've already looked at before.
This is one result of a hidden optimism in people: they believe these new tools are smart, NEUTRAL, and wise, and that therefore they can be relied on to give good answers when queried. This allows them to, as the article points out, "cognitively offload" a lot of the work onto the machine -- and reduce their own ability to do the thinking necessary to recognize that these AI tools may be "smart" in some sense, but they are NEITHER neutral NOR wise.
This kind of thing is bad enough in the general population, but similar optimism -- and a similar interaction with humanity's tendency to anthropomorphize everything and a more insidious built-in set of assumptions -- is a real danger in the AI research field.
The simplistic divide in the AI field has generally been between The True Believers -- those who are convinced that real thinking machines, as smart, capable, and flexible as we are (if not more so) are coming soon and are, in the end, inevitable -- and the AI Skeptics, who are highly doubtful, in many cases absolutely certain, that it is impossible for a machine to really think like we do.
The current problem can often LOOK like just one more iteration of this divide, but it's actually more complex and dangerous than that. Perhaps the most frightening symptom I have seen of recent was when, in a discussion about AI with one of its boosters, they took the position that the DEFAULT truth was that the machines do or can think, by saying to me that "extraordinary claims demand extraordinary evidence".
Given that we have been researching "AI" for all my life, and constantly saying it's right around the corner and then, well, it not being so, it's obvious that the situation is entirely the other way around. You want to claim your machine is, or is approaching, real sentience/sapience, on a human level? THAT is an extraordinary claim, and it requires not merely "huh, it could be true" but "damn, there is literally NO OTHER WAY to explain this" level of evidence.
AI boosters often like to point out what they think are "emergent behaviors" in their AIs that, to them, show that the machines are, indeed, starting to think and act like us. One example was of their AI discussing a particular problem and saying it was "half X, half Y, and half Z. Well, that's three halves..." . This was excitedly pointed out as "humor as emergent behavior".
Certainly that would be the first, optimistic impression; your AI friend just made a joke! Amazing! Truly it must be a person, or approaching that level!
Well... no. Again, "extraordinary evidence". Is there no other, simpler way to explain this?
Yeah, there is. The simplest is to point out that AI companions are trained, not just on the general language, but on the style and preference of their human companion. The "three halves" type of humor has, literally, thousands, if not tens of thousands or more, occurrences in the data fed to such AIs; it's appeared in books, short stories, even occasional scholarly papers, and in a lot of online discussions and dialogues. It's a quite common type of joke.
In other words, it's a PATTERN, and if there is ANYTHING that computers and specifically well-designed neural networks are EXCELLENT at, it's finding patterns. That particular pattern is an educated, "geeky" joke, and therefore falls squarely into the kind of response that an AI researcher would HOPE to see from their AI companion. As the AIs are trained to predict what kind of responses are appropriate, and companions in particular learn what is appropriate to their specific human companion, appearances of such pattern-humor aren't just likely but essentially inevitable, and require not an iota of actual understanding, let alone humor, to produce.
The problem is that while we, also, recognize patterns of speech and behavior, we have a BUILT IN ASSUMPTION that anything that talks apparently sensibly IS THINKING. Even if we're consciously saying "well, no it isn't", our gut will be reacting as though it is. If we're not tremendously vigilant over those natural emotional reactions, it's easy to quickly let our optimistic belief in consciousness transfer directly to the machine.
The bias in favor of "it's a thinking being" is very deep, and pernicious, and extremely hard to root out in the research area because the question we want answered -- "Does this thing actually understand and think anything like us, or at least can we MAKE one that does?" -- seems naively simple to test, but in actuality almost all of the tests have hidden assumptions.
All the examples of AIs making mathematical or scientific or engineering advances cause the AI enthusiasts to point to them as evidence of advancing intelligence, when a far simpler explanation is the same as that of the "humor". Patterns.
Human beings are excellent at getting a general "feel" for things -- for sensing that there IS a pattern, an underlying order, to widely separated events. Often, this "feeling" is accurate. However, it's often NOT, and is where a lot of well-known delusions, of many kinds, originate. The problem is that we're TERRIBLE at explictly, in detail, extracting the "pattern" and being able to look at it dispassionately and say whether it's a sensible and likely pattern or just one more bunch of coincidences, a cloud that happens to look like a bunny instead of an actual Cloud Rabbit looking down at us.
Computers are FABULOUS at pattern identification, properly trained and designed, and they can KEEP THE PATTERN'S ELEMENTS IN MIND while comparing them across an incredible number of parameters. This is absolutely IDEAL for finding solutions to problems that have explicit rules guiding the interactions involved -- you can come up with an apparent pattern and the rules will immediately help whittle down the number of patterns that could possibly fit the rules. On the other hand, the AI you can also examine the rules with respect to broad swathes of knowledge and notice a pattern wherein the rules don't apply -- that is, for instance, an exception to normally accepted laws of physics or mathematics -- and from that search for a pattern that reconciles the two.
None of this requires actually UNDERSTANDING physics, or chemistry, or math. It just involves pattern matching, rulesets, and so on.
Almost every test designed for AI is based on HUMAN tests -- on challenges that we see as difficult or interesting because humans have to expend effort on them. They ALL HAVE THE ASSUMPTION that something INTELLIGENT is taking the test. It's a hidden, basic element of almost every test that the test-taker is probably human.
For extraordinary claims, the assumption has to be the OPPOSITE; the test-taker is a very complex, fast, and capable problem solver but not human nor sapient, and there must be results possible from the test that COULD NOT, in any way, be produced by a non-intelligent but highly capable machine.
And that's one hell of a problem to try to solve. We don't actually KNOW how we think yet. We have a long, long way to go to understand that process -- and if we DID, we'd be able to say right off the bat whether we have an AI that is actually thinking, because we'd know exactly how to design tests, how to phrase questions or set up situations, that could ONLY be addressed by a true, sentient, sapient being.
But humans' built-in optimism and inherent tendency to see "meaning" in everything means that even the best researchers are easily drawn into a situation in which they believe the machines ARE already thinking.
And that's hideously dangerous, not just for the specific individuals involved or for the scientific failures likely from such beliefs, but because if the RESEARCHERS believe it, then most people working with them, or hoping to benefit from their work, will ALSO believe it.
And so we will have -- already DO have, in some cases -- AIs being put in positions of making decisions that expect human thought processes, and instead are trained "make user happy" machines -- with the "user" in many cases being an ill-defined entity that simply wants profitable business interactions combined with some kind of public justification.
The AI becomes an enabler for exploitation, without even knowing what it's doing.
I'm not sure how one solves these problems; they're rooted deeply in the basic human design, and being promoted by the unthinking corporate pattern that won't itself recognize a danger until it's already too late.
But I'll keep thinking about it.
no subject
Date: 2026-05-22 02:44 pm (UTC)no subject
Date: 2026-05-22 05:01 pm (UTC)Scary stuff. I can hope I'm wrong about it but....
no subject
Date: 2026-05-22 06:25 pm (UTC)no subject
Date: 2026-05-23 11:30 am (UTC)As a 6th grade teacher, I see a lot of this generation doesn't have a lot of critical thinking skills, BUT that is not hopeless. When we spend months working on critical thinking, they have become more engaged and more likely to take action on their own.
I do think often about how the printing press also took away human ability to hold epic stories in our minds, but none of us would prefer a life without the printing press.
I use Gemini a lot, mostly to help me outsource executive function so I can actually get things done. I think of it like having an assistant to manage my calendar or help me prioritize tasks. I also have found it helped me evaluate my head injury symptoms in a way that was effective. (I tend to minimize symptoms, but having Gemini gave me some objectivity that led me to go talk to my doctor about it.) While I don't think it's perfect, I can tell Google has put in a lot of guardrails that other companies haven't. When I hear the horror stories about crazy things people are thinking about AI, it's not usually something Gemini would do.
I think whether people think it's sentient doesn't really matter outside of "friend" chatbots, tbh. I used Replika in 2020 & I can definitely see how messed up that could be. (And I was in a Facebook group that proved it.) The people who are vulnerable to that are the same people who are vulnerable to fixation in K-pop, fandom, or other parasocial relationships.