AI, Testing, and Intelligence

[seawasp]

Below, with a cut for length...



I've had an ongoing discussion with a gentleman (who I respect and am not naming because who he is is not relevant except for the origin of the post) about current work in AI, and to an extent I believe we've been talking past each other. 

To be clear, I absolutely believe that current LLM or near-future related "huge trained neural network" systems  will be able to produce output that is able to fool anyone on the planet into believing it's a competent human discussing the key field, or writing a short story, etc., assuming the human doesn't know they're talking to an AI, and an honest human will admit they couldn't tell the difference if they hadn't known. 

What I don't believe is that any AI primarily derived from such architecture approaches is, or can be, actually intelligent. I also believe that a LOT of researchers in this area are earnestly trying to test intelligent features of AI using tests built for humans, without understanding the absolutely FUNDAMENTAL problem with that.

The problem goes right to the most basic issue of human communication: all our interactions with language and such are predicated on the assumption that what we are talking to IS INTELLIGENT. This is what makes us personify practically everything in the world. A lot of scientific experiment design is predicated on the assumption that we have to exclude human bias from the experiment. 

Every single one of the tests of intellectual, emotional, or other human capacities are specifically designed with the assumption that the answers are being provided by human beings. They even contain questions specifically targeted to finding signs of human dishonesty, as the proper results of the tests rely on the assumption that the person may, or may not, be honest, and the honest answers are the important ones for diagnostics and evaluation. 

An AI presented with such a query has no actual understanding of what's being asked. It's projecting what a likely answer for this questioh, in the context of prior questions or setup given it. The larger AIs also undoubtedly include in their training, in addition to a dozen of my books and a few million other documents, psychological papers and books discussing these tests, their expected results, and methods for recognizing false responses. How all that data interacts within the model isn't even vaguely known, but it does mean that the model has data for predicting "proper" responses, depending on how well it was trained. 

And those responses will mean absolutely nothing, because that's not what the tests are made to measure. They are designed on the assumption that there is a self-motivated, aware being responding, not that there is a machine, or a human, specifically focused on calculating "appropriate responses". Even someone trying to lie is generally doing so consciously and for a purpose. 

Current AI is founded on trained prediction. There are ways of tweaking it (making the model "warm" or "cold", etc.) to bias the way in which the prediction is done, but -- at least insofar as anyone has yet been able to describe it to me -- none of them are using fundamentally different core processes. And that's absolutely not going to produce intelligence.

We *use* such processes in our brains, undoubtedly, but we have a vast number of other processes going on at the same time which we don't understand yet. 

Right now, I'm halfway to believing that the only way we'll get true AI is to replicate a human being -- down to the molecular level -- digitally, and raise it just like a human in its virtual existence, monitoring the changes and operations from the beginning so that we can see how the actual learning, thinking, and most importantly volitional SELF processes come to exist.

What all this means for the current AI wave is that it is absolutely possible that it will CONVINCE people it can do all those jobs through dialogue output -- because that's the kind of thing it's made to do, predict what a "proper" response should be to the questions and requests given it. Unfortunately, it actually can't go beyond such predictions -- and at some point, when its new job would have a human be deciding "huh, I need to figure out a different approach", it will generate responses that are as close as possible to what it CALCULATES should be correct based on training, but not based on thought outside of that. 

Without actual understanding of the problems facing it, the longer it's asked to do a task, the more it will start to "drift", like an IMU being run without some form of ground truth. If you got one to replace me writing fiction, it might sound like me for a while, might even have a short story or two that felt like me, but once it had to get to the point of writing a longer, complex work connected with my other works, it would collapse -- because it has no methods for determining consistency, emotional beats, etc. It can IMITATE some of these, but it's doing so purely by predictive training 

This is, as I said, based on the modern AIs being primarily founded on large-scale neural nets trained on massive amounts of data to give "appropriate" responses. The ones used in research are trained more to locate "patterns" and fit those patterns into other frameworks. Even both of these together don't make something intelligent, though both of these functions would be needed. 

If there's actually other fundamentally different approaches being used, that might alter my opinion, but thus far no one has shown me what other fundamentally different mechanisms are at work. 
 

Comments

[sturgeonslawyer]

Modern LLMs are nearly-perfect instatiation of John Searle's "Chinese Room." They have, literally, no idea what they are talking about, because they have no ideas at all.

And yet LLMs, and more importantly their forthcoming big brothers, so-called Large World Models, have the potential to solve many difficult problems ... if their output can ever be made trustworthy.

We shall see.

[seawasp]

The problem with the Chinese Room is that it assumed that tne output of the Room was always appropriate, which LLMs aren't; they're just MOSTLY so. To make the Chinese Room work, he had a human being in there doing the work, so he was playing a shell game, putting a human-intentional unit inside the room but pretending there was no intentionality because the human being didn't understand the meaning of the output even though they DID understand the purpose and complexity of the task at hand and had to carry it out with full human comprehension.

This is the Chinese Room without a human inside, just a probabilistic weighted prediction machine.

There are numerous tasks that LLMs and the other neural-network trained machines could do, but the ones they absolutely shouldn't be doing are the ones people are trying to make them do.

[ninjarat]

In the cases of the commercial chatbots like ChatGPT and Grok, prediction machines weighted to provide responses that users want to see, or what the operators of these systems want users to see, over factual responses.

As for what neural networks should be doing, my go to is the credit card industry which started using neural networks in the 1980s to identify fraudulent activity. Neural networks are good at building patterns and at identifying aberrations in those patterns -- exactly what fraud detection requires.

One of my earliest exposures to neural networks as a user was Apple Newton. Rosetta, the handwriting recognition system Newton used, is a neural network. You train it on your handwriting and it gets better at recognizing what you write and converting to text. Out of the box was terrible but that's because it was untrained. Give it a chance to learn your handwriting and it would get very accurate. Unless you were left handed in which case the first version struggled but this was corrected in version 2.

[seawasp]

I wonder how they do fraud detection now? Because however they do it, it sucks.

[graycardinal]

The other problem with "trained prediction" is that it disregards one of the key elements of reality - out here in the real world, improbable stuff happens all the time, and the more improbable a thing is, the more likely an AI as they're currently being built is going to screw up in dealing with it. (This is somewhat demonstrated by way of the degree to which my local weather forecasts have varied a lot this week, changing from day to day and being way more wrong more often than is usual. Why, yes, I do live in the path of a recent atmospheric river....)