AI Evaluation After A Few Weeks Working With It...

[seawasp]

 

My employer made it clear that I was going to have to make use of AI, specifically ChatGPT 5.5, in the course of my duties. I did express my philosophical objections, but in this case sticking to my ideals, while far preferable, was not an affordable position for my family.

 

Having been forced into this position, therefore, I have chosen not to be an ass and drag my feet, but have focused on learning this new toolset and understanding what it does and does not do. The advantage of this is that now I can speak from a point of view of some experience, rather than merely theoretical understanding of these systems’ operations.

 

I’ve learned quite a bit in so doing. Current (as in, GPT5.5 – obviously I don’t know about “to be released” versions) AI demonstrates, very clearly, that is is a powerful and flexible pattern-matching system, with considerable ability to extrapolate, and now that it has access to WWW search capabilities, is able to locate sources and materials of interest to its current task.

 

What it is NOT is intelligent in any recognizable sense. It can SOUND intelligent. I am sure that in the case of the tailored LLMs made for companion/assistant roles, it can maintain a fairly convincing facade of being a helpful and reasonably competent aide or friend.

 

But that is ALL it is, and it’s very easy to “break” that facade if you want to, or even simply in the process of trying to do any kind of sustained work. I don’t know about the others, but ChatGPT 5.5 is at least honest about that; it will tell you exactly what obviously-not-thinking-but-logical process it followed to give you a delusional or simply incorrect answer.

 

That said, there ARE things it can do very well, or assist in, and it’s really quite a shame that the people making it couldn’t be bothered to properly pay for and license the material they trained them on, because if they were built with ethical methods and purposes, they could be great assistants; and in fact, it’s going to be a decent assistant for the specific sort of work that I do (at my primary employer).

 

This DOES NOT, note, change my basic position about current AI. I do not, and will not, be using it in my preferred role (as an author), even if in my specific employment I will have to do so.

 

So what CAN it do?

 

It can analyze and summarize documents, generally pretty well. It can extract structures, such as formats – in my case, proper submission formats for proposals and related documents. It can do some general research on a given topic, if you’re good at formulating a description of what to look for, and you can tell it to retain and provide sources, so you can make sure it’s not confabulating. With that and some guidance, it could even provide a reasonable first draft of some kinds of documents.

 

However, as it doesn’t actually understand anything it says, merely generates likely patterns of response, you HAVE to check everything it writes. It can be speaking very coherently and properly about a topic and then start wandering off and assert untruths, or switch referents in a sentence, or otherwise make both subtle and glaring errors.

I do have to emphasize this. It does not understand a word it says. This is demonstrated to me numerous times every day, although it can, and does, sound like it understands because the patterns of speech and response have been taught to it in grand and terrible volumes of training.

 

This lack of understanding is shown most easily in the making of images. ChatGPT5.5 includes an image generation capability, and this can at first appear to be absolutely brilliant. But any scrutiny of an image meant to perform a function (a diagram of a concept, etc.) will reveal small-to-large flaws that simply would not be done by even a mediocre human artist, because it does not understand objects as objects, designs as designs with functions, and so on.

 

This doesn’t mean its image generation capabilities are useless – far from it – but that they absolutely cannot be TRUSTED without careful oversight. One thing I’ve learned is to not only create very careful, specific prompts, but to stage the design, freezing an image whenever it has achieved some particular goal and then only doing changes based off the new, frozen design. Even then, sometimes an instruction will cause a reversion to prior training that results in truly hilariously wrong images (such as a railcar with a lovely axle and wheelset sitting beneath it at right angles to its proper location).

 

Properly watched over and tweaked, it can in fact produce quite useful and presentable images, far better than I could make myself with the tools I’m halfway competent with (Visio and SketchUp, primarily). It’s not nearly as good as what a good mechanical engineer/modeling person could do in the same or less time, but it DOES allow me to do better diagrams without having to take up the very expensive and limited time of our engineers.

 

I have also done some experimentation with analysis of writing (my own, not using anyone elses) to see what its pattern-matching sees. Again, I’m not going to USE any of its output, but it’s an important other step to see exactly how it works with the kind of writing I know best of all. (Note that the material I input is not used for training of the overall model, so it’s not being handed to GPT to use except in the specific limited chat involved).

 

Here it can be quite convincing about “understanding”, by noticing patterns, extracting relationships, and generating comments on them. It’s also trained to emulate proper conversational styles, so it will say things like “That’s really impressive” or “I particularly liked X”, even though it doesn’t actually have any of these sensations. Nonetheless, as such conversations continue, it inevitably gives away its lack of understanding by making statements about what it “read” that are clearly dead wrong in a way that a human reader would never manage.

 

Of course it’s also obvious that it’s not human because it has effectively infinite patience and no actual, personal input. It never gets tired of being talked at about a given subject, as opposed to even the most patient human being, never loses the apparent enthusiasm, never refuses any task (I suppose there must be some it’s actually programmed to refuse, but I haven’t come across any), and never brings up any subject on its own.

 

To sum up, it’s a potentially useful tool that LOOKS more generally applicable than prior work – and in some ways, it is. But it literally does not understand what it’s doing, and that can be disastrous in both smaller and larger ways.

 

It is a genuine shame that it, and its relatives, were all built on stolen work without permission or even acknowledgement.

Comments

[rdm]

This basically tracks with my experiences my $dayjob.

And even if the data used to build these models were ethically obtained, I also would not touch them for any form of fiction writing. And that is not even considering the environmental costs, or the ownerships of the tools.

[jreynoldsward]

It sounds like the one use I could think of for it (based on my years of special ed case management) would definitely be useful--using it to draft consistent Individual Education Plan/Program documents. There are portions of the IEP--specifically, the narrative involved with Present Level of Performance summary, plus the Goals and Objectives--where having a consistent style and standard enforced by genAI could be useful. I worked in a district where those standards were somewhat enforced, but I got far too many IEPs that were, essentially, useless because the case manager wrote the bare minimum.

Plus then it would be feasible to write IEPs for EVERY student, not just special education-qualified students.

[dsrtao]

The general algorithm for working with LLMs is unfortunately close to Bogosort, the canonically worst sorting algorithm.

For those who don't recall, Bogosort goes like this:

1. Randomize the order of the items to be sorted.
2. Check if the list is now sorted properly.
3. If the list is sorted, we are done. If not, return to step 1.

For LLMs:

1. Use an LLM to generate stuff.
2. Check every possible aspect of the stuff for correctness.
3. If everything was correct, we are done. If not, return to step 1.

[claidheamhmor]

We're using it at work for various things, but my best use is using it as a chatbot to find information in all of our documentation.

For building model aircraft, I use the various AIs to look up colour and detail information. Sometimes it can be dramatically wrong, and argumentative about it, even showing me pictures to prove it's right, when the pictures show the contrary. Claude is the most reliable, and honest about limitations. Gemini is pretty decent, but must be checked. ChatGPT and Copilot can be hilariously wrong.