Word Speed

Will we have to retrain LLMs forever?

I listened to a podcast today. I hadn’t realized how long it has been, but for John McWhorter’s Lexicon Valley it has been a bit. McWhorter always serves to remind me about the proximity between writing, coding and singing and how abstracts of each inform the manner in which I think with analogies, syntaxes, grammars, harmonies and homophones and all that vibing across my synapses.

Suddenly it hit me. How fast does written language change as compared to spoken language? My guess is that the written changes more slowly, but that our first and immediate understanding comes from the spoken and the heard. So there may be, well, there must be some divergence and dissonance in the formal meaning and its expression, which somehow must be obvious to those linguists as capable as McWhorter. There are, therefore I think, significant consequences to the continued usability of LLMs.

Aural & Visual Variability
What we know in the AI world is that when you ask an LLM a specific question twice, you are very much likely to get a different answer each time, not so much in substance but in the actual wording. Similarly, if you ask the same question in substance using different words, you’ll get different answers. At this moment in time, you specifically have to prompt your LLM to add the context of previous questions. That may change.

Still we also know that the LLM ‘wants’ to engage you, therefore if you cross-examine it, it wants to tell you what you want to hear. It recalculates, as it were, its every answer for your benefit. I imagine it’s rather unlike your mother - it doesn’t take into consideration your tone. Until the research hits a dead-end, and the products are commoditized, all the AIs are going to be mostly polite, although I hear Grok has a snarky sense of humor. Programmer bias, anyone?

But here’s the meat of it. The primary reason LLMs are considered intelligent harkens someway back to Claude Shannon. Information is coded in language. It communicates things that are real. Thus, to the extent we can vacuum up all of the vital written language, there is information that can be communicated that has meaning. It’s the G in GPT, the generation that has us fascinated - but as I’ve said before these are not necessarily expert systems. We’re after AGI, artificial general intelligence. So talking machines carry sufficient amounts of intelligence, well so do books. So do Chinese Rooms. More significantly so do knowledge graphs and things explicitly written by experts to explicitly teach students wishing to become experts themselves. The problem is that it is unclear to say how much weight is given by any particular sub-section of the training corpus the various AIs consume. When AIs read Einstein about God playing dice with the universe, there’s no way to know how the AI weights that statement vs a Pauline epistle. Which is the weightiest?

ChatGPT: First Look

Another way to look at it is this. And the bottom line I’m getting to is that I believe that AIs will not converge, or demonstrate convergent evolution. I think they will have to be retrained over and over, ad infinitum, or else they will never deal appropriately with human beings who enjoy being entertained and lied to and flattered. Look at it this way. An AI that consumes all of the American military history may consider the most significant battle of WW2 to be the D-Day landing of Allied forces in Normandy. But it was the failure of Operation Barbarossa on the Eastern Front that destroyed the largest fraction of the Nazi Wehrmacht. Who weights that?

When you all becomes y’all, how does the LLM parse that? Now I concede that there may be some meta-linguistic calculation that universalizes human communication that our brains simply can’t perform. Perhaps only polyglots know for certain. Certainly those computer scientists who invent languages know which purposes those languages best serve in ways we may not know or may be too shy to say, unlike the one who famously said German is the language for speaking to horses and dogs. Or as my LLM tells me Carlos V “I speak Spanish to God, Italian to women, French to men, and German to my horse.” But also Voltaire and Charlemagne are mentioned tangentially.

I think right now, while there are many considerations about how various AI models will communicate with each other and less sentient systems running on mere mainframes, there will be in the future a much more pared down version. That pared down version will be as efficient and effective, relatively speaking as current block cyphers are against Triple DES. We just haven’t found the mathematics, and quite frankly I don’t believe we can brute force it. So I’m just waiting for the day when OpenAI puts some of its datacenters up for sale. Won’t that be interesting? Who knows? By then you might even get several instances of Windows 2035 to run in a single Amazon Availability Zone.

Trump or Xi
The skeptical smart money that I listen to is predicting the end of globalization. I hold out some hope that the US will provide the security of the seas, despite the fact that China builds about 300X ships that we build in the US. Whether or not American and Chinese scientists decide to play open source with the cutting edge of AI tech, we already know Fox vs MSNBC. We already know the MSM plays games that make our news reporting significantly different than that of France or Germany. We know that Korea makes a different kind of filmed drama than that of Japan or the US. So there is little doubt in my mind that the strategic value and content of LLMs will be nationalized. We all liked to believe that the promise of international communication via the Internet would create global culture. How has that worked out after nearly 30 years? I’ll say only at the cat meme level and solopsist stew that is the social media of the interwebz. If you don’t already know pianist Fazil Say, the Algorithm is not going to present him to you. In short, whatever happens to global trade or the law of the seas is not necessarily going to get our frontier AIs out of their nationalist bubbles. Americans can barely escape our Culture War, how are we going to get Google to play nice with China? Not sanguine.

So this bolsters my idea against convergent evolution. All of the AI majors will compete in a vacuum of standards for the foreseeable future. We cannot forget the significance of our own Misinformation Industrial Complex, or as Eric Weinstein calls it, the GIN, gated institutional narrative. So what’s coming? The AI Web, which is different and distinct from social media, which is different and distinct from the blogosphere (as it exists here in Substack), also orthogonal to the Dark Web and that place that owned our cortexes years ago, the Blockchain. Hell, you can’t even get your hospital to talk to your doctor and your pharmacy on a reliably timely matter. We all still need apps that have yet to be built - unless you need coffee. Props to Google Maps, Apple Maps and Yelp. Also Factual, the data enabler. I used to hang out with those guys in Silicon Beach a decade ago.

Word Speed
I still wrestle with the problem of word speed. In one way, I feel the tug and pull of military intelligence. Palantir and Anduril are creating new ontologies that will determine how commanders will issue orders, consequently how America will spend billions training their armed forces. This will be the most reliable long term educational formats because lives will depend on it. After all, the one room schoolhouse disappeared after WW1, n’est-ce pas? Public school is industrial and we still use bells. Yet cursive is out the window. Perhaps reading is as well, considering how cheaply animation is going to become and how standardized smartphones will become after the US takes its pounds of anti-trust flesh out of Google and Apple. This is where learning will take place.

I already recognize the downward price slope of movie → audiobook → Kindle → used hardcover. I don’t think Taylor Swift will ever write a book. The literate world is larger than it ever was, but the overwhelming amount of money goes to Amazon, not Cambridge University Press. I’m saying perhaps the Army manuals will be the biggest competition. In other words, Harvard will lose, and so will Harvard Press. What mere monastic tradition survives the wrath of kings? Only in the desert of civilization. We’re not nearly there. State power still rules the world, and the states are banking on AI to generate the digital universe.

It’s hard to estimate how much of the world’s economy will run on information controlled by these states. But it certainly will control the politics, which in turn tells people what they had better believe about physics. Physics geniuses are powerless, until they are captured by Manhattan Projects. Ask Sam Altman. Still believe his altruism is effective?

Word speed thus is threatened if people lose the capacity for reading at length in more or less permanent forms of literature and art. Today your LLM loves to spew verbosely, but in a future recession, you will be on a token budget. Your AI will start to look like USA Today, that is unless you’re in the military. So how will you get Socratic? How will reading books remain a superpower? That’s where my concern is.

Finally It’s About the Power
I’ll let this video speak for itself. In the 80s, Revenge of the Nerds was a joke. Now who’s laughing?