No Free Lunch at the Slop Trough

The failure or success of an AI initiative within an enterprise comes down to "did it reduce or increase entropy?". Entropy here in the physics sense and the Shannon sense. Entropy as disorder, entropy as uncertainty.

A great use of LLMs within an enterprise context is to replace labor on tractable, measurable tasks such as classification (should this support thread be escalated?), extraction (what is the billing adress in this pdf?), summarization (what did we discuss on our last call?). Those jobs, when translated into tokens, are all compression jobs. Take many tokens, return fewer, relevant tokens.

Staying with the Shannon theme, a great mental model for LLMs is to frame them as semantic compressors, operating through moisy channels. Tokens go in, are chewed on by the weights, and other tokens come out. While the output is not the exact message that went in, in theory it is semantically close enough that you can say signal was preserved to some degree. There is also some level of noise introduced by the nature of the model.

Within an enterprise context, any semantically new concepts or entities in the ouput that were not present in the input are noise, hallucination. If I feed in a list of customer names, and the output contains Acme Co and that company wasn't in the input, that is noise.

Like all noisy channels, redundancy can help. Coherent context mitigates noisy oputput. The opposite is true, and it is where most AI use cases fail. The opposite of compression is expansion. Take a few input tokens, and expect many more output tokens.

Give an LLM a few bullet points and ask it to design an API spec for you, and it will do it in a fairly reasonable and polished way. The output will, at a pass, seem decent. But you have introduced entropy. Noise, uncertainty. Why? Because at this point in time, it seems to be the case that LLMs are very good compressors, but bad expanders.

They can reproduce patterns they've seen in their training, and they've seen a lot. And so the API spec in this case is about as reasonable and plausible as the average of all that the model has seen. Assuming that content in the training pile runs to a standard distribution of quality, you are getting about an average output.

Well average isn't bad, is it? That depends on the context. The average answer to "what's the capital of France?" is Paris, and when you ask that question the average answer is the one you want. But the average API spec is unlikely to shape out in all the nuances and details in a way that will work. There are many cases where the average is basically an abject failure. For example, when ordering a meal you would not want the average flavor and texture of all the options on the menu. I do not want shoes of the average size, and the average tailored suit is of little use to me.

And so, in offloading the work of this type to the LLM, you may have saved yourself several hours of thinking, writing, and hard work. But, that work has still not been done in any real sense. You have generated a simulcrum of the work that may or may not be correct. It is Schrodingers API requirements doc. It could be correct or incorrect, but as of yet, no one knows. It is neither 1 nor 0. And so now you, or more likely, the poor engineering team whos job it is to build the thing have to review it in detail.

It costs energy to compute and eat entropy. Those engineers meat CPUs will burn glucose far from the Landauer limit (but still much closer than GPUs) and will find things that don't make sense. They will ask questions. It will take energy to resolve Schrodingers doc.

The pernicious part of this, the true failure mode is that when LLMs are used like this it takes more effort than if someone just did it themselves in the first place. It may not be effort for that person individually, but for many people downstream. An engineer who vibecodes and doesn't check is relying on QA and PR reviewers to eat her entropy. She is locally incetivized to ship code and so she does. Somewhere, the entropy will accumulate and need resovling.

As we look at where AI and LLMs can be useful, and drive outsize returns in an enterprise (there are many places), we should keep this in mind: There is no free thermodynamic lunch. Especially at the slop trough.