AI code generation as an agent of tech debt creation
Take a moment to let the shock, horror, and betrayal to pass through you (or, potentially, the giddy "OMG I'm not alone!" squeal burbling up from deep inside), because I'd like to expound for a bit about why I've done this, and I know the title is a bit of a giveaway, but no, it's not because I'm some cranky old man who can't stand that new-fangled ding-dang technology. Well, I mean I am old and cranky, but what I mean is I work at a company that heavily uses AI to augment real, serious professionals' capabilities. I'm not an anti-AI absolutist or anything.
LLMs will never be as capable as we have been told they will be. All of ginned-up demos that we've been shown around LLM-based tools being given a three-sentence description of a feature and building it and writing the tests and reviewing itself and deploying it, all automatically? They don't work now, and they never will. LLMs can never "know" or "understand" anything. They seem to have interesting emergent properties, but every new announcement about those capabilities quickly falls to pieces in the Hacker News comments, as curious devs give it a try and find that it actually doesn't work quite that well, at least not under all circumstances, and critically, not to achieve the overblown promises we're being sold around feature-scale AI code generation. Hell, this just happened again with the release of OpenAI's o1 series of models, models that we are told "think" but in actuality do nothing of the sort. They mostly just go "hmm" and "ah!" a lot, and then over-charge you via a new secret token mechanism to return dangerously close but not-quite-right code just frequently enough to cause serious problems down the line.
It would appear that we're hitting the upper limits of practical model sizes, too. Larger models have been the source of new and interesting improvements of these models, the "secret sauce" of the AI boom, as it were. And larger models can be trained, but the cost to train them (and retrain them when necessary), plus the cost of then running them looks to be prohibitive. We can see this in the latest ChatGPT models, the 4o series, which are smaller and less expensive than the original ChatGPT4. Even o1 doesn't appear to be a significantly larger model, it's likely just a pretty solid implementation of "chain of thought" prompting with existing models. ChatGPT5 it ain't, but maybe, when and if we get ChatGPT5, it'll will be another significant jump, like 2 to 3 was, or at least 3 to 4. But the basic architecture will almost certainly still be the same, the LLM architecture. We were in an "AI winter" before someone discovered that Big LLM Make AI Go Brr, and we'll be right back in another one once we hit the practical maximum sizes of LLMs, if we haven't already.
