Elliot Morris

These magic thinking boxes. They understand things, right? They sure seem to. They grasp things, they hold concepts in their robotic mouths and can taste the gooey center?

I doubt it, but the jury’s still out on whether humans are really doing any real understanding anyway, whatever in the metaphysical heck that means. I wonder why we seem to care so much? Endless tedious debates on whether the thinking machines really get why DRY is best practice except for when it’s an anti-pattern, almost never with any interesting philosophy of mind to go along with it, much to my chagrin.

I sure don’t care if LLMs understand things, and I also don’t really care if my colleagues understand the things they are working on. I do however tend to insist on understanding as a practical necessity both for myself and anyone inside my sphere of influence. Why is that?

If you’ll allow me a strained metaphor, let us think about restaurants. Restaurants have deliverables, it’s food. They make that food in a production environment called a kitchen. Does it matter how they make it? You might say no, so long as the food is tasty, but I bet you’d be upset if you got food poisoning because the kitchen was filthy. You might even demand that the chef be punished, or fired! They should certainly apologize right, for running such a disgustingly below standard kitchen?

Alright, I see what you’re getting at here Elliot, the dish is the product and the bacteria is the intangibles … boring.

No, well, yes, but not entirely. As you will have deduced from the title of this article, this is about responsibility. Responsibility is the actual thing we look for out of colleagues and collaborators. It’s why any decent chef keeps a clean kitchen, and why we get so frustrated when our anthropomorphized LLMs contradict themselves or mislead.

If my colleague, in their wisdom, chooses to run their projects in such a way that they do not understand what they are doing, I am completely fine with that on the condition that they act genuinely responsible for the outcomes of that project. If a breakage occurs, or we discover they have built the wrong thing, they take responsibility. Not just in words, but in actions. This might mean working weekends to fix their fuckup, this might mean eating a massive slice of humble pie and explaining to our customers why we must delay, it at least means apologizing. It’s not going to be pleasant.

Responsibility also comes with another facet, self-reflection. Are you actually being responsible if you continue to make the same sort of mistake time after time? No, I expect you to be capable of introspection and self improvement. This means that you need to find a way to make sure you don’t keep letting others down. How would I do that in this situation? Let me think.

You know, I think I would probably come to the conclusion that, seeing as taking responsibility in tough times is painful, I should try as best I can to avoid having to do that. Perhaps I should make a real effort to understand what I am doing, such that I can be more certain that I can actually deliver what I promise, and more importantly, that I am capable of rising to the chaos of unexpected failures when I must.

Do I need to have understanding in order to deliver software behaviors? No. However, much like every high performing kitchen in the world takes reasonable, continual cleanliness as an obvious requirement for high quality cuisine, I also take reasonable, continual understanding as an obvious requirement for high performing technical products. A professional standard, if you will.

This is why I can say engineers don’t formally require understanding, but I am damn suspicious when they don’t have it.

LLMs can’t be responsible. Anyone who has used one firsthand knows that they can’t actually apologize, but putting both metaphysical and legal questions aside, there’s still the matter of mechanics.

Responsibility requires both preemption and reaction. Sure, you could invent the “Officer Clancy Responsibility Loop” or some other nonsense, but that would be a pale imitation of true responsibility, capturing at best only actions of immediate remediation. You may try to put faith in some sort of mutable, permanent store of knowledge such that this LLM can self-improve and grow in order to teach itself not to make the same mistakes again and thus become more responsible. If you find yourself agreeing with this line of reasoning, then I must conclude that you are a believer in the imminent arrival of true AGI. I also must conclude that you intend to hold beings capable of true responsibility captive in order to perform menial tasks for you without compensation or freedom to refuse. Gross.

* I’d like to make a comparison here about how street-food is still tasty despite not being quite as rigorously clean as high end restaurants, just so you know that it’s fine to vibe non-professional projects if you want. However, I think that might be unfair to street-food vendors, I’m sure they’re honorable people who don’t deserve that comparison.

* You’ll note that responsibility as the deliverable also helpfully avoids the uniquely difficult task of needing to exhaustively define what specific product outcomes we are after. It allows us to live in the higher bandwidth dynamic space of human conception, and sidestep a lot of process that only works if we laboriously mine all that tricky human stuff and convert it into static language. That sort of thing isn’t too bad if taken in moderation, but be sure not to get addicted.