Elliot Morris

I have spent much of my career begging people to build more prototypes. Please, please just try to make a small version of the thing before committing months of time and money to it.

Prototypes were both an excellent educational tool to try an instill engineering sense into less experienced engineers, as well as a useful artifact for corporate decision making, but less so today.

To my eye, the production of a prototype used to necessarily manifest the following benefits:

• Allowed the engineer to communicate what the fuck they even mean to other people in the organization.
• Provided the organization a representative sample of the sorts of problems and difficulties it would encounter if it took a certain technical direction.
• Demonstrated to the organization that an engineer has a legitimate command of the domain, and can therefore be trusted to speak to it in the future. Conversely, granted the engineer a persuasive tool, allowing them to make projects happen, and take agency over their own career by convincing stakeholders that they were the right person for a job.
• Forced the engineer to engage with reality before embarrassing themselves, revealing unknown unknowns, necessitating essential research, and building an engineering instinct.

Some of these were good for the organization, and some were good for the engineer. It used to be very win/win, as long as you were willing to put in the work. It’s rare to find a practice so easily prescriptible that causes such universally good outcomes. This was a “wash your hands before surgery” levels of cost/benefit. An overwhelming, across the board win when it mattered with almost no downside when it didn’t.

Let’s go through these one by one. If you’re presented a prototype, you now have to assume it is an LLM prototype, but this sort of prototype still might not be entirely useless, some of these benefits may still apply.

Communicating what you mean

This is the quality that survives the transition to AI prototypes the best. If you purely want to communicate, then I would encourage you to consider an AI prototype, maybe even more than one to explore options. It is the same class of thing as a sketch on the back of a napkin or a whiteboard drawing. Just something to get the idea across. Arguably, an AI prototype is a better way of doing this, because you can encode interaction without having to write awkward little “when you push the button, X happens!” notes.

However, beware! It is all too easy to let the LLM guide you away from what you actually meant, and towards the almost-but-not-quite-the-same LLM-average. It is especially easy to not realize that the LLM has made a decision for you, and accidentally communicate something contrary, or more commonly merely confounding, to what you wanted to be saying. The skill in communicative LLM prototypes is in preventing them from saying too much, not getting them to say more.

I suspect over time, the close to the mean, sloppy feeling of even purely communicative prototypes will begin to decay the background legitimacy of prototypes in general.

Seeing the problems ahead of time

This benefit is greatly reduced in LLM prototypes, at least in terms of inspecting a specific decision. Whilst you can inspect what your LLM has done, and even ask it for a report on the problems it encountered and solved, you will not have a sense for how truly significant those problems are in your context. Unless already highly experienced, you will also not be able to effectively project those problems forwards onto uncertain requirements shifts and twists of circumstances common to your organization.

Remember that organizations are unique, and prototypes are more than just a test of technology, they also used to serve as a limited test of an organizations social and management structure. Perhaps an engineer simply could not complete the prototype as they kept getting pulled off by fires, or maybe everyone in the organization loathes their guts so much that they do as little as they possibly can to help them overcome the technical or political problems necessary to deliver the prototype. If the employee you’re trusting with your technical strategy can’t figure out how to get a prototype server provisioned without involving 3 layers of management, for better or worse that is something an org wants to know before assigning them to a long term back-end project.

Before we get too doom and gloom, there is a potential upside here. You can now shallowly inspect the immediate technical problems across a broader set of potential options all at once. This ain’t nothing to sneeze at, although beware of all the above.

Demonstrating competence and making projects happen

Knowing who is competent and who is not is rather important for an organization. It’s one of those back-room lubricants that is essential for actually getting shit done. Prototype projects used to serve as a reasonably comprehensible marker for any given engineers competence in a certain domain. It really is as simple as “Oh, they managed to make this? Then they can’t be a total idiot.”

This benefit no longer exists at all, prototypes will not be a meaningful competence signal for very long, and engineers will have to find another way of differentiating themselves. This is a shame, as this was a tool more technically focused engineers with initiative could use to demonstrate their own value whilst applying some influence to organizational direction,

People are still using prototypes to make projects happen, but they are less powerful as a tool than they once were. My take is that if you are still seeing good results from this technique, you are borrowing from the historical legitimacy of prototyping, and it won’t last forever.

Forcing engagement with reality

Like most of this list, you can still make yourself engage with reality if you already have the skill to do so, but it’s no longer a guarantee prototypes provide. Whilst an LLM prototype is still, technically, an object that must interact with reality to function, full on LLM prototyping isolates the engineer from reality quite strongly, especially now that modern harnesses are hiding much of the chain of thought stuff from the implementer and encouraging auto mode.

Engineers who defer engagement with reality tend to be unable to follow up on one of the prime responsibilities of being an engineer, which is to be mostly correct about most things most of the time.

The unfortunate truth here is that some inexperienced engineers are now embarrassing themselves by making unqualified assertions about the future, based on prototype findings which cease to be true once they interact with the organizational reality.¹

Even in high directness organizations, this sort of distrust is almost never expressed publicly, or even understood in such black and white terms by the people who hold it. You will likely not be told if you are doing this, unless you have an unusually proactive manager or a very good friend in the org.

---

You’re supposed to have advice at the end. I think I have some.²

Prototyping has worked so well for so long precisely because it was genuinely difficult to build a prototype without also manifesting most of these benefits. Crucially, everyone else also knew that to be true, so trust came for free. The prototype itself was never the point, much like how code or pull requests have never been the point, they are practices and conventions that are tightly coupled to the inexpressible thing we are all actually after.³

Prototypes also used to take time, and from my observation, experienced engineers became more skillful at making them smaller. In fact, the more experienced an engineer was, the rougher and more threadbare their prototypes might look. Why is that? I have some theories.

• Experienced engineers have less time to waste and generally less to prove. They are not looking to impress anyone with shiny prototypes.
• It is more difficult to hone in on a small, sharp thesis than it is a large, sprawling one.

I think these are both true, and both will come to serve as markers of legitimacy in the coming years. Really sharp prototypes test one thing and one thing only. They are experiments, it’s science. When you think about what it means to control an experiment, it becomes clear big fancy LLM energy can often do nothing but introduce confounding variables.

It saddens me when I wonder how folks are to learn how to produce a sharp thesis, or how to build experimental prototypes that really test them. I doubt I would have ever really engaged with the problem if not for a need to get more things done faster. I simply had too much to do, and was forced to figure out how to avoid spending weeks writing boilerplate, which led me to really start evaluating what was important in a prototype. It is harder to learn how to do things when you are not being coerced in some way.

My best guess is that the preciseness of your thesis will become at least part of a new rubric. Why does this prototype exist, what specifically does it demonstrate about the future, and why should I care? A lot of prototypes I see today seem to have a rather broad thesis, something along the lines of “Can I make this thing I want to happen on my computer happen.” This isn’t a useful question to an experienced engineer, and in the age of LLMs, it’s not really a useful question to anyone.

If you build LLM prototypes, seek to use them as a sharper tool to prove out a specific thesis, understanding that they do not bring the same benefits as the one-size-fits-all, prototype-as-a-plan strategy of the past.

If you are just now dipping your feet into prototyping as a project planning tool, or self-directed R&D style prototyping, please do not get offended if people seem underwhelmed. I know a lot of people are still recommending prototyping as a sort of panacea, as a lot of goodwill for the technique has built up over time, but that doesn’t mean those same people won’t already be intuitively drifting towards a more skeptical outlook.

Remember that prototypes in of themselves have never had any value, and try to figure out what interesting experimental question you are answering, and prove specifically that.

If you are in a more managerial position, I encourage you to disavow yourself sooner rather than later of any reverence you held for prototyping. Insist on seeing one, sure, but do not hold yourself so tightly to this specific indicator, things have changed.

1. For the love of god, do not give a concrete end-to-end delivery estimates based on nothing more than a personal LLM prototype. Nothing will kill trust quicker than the sorts of overruns you may end up holding the bag for.

2. Not before I express my distaste for how willing we are to throw away reasonably well established social technologies in favour of chasing any benefits of material technology without even thinking about trade-offs. Oh well, it’s nothing new, not like anyone goes to church anymore.

3. How unbearable it must be to be a physicalist these days.