Agentic Engineering Lesson #2: Experiments

I'm currently in the midst of introducing agentic engineering to a team that's been working together and doing things what is quickly becoming "the old fashioned way" for over 4 years. As part of that process we've been putting together our LLM do's and don'ts.

This is the second article in the series. So go read the first if you haven't already.

The second item on our list is when you're allowed to break the first one - experiments.

I've been working on new features for several projects recently and like most tasks I find myself presented with, they didn't come with detailed specifications. I knew what the customer was trying to achieve, but the exact way of achieving that both in terms of the UI and the underlying code was up to me.

Most developers when they find themselves in this situation are working to a deadline or at least an expected amount of time for which they should be spending on a particular task. The customer has most likely been given some sort of assurance that the job is in progress and an indication of when it might be delivered.

This means that coming up with a solution that fits the brief requires some quick thinking, experience having built enough things to know what won't work, and most likely going with the first version you build because there's just not the capacity to iterate on it over and over.

Welcome to small software teams.

But with an LLM at hand, this no longer needs to be the case. You can now task the LLM to help you come up with a solution. To challenge you on all your technical decisions and UI flow. It's not that the LLM is necessarily an expert in everything, but simply that it's good at coming up with questions. Even simple questions are a great way to make sure you've got it all covered, and I find that more often than not there will be a question that I had not thought of. The superpowers brainstorm skill works well here, or Matt Pocock's grill-me skill.

Next you can take advantage of that cheap code generation ability and just have the LLM take a shot at it. Take the answers from the previous step and just go for it. Let's see what it comes up with.

Six months ago you'd probably get something half-baked that barely works. With a frontier model at hand today, you can probably just send the result for peer review and move on (but you shouldn't). But that's not the point here, the point is to look at how it achieved it and look at what the resulting UI is. How does it feel? Does it feel easy to use? Does it solve the problem? Could it be better?

And if it could be better, then just do it again. Take that experiment, put it in the bin (or another branch) and have another crack it. Answer the questions differently, think about the context you're providing - did the LLM misunderstand something because you didn't provide it - and run the experiment again. And again.

Software mockups are so often pointless. Wireframing pages, making them all link together so it pretends to go to the right place when you click on a button - it's slow and tedious.

But an LLM lets you experiment with a properly working UI, using your data, in your codebase. It can give you something you can show to stakeholders, it can give you four completely working, totally different approaches to show to stakeholders. And then it can create a fifth version creating the best bits of those.

It's worth really thinking about what happens when the cost of producing each version goes from a week or two of work to an afternoon of work and how much better that can make your product.

And if you do still want to write the final version by hand, or at least a good chunk of it, then that's fine. You'll have learned so much from the experiments that you'll be in a much stronger position to do so.

But I think you'll probably find that tapping all those keys is time best spent elsewhere. So stay in the loop, and work on that final version with your clanker friend.