Let's Talk About AI Workflows

In this post, I want to provide an understanding of AI workflows, why you would use them, and an example of how they can be used. AI workflows provide a modular approach to building with LLMs. As always, let me know if you have any feedback on these posts. I really appreciate it! If you want to get notified about future posts, I’m starting a newsletter (subscription link at the bottom of the page).

What is a Workflow?

Anthropic, who more or less coined the term, defines AI workflows as “systems where LLMs and tools are orchestrated through predefined code paths”. AI workflows are great for breaking down complex problems into multiple LLM calls, where each LLM call can be considered a unit function with a specific task. These LLM calls can either be parallelized or serialized so that the output of one LLM call can be used as part of the input of another LLM call. To be very clear, workflows are not agents. Agents are "systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks." More on this later.

Why Use a Workflow?

In general, LLMs are much better at "solving" smaller problems than bigger ones. They are more likely to return the desired result and less likely to hallucinate, in part because the prompt and context can be tailored to the specific task. Think about it. It's similar to humans. We make fewer mistakes when we are tasked with a simple task. Similarly, it's much easier to structure a simple prompt that is meant to do a specific thing. So, if you have a complex problem, it's better to break that problem up into its components and have LLMs handle each component.

My Experience with Workflows

A complex problem can be anything from a chat bot to document extraction. With Kalen, I built a workflow to make the productivity coach more effective. Structurally, the coach is broken down into three LLM calls. There is one LLM call to understand the user's intent. Are they feeling stuck, are they talking about their wins, etc.? There is another LLM call to understand what updates the user wants to make to their tasks or habits. It breaks down natural text into a set of operations. Finally, it uses the intent and the updates (plus other information) to generate a response. The separate components allow me to iterate on different aspects of the coach. Is the coach not understanding the intent of the user's message, then update the intent LLM. Is the coach too verbose in how it responds, update the response LLM. To be clear, this is far from perfect, it still feels like whack-a-mole at times, and I'm curious if a more agentic approach will be better.

Testing AI Workflows

One of the key benefits of workflows is testing. Each LLM call is a unit that can be independently tested against a set of scenarios. If there are integration tests that test the whole flow, then the units can be swapped out easily. Speaking of testing, I have a testing AI workflow as well. When I run tests, I pass the output of the tests to an LLM that acts as a judge and grades the response on a rubric. This is a common workflow and an important thing to emphasize. An LLM workflow doesn't have to be siloed to product or some specific use case. It can be extended to any domain.

The Trade-offs

The big downside of AI workflows is cost in both time and money. Each LLM call has a cost and latency, and many times you have to serially stitch LLM calls.

Wrapping Up

So when should you use workflows? Use them when you have a complex problem with clear sub-tasks that you want to test and iterate on independently. Skip them when your task is simple enough for a single prompt or when latency is critical.

The key thing to remember: workflows trade simplicity for control. You're adding complexity, but you get to debug and improve each piece on its own.

Agents are the next level up from workflows. They can choose their own paths and adapt on the fly. If workflows feel too rigid for your use case, agents might be what you're looking for. I hope to dig into agents in a later post.