An agent analyses a request and calls a tool. The response? Takes its time. Once it arrives, the next system is called and the data is processed. Only then is an action executed. Each of these steps, taken on its own, is unremarkable. Two seconds here, three seconds there.
For a single step, the delay is barely noticeable. In a longer agent chain, however, these seconds add up quickly. A request that a person would have answered in ten seconds becomes an operation that takes a minute. Not because the agent is wrong, but because it passes through too many stations on its way to the right answer.
Latency thus becomes an architecture question in its own right. A productive AI agent must not only deliver correct answers. It must deliver them within a time that makes sense for the process. Because in the enterprise, speed is not just a matter of user experience. It can be part of process quality.
Where the Time in an Agent Chain Is Actually Lost
Anyone who wants to understand why an agent is slow has to break the chain into its parts. A typical run consists of a sequence of model calls and tool calls that depend on each other. The model decides which tool to call. The tool returns a result. The model interprets the result and decides on the next step.
| Step | Typical duration | What determines it |
|---|---|---|
| Model call for planning | One to several seconds | Model size, length of context, length of output |
| Tool call | Milliseconds to seconds | Target system, network, authentication, data volume |
| Model call for interpretation | One to several seconds | Another call, with a larger context than before |
| Waiting for an approval | Minutes to hours | Process and availability of people, not technology |
The crucial point is that this pattern repeats for every step. An agent that needs eight tool calls goes through at least nine model calls. If each cycle of model and tool takes three to five seconds, half a minute has passed after eight steps, without any single component having been slow.
On top of that comes an effect that is often overlooked: with every step, the context grows. Every tool result becomes part of the conversation the model has to process on the next call. The tenth model call is therefore not just another call, but a slower and more expensive one than the first.
Why Latency Is Different for Agents Than for Classic Software
In a classic application, the response time for a given use case is largely predictable. A request follows a defined path. Anyone who knows the path can measure latency, find bottlenecks, and optimise. The result is a response time that, under the same load, stays within a narrow band.
With an agent, the number of steps is not fixed. The same task may be solved in four steps one day and twelve the next, because the model chooses a different route, a tool returns an unexpected result, or an intermediate step has to be repeated. Latency therefore becomes not only higher but, above all, more variable. For a process, a variable response time is often harder to deal with than a slow but reliable one.
In addition, an agent can decide on detours that are technically correct but unnecessary. It queries the same data twice. It fetches a complete document when a single field would have sufficed. It calls a model to decide something a simple rule could have decided. Each of these detours is harmless in isolation. Together, they are the reason an intelligent agent can still be too slow.
Four Questions the Architecture Has to Answer
Which information has to be processed synchronously, which can be loaded in parallel, which steps can be prepared in advance, and where an additional model call is necessary at all: none of these are questions for the model. They are questions for the architecture the model is embedded in.
- What really has to run synchronously? Only steps whose result is needed for the next step have to wait for each other. Much of what agents do one after another today does not depend on each other.
- What can be loaded in parallel? If an agent needs customer data, contract status, and open tickets, there is no reason to run these three queries sequentially. Three parallel calls take as long as the slowest one, not as long as all of them combined.
- What can be prepared before the agent starts? A lot is already known before the first model call takes place: who is making the request, which customer is affected, which data is typically needed. This information can be loaded in advance and handed to the agent, instead of letting it gather it step by step.
- Where is a model call necessary at all? Not every decision in a chain needs a language model. A validation, a mapping based on fixed rules, or a format conversion is faster, cheaper, and more predictable in code.
These four questions should be answered before going into production. They shape the structure of the agent far more than the choice of model does.
Synchronous or Asynchronous: A Question About the Process
Not every agent task has to be answered within seconds. An assistant that a service employee uses during a customer call has different requirements than an agent that classifies incoming documents overnight. Before latency is optimised, it should therefore be clear which interaction pattern the agent operates in.
| Pattern | Example | Latency requirement | What helps |
|---|---|---|---|
| Interactive | Assistant during a customer call, chat in daily work | A few seconds per response | Streaming, intermediate results, preloaded context |
| Semi-synchronous | Quote preparation, research on a case | Minutes, with visible progress | Status updates, notification on completion |
| Asynchronous | Overnight processing, bulk verification of receipts | Hours are acceptable | Throughput and parallelisation instead of per-item latency |
The right question is not how fast the agent can be. It is: how fast does the process need it to be?
A common mistake is to design every agent as an interactive chat, even when the underlying process does not call for it. Conversely, it is problematic to embed an agent in an interactive process when its chain structurally cannot be brought under ten seconds. In both cases, the problem is not the model but the mismatch between agent and process.
Deterministic Steps Do Not Belong in the Model
A substantial share of the latency in many agents comes from model calls that are not necessary at all. The model is asked to extract a customer number from an email, even though it appears in the subject line in a fixed format. It is asked to check whether a date lies in the future. It is asked to decide which department a case goes to, even though the rule for that has been documented for years.
Each of these calls costs seconds and introduces variance into a step that needs none. The rule is simple: whatever can be written as code should run as code. The model decides where judgement is required. Everything else is handled by the integration and tool layer.
This has a second advantage. Deterministic steps are testable, measurable, and identical on every run. Taking them out of the model not only shortens the chain but also makes it more predictable. It is the same separation that is decisive for error handling in agentic architectures: the agent acts within a frame that the infrastructure defines.
Model Choice, Context Size, and the Cost per Step
Not every step needs the largest available model. A classification, a routing decision, or a simple summary can be handled by a smaller model in a fraction of the time. The large model is reserved for the steps where actual reasoning is required. An architecture that picks the appropriate model per step is faster and cheaper than one that uses the same model everywhere.
The same applies to context. What is sent along with every call directly determines the response time. An agent that carries a complete contract document in its context at every step is slower than one that receives only the relevant sections. Targeted loading of information, summarising intermediate results, and deliberately discarding context that is no longer needed are therefore not finishing touches but part of the design. How much an agent should keep in memory at all is a closely related question.
The interfaces themselves matter too. An API designed for human use in a user interface often returns too much or too little for an agent. If the agent needs three calls to assemble a piece of information that a single well-suited endpoint could deliver, that is a latency problem that cannot be solved inside the agent. The requirements agents place on APIs are therefore also a question of speed.
Measure Latency Before Optimising It
Optimisation without measurement leads in the wrong direction. In many cases, it turns out that it is not the model that consumes the time, but a single target system that needs several seconds for every request. Or that an agent needs only five steps on average, but twenty in ten percent of cases. Such patterns are only visible if the chain is instrumented.
| Metric | Why it matters |
|---|---|
| Duration per step, split by model and tool | Shows whether time is lost in the model, in the target systems, or in waiting |
| Number of steps per task | Makes detours and unnecessary repetitions visible |
| Distribution of total duration, not just the average | The average hides the outliers that disrupt the process |
| Context size per call | Explains why later steps are slower than early ones |
| Share of operations above the target value | Translates latency into a statement about process quality |
This data is the same data needed for observability and cost control. Anyone who records the duration of every step also knows the number of model calls and thus the cost per task. In agentic systems, latency, cost, and traceability are so closely linked that it makes sense to measure them together.
Speed as Part of Process Quality
In the enterprise, latency is rarely just a matter of convenience. An agent that needs ninety seconds to prepare a piece of information in customer service is not usable while the customer is waiting on the phone, even if every piece of information is correct. An agent that pre-qualifies leads overnight, on the other hand, can take a minute per lead without anyone noticing. Response time determines where in the process an agent can sensibly be deployed at all.
Latency also has technical consequences beyond perception. Calling systems have timeouts. If an agent regularly takes longer than the upstream system is willing to wait, results are lost or requested again. A latency problem thus turns into an error-handling problem, with all the questions about retries and duplicate execution that come with it.
And finally, speed influences whether people work with an agent at all. A tool that is reliably slower than the previous manual route gets bypassed, no matter how good its results are. Acceptance does not come from correctness alone, but from correctness within a time that fits daily work. Anyone introducing AI agents in the enterprise should therefore treat response time as a requirement from the start, not as an optimisation to be done afterwards.
Conclusion
The discussion about AI agents usually revolves around capabilities: which tasks can the agent solve, how good are its answers, how autonomously does it work? The question of how long it takes often only comes up when the first pilot is about to go into production and it turns out that a minute per operation is not viable in the real process.
For agents, latency is not a detail that can be fixed at the end with a faster model. It arises from the structure of the chain: from the number of steps, the order of calls, the size of the context, and the decision of where a model is consulted at all. These decisions are made at design time.
A productive agent must therefore not only deliver correct answers. It must deliver them within a time the process can tolerate. In the enterprise, speed is not just a question of user experience. It is part of the quality of the process the agent works in.