Designing HTTP APIs for slow operations

We got used to having everything on the Web, and so many backend operations became HTTP APIs. That’s also the case in the areas of AI and optimization, because it gives an easy-to-understand interface for clients that can be used without having to learn new technologies. There’s a problem, however: Some of these operations are very slow, and at times systems become a brittle mess of regular timeout hits, latency optimizations, and support issues.

In this post I’ll discuss why the synchronous HTTP approach doesn’t work for slow operations and what common patterns are available to design for a good API experience.

The limits of synchronous HTTP

The HTTP request-response model assumes an answer arrives quickly — in the order of seconds. Typical timeouts in client and proxy components are 30 or 60s, and operations like OCR pipelines, route optimization, report generation, or LLM inference can easily exceed that. Of course, timeouts can be extended, but that only reveals a deeper set of problems:

• Timeouts across the proxy chain. Connections often stretch across multiple steps (load balancer, reverse proxy, API gateway, etc.), each with its own timeout configuration. A slow API has to go through a painful learning phase where you discover all these moving parts and how they behave when something takes 15 minutes. An especially uncomfortable example is an ingress controller whose logs are not accessible to the application team: timeouts don’t surface in the backend at all, only on the client side, as an HTTP 504 Gateway Timeout.

• Resource consumption from holding connections open. With 10 requests per second and a 10-minute average processing time, a server would need to hold 6,000 connections open at any given time. Even if the service doesn’t need a lot of resources for the sessions internally, each open connection is an open TCP socket, and each socket consumes a file descriptor. The standard Linux limit is as low as 1,024 file descriptors per process — something you tend to discover under production load.

• Peak loads must be processed instantly. It is not possible to distribute request bursts over a period when the client expects an immediate response. Hence, sufficient resources for the peak load must always be available.

• Async is needed on the client side. A user interface cannot be frozen for five minutes after submitting something. So if the client is a frontend, it has to wrap the backend call in an asynchronous function to stay responsive. But without any indications about the status, it can not do more than showing a spinning wheel.

• Latency pressure leads the architecture. In the backend, tight timeout budgets push the developers to use smaller models, skip retries, or parallelize heavily. Other targets such as the output quality may suffer as a result.

Architecture styles for slow operations

A classical synchronous API suggests that the operation internally is also handled in a tight sequence of function calls. With an asynchronous design, we can decouple job submission from execution. Processing can happen in the background, in a robust setup which allows a lot of optimizations: Dedicated compute, peak load buffering, retries, and easier remote calls to name a few.

There are several implementation options with certain trade-offs:

• Message queues (e.g. RabbitMQ, Celery, Redis Streams) — lightweight and with good support for tracking job status. A solid default choice.
• Kubernetes Jobs — option for heavy or isolated workloads, since each job runs in its own pod. No extra components needed, but more effort to make robust.
• Kafka — especially useful for high-throughput pipelines, or where multiple consumers react to the same work or where the processing spans several steps.

In all these options, we create the worker pool design pattern:

API patterns: bridging sync clients and async backends

So, with all these arguments for asynchronous processing, what does that mean for the HTTP API? It stays the dividing layer between the client and the async backend. It hides the complexity and serves as an adapter with a familiar interface to whatever backend architecture is chosen.

The following sections show five concrete solutions for how this boundary can be designed.

Job submission and polling

This approach is the most straightforward for an HTTP API, because it resembles the structure of a typical REST API:

• The client calls POST /jobs to create a job resource and it gets back a job_id immediately (HTTP 202 Accepted)
• The client repeatedly calls GET /jobs/{id} until status is done

The advantage of this pattern is the simplicity of implementing it with any HTTP client. The disadvantage is that polling is in general wasteful. Either the client has to poll very frequently, which wastes resources, or it has to stretch the polling interval, which leads to a high perceived latency. One attempt to solve this dilemma is long polling, where the server delays the response until the job is done or a timeout is reached.

Overall, the approach is a good default choice and is used widely. Some examples are:

• The bulk import of the Pinecone Vector Database
• Document processing in the Amazon Textract text extraction service
• OpenAI’s batch API

Streaming over Server-sent events

A very different design is to use Server-sent events (SSE). SSE is a push protocol that allows the server to stream data back to the client over a single, long-lived HTTP connection.

The flow is as follows:

1. The client connects to the server via an HTTP endpoint and sets the Accept: text/event-stream header
2. The server keeps the connection open to send status updates and results back to the client:

data: {"status": "processing"}\n\n
data: {"status": "processing"}\n\n
data: {"status": "done", "result": 42}\n\n
...

This approach uses the so-called EventSource API which was originally designed for real-time updates in web browsers. This is also where it is supported best, with automatic reconnection to continue the stream at the same position.

SSE is well supported in popular web frameworks like FastAPI and it is used for slow tasks where the client wants to see progress updates:

• The Model Context Protocol uses SSE for delivering the responses to remote procedure calls.
• LLM APIs such as OpenAI and Gemini stream generated tokens over SSE, so the user sees output appear progressively instead of waiting for the full response.

From the client side, SSE is best supported in web browsers, although many programming languages have libraries to consume SSE streams. But the LLM API examples already show how the client-support deficiencies are typically tackled: by providing a client library which turns the SSE stream into a convenient iterator of update events.

Callbacks

A similar approach to job submission and polling is the callback pattern. In this pattern, the client provides a callback URL when submitting the job. The server then calls that URL when the job is done (HTTP POST with result).

This requires no open connection and scales well. Especially in a server-to-server setup, it is easy to implement. But it requires the client to expose an accessible HTTP endpoint, which might not always be easy depending on the client application type or network design. Alternatively, other response protocols can be used, such as messages to a pub/sub system.

The Replicate inference API is a good example: when creating a prediction you pass a webhook URL that gets called with the result once the model run finishes.

Inversion of control with webhooks

A very different approach is commonly used in CI/CD systems. The responsibility of starting jobs and delivering the results is handed over to the processing system. It fetches the necessary data and submits results to the target system on completion. In the API, a general callback endpoint is provided, which merely informs about the need to check for possible jobs.

Concrete example with GitHub and Jenkins:

• On every commit, the GitHub server calls a webhook in the Jenkins server
• Jenkins fetches the commit data and triggers a build
• When the build is done, Jenkins submits the result to the GitHub server as a commit status

This is a form of Inversion of Control. It also requires giving the processing system direct access to the data and the knowledge needed to process it (e.g. via configuration). It is most applicable to background processing scenarios where the client doesn’t need quick results.

Alternative protocols

Although HTTP is very common, other protocols may be better alternatives for asynchronous processing. The client can directly speak to a queue/broker (e.g. publish to Kafka), removing the HTTP API as dividing layer.

This reduces the translation overhead and can simplify the architecture and development. On the other hand it requires the teams on both sides of the interface to speak the same technology and may couple the interface to a certain infrastructure.

Choosing the right pattern

The patterns above are not mutually exclusive, and a single API can offer more than one. A good example is the OpenAI Responses API, which features a background mode in two flavours: polling and streaming. The client picks whichever fits its situation.

The main question is: what kind of client am I serving? The backend processing is independent from this, but the API interface can make a big difference in terms of ease of use and performance.

The following table gives an overview of the different patterns:

Summary

Synchronous HTTP reaches its limits in many ways when processing can take longer than 10-30s. Asynchronous decoupling becomes necessary.

The choice of the backend architecture (with queues, jobs, databases, etc.) is independent of the choice of the API design. What the solutions have in common is that they decouple submission from execution and let the work happen in the background (worker pool pattern).

The best API design depends on the specific use case and the client technologies and architectures. Popular APIs offer multiple patterns alongside each other for different usage scenarios. The simple job submission and polling approach is a common default choice which is often sufficient.