ALTO: An Efficient Network Orchestrator for Compound AI Systems
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: ACM Digital Library.
A compound AI system chains generative language models (LMs) with other components: retrieval-augmented generation, chatbot verification, agents, SQL generation. The paper's running example, inspired by FacTool, fact-checks a chatbot in six stages: answer a question, extract claims from the answer, generate search queries for each claim, retrieve documents with BM25, and rerank them with ColBERT. An LM produces its output one token at a time, but the stages after it consume different units: a whole claim, a whole search query, or single query tokens. A pipeline that waits for each stage's full output before handing it on leaves downstream stages idle while upstream LMs are still generating.
The key observation is that streaming partial outputs between distributed stages reduces latency, because downstream stages start on the first claim or query token before the upstream LM finishes, and increases throughput, because dynamic scheduling keeps stages from falling idle when there is work to do. Streaming at scale, with several instances of each stage, raises two problems. Correctness: some stages are stateful. The BM25 stage sums per-token relevance scores across a search query, so every token of that query must reach the same BM25 instance. Load balancing: each prompt fans out into a different, unpredictable number of outputs, and LM engines run faster when requests sharing a prompt prefix go to the same instance, which conflicts with spreading load freely. The paper argues that existing streaming systems (Kafka, Spark Streaming, Naiad, Ray) do not easily express the hierarchical nesting of streams and the ancestry of partial outputs that this routing needs.
ALTO (Automatic Language Token Orchestrator) is a serving system that places queues between pipeline stages and a central runtime that forwards data among stage instances. Its answer to correctness is aggregation-aware routing: every data item carries a header listing the IDs it must be aggregated by (for a query token sent to BM25, its object, claim and query IDs). When an item has such a constraint, the scheduler hashes it modulo the number of destination instances, so items with the same constraint land on the same instance; items without one are spread round robin. For load balancing, the paper motivates distributed prompt-aware scheduling and sketches a design, but the current scheduler does not implement it.
The runtime is about 7,000 lines of Rust; stages are written in Python and use vLLM or SGLang as LM engines. The evaluation runs the FacTool-inspired pipeline on one 8-GPU node with SQuAD questions as input. Both configurations run on ALTO and differ only in whether stages stream. Streaming partial outputs sustains up to 3× higher load at a latency target of 4 seconds per request and gives 1.8× lower P99 latency at low load. The prototype used for these numbers routes in-flight data by hashing the request ID at every stage; the constraint-aware scheduler of Section 4 is a design the paper describes but did not use for these results.
Key takeaways
- LMs emit output token by token, while downstream stages in a compound AI pipeline consume claims, queries or tokens. Forwarding each partial output as soon as it exists lets stages overlap.
- Streaming across many stage instances creates two problems: stateful stages need all related partial outputs at one instance (correctness), and prompts fan out unevenly (load balancing).
- Aggregation-aware routing: each item carries a header of the IDs it must be aggregated by; ALTO hashes that header to pick the destination instance, and spreads unconstrained items round robin.
- ALTO is a central Rust runtime plus queues over UNIX domain sockets between Python stages, with vLLM and SGLang integrations. Distributed prompt-aware scheduling is motivated and sketched, not implemented.
- On the FacTool-inspired pipeline (one 8-GPU node), streaming sustains up to 3× higher load at a 4-second latency target and 1.8× lower P99 latency at low load, compared with the same pipeline on ALTO without streaming.
Streaming partial outputs with aggregation-aware routing
The pipeline, why streaming helps, the two problems it creates, ALTO's interface and runtime, and the evaluation.
1 · Compound AI pipelines and partial outputs
A compound AI system combines one or more LMs with other components such as retrievers. An LM generates autoregressively, emitting one output token per iteration, so its output exists as a growing sequence of partial outputs long before it is complete. Downstream stages may consume those outputs at different granularities, or quanta: single tokens, sentences, claims or paragraphs. The paper's evaluation pipeline (its Figure 1) is modeled on FacTool, which fact-checks chatbot answers against retrieved evidence:
| Stage | What it does | What it passes on | Instances | GPU |
|---|---|---|---|---|
| 1 · Question answering | LM answers the user's question | a full response (paragraphs) | 2 | yes |
| 2 · Claim extraction | LM extracts factual claims from the response | a stream of claims | 2 | yes |
| 3 · Search query generation | LM writes search queries for each claim | search queries, and their tokens one by one | 3 | yes |
| 4 · BM25 retrieval | Scores documents token by token and sums across the query's tokens | candidate documents | 4 | no |
| 5 · ColBERT query encoder | Encodes each search query | embedding tensors | 1 | yes |
| 6 · ColBERT reranker | Reranks the BM25 candidates | claims with evidence | 4 | no |
Two differences from FacTool: the final claim-appraisal stage is omitted (the paper says a later version will include it), and retrieval uses local BM25 and ColBERT deployments instead of Google search.
2 · Why streaming helps
- Lower latency within a request. Each claim from stage 2 can go to stage 3 as soon as it exists, instead of after the last claim; each search query from stage 3 can go on to stages 4 and 5 the same way. Work on one request overlaps across stages.
- Higher throughput across requests. Scheduling streams dynamically across stage instances keeps stages from falling idle while there is work to do.
- The paper's illustration. Its Figure 2 traces three requests, each with two claims and two queries per claim, through all six stages. Without streaming, every stage waits for its upstream stage to finish a request before starting on it; with streaming, the later stages start on a request's first outputs much earlier, and the trace ends sooner.
3 · Problem one: stateful stages need all related outputs in one place
Some stages aggregate across a stream. Stage 4 is the example: it must sum per-token relevance scores across all tokens of one search query, so every token of that query must reach the same BM25 instance. Other aggregation operators the paper lists are top-k, counters and filters. Stateless stages, by contrast, can fan their outputs out across any instance for more parallelism.
The Section 2 experiments enforce this with one global rule: hash the request ID to choose the instance at every stage. The paper calls that suboptimal, because stages that do not aggregate are forced to obey it too, which limits load balancing (its Figure 4). The better approach applies aggregation-aware routing only at stateful stages, which needs an interface that names those stages and their rules. A complete rule for the stage 3 to stage 4 edge must say that the quantum is a token, that aggregation happens at stage 4, and that the tokens are grouped by search query.
4 · Problem two: prompts fan out unevenly
Each LM stage runs its own prompt, and each prompt produces a different unit and number of outputs. The paper measured this from a 12-minute run at 15 requests per second (its Table 1):
| LM stage | Overall count | Per-output quantum | Outputs per input | Length per output (words) | Mean time per output |
|---|---|---|---|---|---|
| Question answering | 10,795 | response (paragraphs) | n/a | 62.5 ± 57.2 | 1,292.3 ms |
| Claim extraction | 10,795 | claim | 3.3 ± 1.8 | 9.8 ± 3.5 | 403.6 ms |
| Search query generation | 35,516 | search query | 2.5 ± 1.3 | 5.5 ± 3.1 | 326.6 ms |
| search query token | 5.5 ± 3.1 | n/a | 59.8 ms |
The paper's Figure 5 plots each prompt's latency as load rises, with each prompt saturating a single GPU at a different rate: the curves diverge sharply, with claim extraction saturating at the lowest load and search query generation sustaining the highest. This creates a tension:
- Static assignment wastes capacity. The Section 2 experiment fixes one prompt to each LM instance. If the split does not match the prompts' relative frequencies, some instances sit under-used while others are overloaded.
- Fully prompt-agnostic routing loses locality. LM engines such as vLLM and SGLang reuse key-value (KV) cache state for requests that share a prompt prefix, which raises throughput when many such requests reach the same instance. Engines already do this within a batch; across distributed instances it has to be arranged by the network-level scheduler.
- The goal is both. Balance load across all LM instances without a fixed assignment, and still keep requests for the same prompt together where possible. The paper calls this distributed prompt-aware scheduling and leaves it to future work (Section 8).
5 · ALTO's interface and routing rule
ALTO has two pieces: an inference interface for pipeline stages with queues between them, and a central runtime that forwards data among stage instances. Its three design goals are to aggregate partial state correctly for every stateful stage, to balance as much work as possible evenly across replicas within those constraints, and to keep the interface for constraints and prompt information lightweight. The model is microservices, with one change: instead of RPCs, stages hand the runtime aggregation constraints and prompt information that the scheduler uses.
- Pipelines are stages connected by queues. Each stage reads from its input queues, processes the data (running an LM, an aggregation or other computation) and writes to one or more output queues. Developers define each queue's data format with Protocol Buffers.
- Constraints travel in a header. Each item written to a queue carries an array of integer IDs. Any items that must be aggregated together carry the same array. The header grows with the hierarchy: a query is tagged with a query ID, and when a later stage generates a claim for it, the application adds a claim ID, so aggregation is correct at both the claim and the query level.
- The scheduler routes by that header. If an item has an aggregation constraint, the scheduler hashes it and takes the result modulo the number of destination instances, so items with the same constraint go to the same instance. Otherwise it picks the next instance round robin. The paper notes that consistent hashing could keep load even as instances come and go.
6 · Implementation
- Queues. Each queue wraps two reliable UNIX domain sockets: one from the source stage to the runtime (the output queue) and one from the runtime to the next stage (the input queue). The paper notes that wrapping network sockets instead would extend this to several machines. Each queue has a user-defined Protobuf type (for example Token or Claim); only applications deserialize, and the runtime forwards in-flight data without deserializing it.
- Runtime. It takes two configuration files: one lists the stages with their resource requirements and input and output queue names, the other how many instances to start for each stage. It adds a global request ID to each request's data headers, enforces aggregation constraints and schedules in-flight data.
- Stages. Stages are written in Python. The runtime passes each stage its queue names at startup, and the stage calls an ALTO library function to set up its sockets. Application-side queues use
asyncio. - LM engines. Any engine that works with
asyncioand streams incremental outputs asynchronously can be used; the authors have integrated vLLM and SGLang Runtime.
7 · Evaluation setup and results
| Item | Setting |
|---|---|
| Pipeline | The six-stage FacTool-inspired pipeline, with the instance counts in Section 1 |
| Hardware | One NVIDIA HGX node with 8 A100-SXM GPUs (80 GB each) and AMD EPYC 7763 CPUs |
| Software | vLLM for LM serving; a retrieve-and-rerank pipeline with a custom streaming BM25 and ColBERT as reranker |
| Input | SQuAD questions, injected as a Poisson process at varying rates |
| Routing | hash(request_id) mod n at every stage |
| Metrics | Achieved load (requests sent in a 12-minute interval divided by the wall-clock time to finish them all), median and P99 latency; only points where achieved load was at least 80% of offered load are shown |
| Comparison | The same pipeline on ALTO, waiting for all LM generations to complete before forwarding (baseline) or streaming partial outputs |
- Throughput: streaming partial outputs sustains up to 3× higher load at a latency target of 4 seconds per request (the paper's Figure 3a).
- Tail latency: streaming gives 1.8× lower P99 latency at low load (Figure 3b).
- Shape of the curves: in Figure 3, the baseline's median and P99 latency climb steeply at a much lower load than the streaming configuration's.
8 · Open directions the paper names
- Inferring constraints. Today developers tag data by hand. Instead, they could annotate which stages are stateful, how many levels of quanta they aggregate, and which stages fan out; ALTO would generate the tags. For the FacTool pipeline, marking stage 4 as stateful would let the runtime infer that all tokens of a search query go to one BM25 instance.
- A library of aggregation operators. Each application now writes its own aggregation logic. Common patterns (sum, top-k, count, filter) could become a library tied to the constraint interface.
- Distributed prompt-aware scheduling. Two goals: balance load under dynamic fan-out by learning prompts' relative resource use and assigning GPU time accordingly, and keep prompt locality to benefit from prefix sharing. A first design would track, per prompt, how much output a request creates, how long it takes to serve and the queueing delay at each LM instance, then solve an optimization that favors "sticky" routing of a prompt to the same instance. Measuring these statistics is hard because engines batch requests internally and sometimes de-prioritize some relative to others (the paper cites RadixAttention). The paper defers this to future work.
- Other extensions. Consistent hashing to keep load even as instances join and leave; network sockets for multi-node deployment; and the claim-appraisal stage in a later version of the pipeline.
FAQ
Q·What does "network orchestrator" mean here?▾
ALTO does not run models itself. It is the layer between pipeline stages: queues carry each stage's outputs to a central runtime, which decides which instance of the next stage receives each item and forwards it there. Model execution stays inside the stages, which use engines such as vLLM or SGLang.
Q·How is this different from LM serving engines like vLLM or SGLang?▾
Those engines make one LM run efficiently by managing its memory across requests, and they already expose token streaming. The paper points out that they do not handle distributed deployments or the correctness and load-balancing problems that appear when partial outputs stream between stages, and that they optimize LMs alone rather than LMs combined with other tools. ALTO uses them as the LM executors inside its stages.
Q·Why not build this on Kafka, Spark Streaming, Naiad or Ray?▾
Those systems offer exactly-once semantics and aggregation operators over streams, and like ALTO they pipeline computation. The paper's argument is that they do not easily let developers specify hierarchically nested streams and track the ancestry of each partial output as it fans out (request, then claim, then query, then token), which is what aggregation-aware routing needs. Stream processors also target long-running queries with long-lived state, while ALTO aims to minimize end-to-end latency of short AI pipeline runs whose resource use varies with LM output length.
Q·How does it differ from microservice and prediction-serving frameworks?▾
Frameworks such as Ray and SEDA, and ML-specific ones such as InferLine, already deploy pipelines of independent stages that communicate through queues. ALTO borrows that structure. What they do not do, according to the paper, is exploit the LM's token-by-token generation to stream partial outputs. ALTO also replaces RPCs with an API that carries aggregation constraints and prompt information to the scheduler.
Q·Isn't this the same as parallelizing LM calls, as LLM compilers do?▾
Related but different. Prior work automatically runs logically independent LM calls in parallel, and ALTO can also run parallel stages concurrently. The paper notes that those approaches neither stream partial outputs nor handle the correctness and load-balancing issues that streaming introduces.
Q·Do stages have to be written to accept partial outputs?▾
Yes, a stage must be able to work on a piece of its input at a time. In the evaluation, the BM25 retriever is a custom implementation designed for streaming: it computes document relevance scores for each query token and then sums across the query's tokens. Each application currently also writes its own aggregation logic; the paper proposes a library of common operators (sum, top-k, count, filter).
Q·Does keeping related items on one instance hurt load balancing?▾
It can, and the paper's Figure 4 shows how: if all five outputs of one request must go to one instance while another request has only three, the second instance falls idle first. That is why the paper argues for applying aggregation-aware routing only at stages that actually aggregate, instead of a single global rule, and lets unconstrained items go round robin.
Q·Who decides which items must be aggregated together?▾
Today, the application developer, by attaching the same array of IDs to every item that must meet at one instance (for a BM25 token: its object, claim and query IDs). The runtime itself adds only a global request ID. The paper proposes letting developers instead annotate which stages are stateful and which fan out, so ALTO could infer the tags.
Q·What is distributed prompt-aware scheduling, and does ALTO do it?▾
It is scheduling that knows which prompt each LM request uses. It would assign GPU time to prompts according to their measured resource use (so instances are not idle under one prompt and overloaded under another) and keep requests for the same prompt on the same instance where possible, so the engine can reuse cached prefix state. ALTO's current scheduler does not do this; the paper motivates it with measurements and sketches a design built on per-prompt statistics and "sticky" routing.
Q·Does ALTO run across several machines?▾
Not in the prototype. Queues are built on UNIX domain sockets, so stages and runtime share one machine; the paper notes that wrapping network sockets instead would extend the design to a multi-node setting. The evaluation uses a single 8-GPU node.
Q·What does the evaluation not show?▾
It covers one pipeline (without FacTool's claim-appraisal stage) on one node, and both configurations run on ALTO, so it measures the effect of streaming rather than ALTO against another serving system. The constraint-aware scheduler and prompt-aware scheduling are not part of the measured prototype, which routes by hashing the request ID at every stage. The paper presents the measurements as motivation and the scheduling designs as future work.