Brief
Scheduler decouples turn readiness from release to cut agentic workflow tail latency
A tail-risk-aware turn release scheduler for agentic LLM workflows matches eager release under light load and cuts P95 workflow flow time under contention by up to 3.50×, according to an arXiv preprint.
Most agentic LLM runtimes release each model turn the moment it is ready. Under contention that eager release piles up submitted but unfinished work, and once a turn is submitted the workflow-level policy can no longer reorder it, so tail latency grows. The proposed method splits readiness from release: it decides which ready turn goes next and how much released-but-unfinished work to hold.
The scheduler ranks ready turns using online estimates of turn work and a mean-Conditional Value-at-Risk objective that tracks the tail risk of unfinished workflows, and it adapts the released work budget to observed queue pressure. Evaluations on real agent execution traces from software engineering tasks across multiple LLMs and workflow arrival rates show comparable performance to eager release under light load and up to a 3.50× P95 speedup under contention.
Our reading
Our reading is that this is a runtime scheduling policy an agentic workflow orchestrator could adopt without changing the workflow definition itself.
What to do or watch
If you run an agentic workflow orchestrator, the practical next step is to check whether your runtime exposes the point at which a ready turn is submitted, since that is the boundary this scheduler needs to control; the preprint itself is only a proposed policy, not a released component. The unresolved question is whether the reported P95 improvement holds outside the software-engineering traces used in the evaluation, and how the released-work budget would interact with whatever admission cont…
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- Agentic LLM workflows are sequences of model turns interleaved with tool interactions.
- Most runtimes release each turn immediately upon readiness.
- Under contention, eager release accumulates released but unfinished work that the workflow-level policy can no longer reorder, increasing tail latency.
- The method uses a mean-Conditional Value-at-Risk objective and adapts the released work budget to observed queue pressure.
- Evaluated on real agent execution traces from software engineering tasks across multiple LLMs and workflow arrival rates, it achieves up to a 3.50× speedup in P95 workflow flow time under contention.
Sources
- arXivText stored 13 September 2026
How this story was checked. Written from the 1 page listed above, stored 13 September 2026; claims checked against that stored text on 14 September 2026.
What that means
- 5 of 5 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.