Inference hints
An agent knows things a model server cannot see in a single request: which
calls belong to one line of reasoning, which dialogs were forked from
which and so share a prompt prefix, how urgent a call is, and roughly how
long the answer will be. InferenceHints sends that with every model call,
in the form NVIDIA Dynamo, vLLM, or SGLang reads, so a KV-cache-aware
router can keep a dialog on the worker that holds its cache, warm the cache
for the next turn, and serve urgent work first.
The session identity comes from the dialogs themselves, so there is nothing to track by hand:
from lllm import Agent, InferenceHints, Prompt
from lllm.runtimes.native.invokers import build_invoker
agent = Agent(
name="planner",
system_prompt=Prompt(path="planner/system", prompt="You plan studies."),
model="openai/Qwen/Qwen2.5-7B-Instruct",
llm_invoker=build_invoker({"invoker": "litellm"}),
model_args={
"api_base": "http://localhost:8000/v1",
"api_key": "local",
"max_tokens": 256,
},
inference_hints=InferenceHints(profile="dynamo", priority=5), # (1)!
)
def hint_fields(session):
"""What the call sent, with each dialog id replaced by its alias."""
sent = session.invoke_traces[0][0].model_args # (2)!
alias = {d.dialog_id: name for name, d in agent.dialogs.items()}
headers = {k: alias[v] for k, v in sent["extra_headers"].items()}
return headers, sent["extra_body"]
agent.open("main")
agent.receive("Plan a study of prefix caching.")
headers, body = hint_fields(agent.respond(return_session=True))
print(headers)
#> {'x-dynamo-session-id': 'main'}
print(body) # (3)!
#> {'nvext': {'agent_hints': {'priority': 5, 'osl': 256}}}
agent.receive("List the risks of the prefix caching study.")
headers, _ = hint_fields(agent.respond(return_session=True))
print(headers) # (4)!
#> {'x-dynamo-session-id': 'main'}
agent.fork("main", "small")
agent.receive("Adapt the plan for a 1B model.")
headers, _ = hint_fields(agent.respond(return_session=True))
print(headers) # (5)!
#> {'x-dynamo-session-id': 'small', 'x-dynamo-parent-session-id': 'main'}
- Hints are off unless you set them.
profilepicks the serving stack. invoke_tracesrecords the arguments each model call sent to LiteLLM, and the hints are inextra_headersandextra_body. Dialog ids are random hex strings, so the helper prints aliases in their place.osl, the expected output length, defaults to the agent'smax_tokens.- The second turn of a dialog is the same session, so the router can send it to the worker that already holds the first turn's KV cache.
- A forked dialog is a new session that names its parent, which tells the router it starts with the parent's prefix.
Turning hints on
Pass inference_hints= to an Agent, or set inference_hints in a
config, under global or per agent. The two are merged like any other
dict, so a tactic can share a profile and give each agent its own
priority:
global:
model_name: openai/Qwen/Qwen2.5-7B-Instruct
model_args: {api_base: "http://localhost:8000/v1", max_tokens: 512}
inference_hints: {profile: dynamo, session_prefix: run-42}
agent_configs:
- name: planner # the user is waiting on it
system_prompt: You plan.
inference_hints: {priority: 10, speculative_prefill: true}
- name: summarizer # background work
system_prompt: You summarize.
inference_hints: {priority: 1}
| Field | Default | Meaning |
|---|---|---|
profile |
openai |
The serving stack: dynamo, vllm, sglang, or openai |
session |
True |
Send the dialog's session identity |
session_prefix |
Put a prefix on session ids, such as a run id: run-42:<dialog id> |
|
priority |
How urgent the call is; higher is more urgent | |
strict_priority |
Dynamo's router queue tier; higher tiers always go first | |
osl |
Expected output length in tokens (Dynamo); defaults to max_tokens or max_completion_tokens |
|
speculative_prefill |
Ask Dynamo to warm the cache for the next turn | |
cache_salt |
vLLM prefix-cache salt, to keep tenants' cached prefixes apart |
Fields you leave unset are not sent, and an unknown field is a validation
error. Hints are computed once per respond() and sent with every model
call it makes, including tool rounds and repairs.
Sessions follow dialogs
A dialog is one line of reasoning, so it is one session: every turn of a
dialog carries the same id, open() starts a new session, and
fork(alias, child_alias) starts a child session that names its parent.
A router that keeps sessions on one worker reuses the dialog's KV cache
turn after turn, and a fork can be placed next to its parent's prefix.
Session ids are dialog ids, with session_prefix in front when you set
one. session=False sends no identity, only the other fields. Inside a
tactic, every call builds fresh agents, so each call's dialogs are new
sessions.
What each server receives
Each profile writes the same knowledge in its server's own fields:
| Profile | Session headers | Body fields |
|---|---|---|
dynamo |
x-dynamo-session-id, x-dynamo-parent-session-id |
nvext.agent_hints: priority, strict_priority, osl, speculative_prefill |
vllm |
x-session-id, x-parent-session-id |
priority, cache_salt |
sglang |
x-session-id, x-parent-session-id |
priority |
openai |
x-session-id, x-parent-session-id |
none |
request_fields() returns what one call would add, which is handy for
checking a profile:
from lllm import InferenceHints
vllm = InferenceHints(profile="vllm", priority=2, cache_salt="tenant-a")
fields = vllm.request_fields(dialog_id="d2", parent_dialog_id="d1")
print(fields["extra_headers"])
#> {'x-session-id': 'd2', 'x-parent-session-id': 'd1'}
print(fields["extra_body"]) # (1)!
#> {'priority': -2, 'cache_salt': 'tenant-a'}
sglang = InferenceHints(profile="sglang", priority=2)
print(sglang.request_fields(dialog_id="d2")["extra_body"])
#> {'priority': 2}
- In LLLM a higher priority is more urgent, as in Dynamo and SGLang. vLLM
runs lower values first, so the
vllmprofile flips the sign.
The openai profile adds only headers, which servers that do not use them
ignore, so it is safe with any endpoint. Body fields are not: vLLM rejects
a nonzero priority unless it runs with priority scheduling. Explicit
extra_headers or extra_body in model_args always win over hints.
On the server side
Hints change behavior only where the serving stack is set up to use them:
| Stack | What to enable |
|---|---|
| NVIDIA Dynamo | KV-aware routing; --router-session-affinity-ttl-secs to keep sessions on one worker; --router-queue-threshold for priority queueing; --router-track-output-blocks to use osl |
| vLLM | --scheduling-policy priority for priority; prefix caching reuses shared prefixes |
| SGLang | --enable-priority-scheduling for priority; --radix-eviction-policy priority to evict low-priority cache first |
Dynamo also reads the generic x-session-id headers the other profiles
send, and it passes priority on to its backend engines in each engine's
own convention.
Ending a session
Dynamo can release a session's state as soon as the session ends, instead
of waiting for a timeout, when a final request carries
x-dynamo-session-final. final_request_fields() builds its fields:
from lllm import InferenceHints
hints = InferenceHints(profile="dynamo")
print(hints.final_request_fields("d1")["extra_headers"])
#> {'x-dynamo-session-id': 'd1', 'x-dynamo-session-final': 'true'}
The agent does not send it for you. When a dialog is done, send a minimal
request with these fields, for example with litellm.completion(...,
max_tokens=1, **fields), using the dialog's dialog_id.
Reference
InferenceHints(profile="openai", session=True, session_prefix=None,
priority=None, strict_priority=None, osl=None, speculative_prefill=None,
cache_salt=None) is a Pydantic model, importable from lllm.
| Method | Returns |
|---|---|
request_fields(*, dialog_id=None, parent_dialog_id=None, model_args=None) |
{"extra_headers": ..., "extra_body": ...} for one call; model_args supplies the osl default |
final_request_fields(dialog_id) |
The same, plus Dynamo's end-of-session header |
session_ids(dialog_id, parent_dialog_id=None) |
The session and parent ids that would be sent |
See Models for pointing agents at your own server, and Dialogs for forking.