Models and providers
Agents reach models through LiteLLM, so a model name is all it takes to switch providers, from OpenAI or Anthropic to a vLLM server on your own machine. An agent never talks to a provider itself: an invoker turns the dialog into a request and the response into a message, so tools, parsing, repairs, and retries work the same with every model.
from lllm import Tactic
for model in [
"gpt-4o-mini", # (1)!
"anthropic/claude-sonnet-4-5",
"gemini/gemini-2.5-flash",
"ollama/llama3.2", # (2)!
]:
reply = Tactic.quick("Name one prime number above 90.", model=model)
print(f"{model:30}{reply.content}")
#> gpt-4o-mini 97
#> anthropic/claude-sonnet-4-5 97
#> gemini/gemini-2.5-flash 97
#> ollama/llama3.2 97
- OpenAI models need no prefix. Other providers are chosen by the prefix
before the
/. - A model served by a local Ollama, with no key at all.
Model names
A model name is LiteLLM's: a provider prefix and the provider's own name for the model. Some common ones:
| Provider | Model name | Credentials |
|---|---|---|
| OpenAI | gpt-4o-mini, o4-mini |
OPENAI_API_KEY |
| Anthropic | anthropic/claude-sonnet-4-5 |
ANTHROPIC_API_KEY |
| Google Gemini | gemini/gemini-2.5-flash |
GEMINI_API_KEY |
| Vertex AI | vertex_ai/gemini-2.5-pro |
VERTEXAI_PROJECT, VERTEXAI_LOCATION |
| Azure OpenAI | azure/<deployment> |
AZURE_API_KEY, AZURE_API_BASE, AZURE_API_VERSION |
| NVIDIA NIM | nvidia_nim/<model> |
NVIDIA_NIM_API_KEY, NVIDIA_NIM_API_BASE |
| OpenRouter | openrouter/<model> |
OPENROUTER_API_KEY |
| Ollama | ollama/llama3.2 |
none |
| An OpenAI-compatible server | openai/<model> plus api_base |
the server's key, if any |
LiteLLM's provider list has the
rest. In a tactic, set model_name once under global or per agent
(Configuration).
API keys
LiteLLM reads each provider's key from the environment variables in the
table above, such as OPENAI_API_KEY. LLLM checks the providers that need
several variables when it loads the invoker: if some of the Vertex AI, NVIDIA NIM, or Azure variables are set
but not all, it raises ValueError and lists them. If no known provider
key is set at all, it logs a warning, which you can ignore for a local
server. A key for one agent only, such as a local server's, can go in its
model_args as api_key. To store the keys a package needs, see
Configuration.
Model settings
model_args are sent with every call. They come from the agent's config,
and respond(args=...) overrides them for one call. The call's session
shows what was sent:
from lllm import Tactic
agent = Tactic.quick(
system_prompt="You write haiku.",
model="gpt-4o-mini",
temperature=0.7, # (1)!
max_tokens=60,
)
agent.open("poem")
agent.receive("Write a haiku about autumn rain.")
session = agent.respond(args={"temperature": 1.0}, return_session=True)
print(session.invoke_traces[0][0].model_args) # (2)!
"""
{'temperature': 1.0, 'max_tokens': 60, 'drop_params': True, 'stream': False}
"""
- Extra keyword arguments to
Tactic.quick()becomemodel_args. invoke_traceskeeps every model call ofrespond(), keyed by tool round. Each entry records the arguments that went to LiteLLM.
LLLM always sends drop_params=True, so a setting the provider does not
support, such as seed on a model without it, is dropped instead of
failing the call. Anything LiteLLM accepts can go in model_args:
| Setting | Does |
|---|---|
temperature, top_p, seed, stop |
Sampling and stop sequences |
max_tokens, max_completion_tokens |
Cap the reply's length |
api_base, api_key |
Where to send the request, and with which key |
timeout |
Seconds to wait for the provider |
reasoning_effort |
Reasoning budget, on models that have one |
logprobs, top_logprobs |
Token log probabilities on message.logprobs (prompts without format) |
extra_headers, extra_body |
Extra HTTP headers and request fields |
Local and OpenAI-compatible servers
vLLM, SGLang, llama.cpp, and LM Studio speak the OpenAI API. Use the
openai/ prefix with the model's name on the server, and point api_base
at it:
from lllm import Tactic
reply = Tactic.quick(
"Say hello in Spanish.",
model="openai/Qwen/Qwen2.5-7B-Instruct",
api_base="http://localhost:8000/v1",
api_key="local", # (1)!
)
print(reply.content)
#> ¡Hola!
print(reply.cost.cost) # (2)!
#> 0.0
- Any string works if the server does not check keys.
- LiteLLM has no price for a model it does not know, so local calls cost nothing in LLLM's accounting. Token counts are still recorded.
In a config, the same settings go under model_args:
global:
model_name: openai/Qwen/Qwen2.5-7B-Instruct
model_args:
api_base: http://localhost:8000/v1
api_key: local
LiteLLM also has hosted_vllm/ for vLLM, and ollama/ uses Ollama's
default address, http://localhost:11434, unless you set api_base.
Servers with KV-cache-aware routing, such as NVIDIA Dynamo, vLLM, and
SGLang, can also use what an agent knows about its dialogs; see
Inference hints.
Retries and rate limits
Providers fail now and then. The agent loop retries, with separate budgets for each kind of failure (The agent loop):
- Rate limits (
RateLimitError, or HTTP 429) waitmin(60, 2**n)seconds plus up to a second of jitter, up tomax_rate_limit_retriestimes (8 by default). - Other provider errors, such as a dropped connection, are retried
after a second, up to
max_llm_recalltimes (0 by default). - Replies that do not parse are not provider errors. They are repaired
through the prompt's handler, up to
max_exception_retrytimes.
When a provider budget runs out, respond() raises the provider's
exception; when repairs run out, it raises AgentException. Here the first
request drops its connection:
from lllm import Tactic
agent = Tactic.quick(system_prompt="You answer briefly.", model="gpt-4o")
agent.max_llm_recall = 2 # (1)!
agent.open("q")
agent.receive("What is the boiling point of water at sea level?")
session = agent.respond(return_session=True)
print(session.delivery.content)
#> 100 °C, or 212 °F.
print(session.llm_recalls_count, type(session.llm_recalls[0][0]).__name__)
#> 1 APIConnectionError
- In a config, set
max_llm_recall: 2on the agent or underglobal.
Chat Completions or Responses
Agents use the Chat Completions API by default. Set api_type: response
on an agent to use OpenAI's Responses API instead; the agent loop is the
same, and only the request changes:
- A prompt's
formatis not supported (ValueError); use a parser. Prompt(addon_args={"web_search": True})adds the provider's web search tool for that turn, andaddon_args={"computer_use": {...}}adds computer use.max_output_tokens(ormax_completion_tokens) inmodel_argscaps the reply, 32000 by default.- Reasoning summaries, when the model returns them, are kept in
message.metadata["reasoning"].
Cost
The invoker records each reply's token usage and its price from LiteLLM's
price table. message.cost is one reply's, and the totals add up: an
AgentCallSession has cost, a dialog has cost, and a tactic's call
session has total_cost, including nested tactic calls.
from lllm import Tactic
reply = Tactic.quick("Summarize Hamlet in one line.", model="gpt-4o-mini")
cost = reply.cost
print(cost.prompt_tokens, cost.completion_tokens, f"${cost.cost:.6f}")
#> 10 20 $0.000013
See Observability for recording costs across runs.
Invokers
The default invoker is LiteLLMInvoker. A tactic builds one for its agents
from the config's invoker key (litellm unless you name another), and
build_invoker({"invoker": "litellm"}) builds one for an Agent you
construct yourself. One invoker serves every call of a tactic instance.
Custom invokers
Write an invoker to reach a model LiteLLM does not cover. Subclass
BaseInvoker and implement call(), which receives the working dialog
and returns an InvokeResult:
from lllm import Message, Role, Tactic, register_invoker
from lllm.runtimes.native import InvokeResult
from lllm.runtimes.native.invokers.base import BaseInvoker
def house_model(text: str) -> str:
"""Stands in for the client of your own model server."""
return f"[house-v1] {len(text.split())} words received."
class HouseInvoker(BaseInvoker):
def call(self, dialog, model, model_args=None, parser_args=None,
responder="assistant", metadata=None, **kwargs):
content = house_model(dialog.tail.content)
errors, parsed = [], {}
try:
parsed = dialog.top_prompt.parse(content, **(parser_args or {}))
except Exception as error:
errors.append(error) # (1)!
message = Message(
role=Role.ASSISTANT, name=responder, content=content,
parsed=parsed, model=model,
)
return InvokeResult(message=message, execution_errors=errors)
register_invoker("house", HouseInvoker) # (2)!
class Count(Tactic):
name = "count"
agent_group = ["counter"]
def call(self, text: str) -> str:
counter = self.agents["counter"]
counter.open("count")
counter.receive(text)
return counter.respond().content
count = Count(
{
"invoker": "house", # (3)!
"global": {"model_name": "house-v1"},
"agent_configs": [
{"name": "counter", "system_prompt": "You count words."}
],
}
)
print(count("the quick brown fox"))
#> [house-v1] 4 words received.
- A reply that fails to parse goes into
execution_errorsinstead of raising, so the agent can send the prompt's repair message. - A builder receives the config's
invoker_config(the whole config if there is none) and returns an invoker; the class itself is one. Registering a taken name raises unless you passoverwrite=True. - The config chooses the invoker. For an
Agentyou build yourself, passllm_invoker=HouseInvoker({}).
A complete invoker also follows the rest of the contract:
- Read the turn's tools,
format, and parser fromdialog.top_prompt. - Return a message with role
Role.TOOL_CALLandfunction_callswhen the model asks for tools; the agent runs them. - Fill
message.usage(prompt_tokens,completion_tokens, andresponse_costwhen known) so costs add up. - Call
stream_handler.handle_chunk()with each piece of text, when one is given (Streaming). - Raise provider failures. The agent decides whether to retry.
To change how calls are made rather than where they go, such as caching or
routing, subclass LiteLLMInvoker from
lllm.runtimes.native.invokers.litellm and wrap its call(). For tests,
lllm.testing answers model calls without writing an
invoker.