Testing
lllm.testing replaces the model, and only the model. Inside
with ScriptedModel([...]) or with FunctionModel(fn), every call a native
agent makes to a model is answered locally, while the rest of LLLM runs for
real: the dialog becomes messages, replies go through the prompt's parser
and format, bad replies are repaired, and tool calls run your functions.
A test therefore checks your program, not a mock of it, and needs no API
key or network.
from lllm import Tactic
from lllm.testing import ScriptedModel
class Moderate(Tactic):
name = "moderate"
agent_group = ["moderator"]
def call(self, comment: str) -> str:
moderator = self.agents["moderator"]
moderator.open("check")
moderator.receive(f"Is this comment safe to post? {comment}")
verdict = moderator.respond().content.lower()
return "published" if verdict.startswith("yes") else "held"
moderate = Moderate({
"global": {"model_name": "gpt-4o-mini"},
"agent_configs": [
{"name": "moderator", "system_prompt": "Answer yes or no."}
],
})
with ScriptedModel(["Yes."]): # (1)!
print(moderate("Great write-up, thanks!"))
#> published
with ScriptedModel(["No, it is a personal attack."]) as model:
print(moderate("You are an idiot."))
#> held
print(model.requests[0].text) # (2)!
#> Is this comment safe to post? You are an idiot.
- Each block scripts the model's replies, so both branches of
call()are tested. - Every request is kept, so a test can also check what the model was asked.
Scripted replies
ScriptedModel(replies) answers calls in order, and once the replies run
out, the last one repeats. A reply is text, a ToolCall, or a list of
ToolCalls, which the model makes at once. Tool calls run your real tools:
from lllm import Prompt, Tactic, tool
from lllm.testing import ScriptedModel, ToolCall
STOCK = {"widget": 3}
@tool(description="Units in stock for a product")
def in_stock(product: str) -> int:
return STOCK.get(product, 0)
ASK = Prompt(path="shop/ask", prompt="{question}", function_list=[in_stock])
agent = Tactic.quick(system_prompt="You run a shop.", model="gpt-4o-mini")
agent.open("chat")
agent.receive_prompt(ASK, {"question": "Do you have widgets?"})
replies = [ToolCall("in_stock", {"product": "widget"}), "Yes, 3 left."]
with ScriptedModel(replies) as model:
session = agent.respond(return_session=True)
print(session.delivery.content)
#> Yes, 3 left.
call = session.tool_calls[0]
print(call.name, call.arguments, call.result) # (1)!
#> in_stock {'product': 'widget'} 3
print([message.role.value for message in agent.current_dialog.messages])
#> ['system', 'user', 'tool_call', 'tool', 'assistant']
in_stockreally ran: the result came fromSTOCK, and the second of the twomodel.requestscarried it back to the model.
Answering from the request
When the replies depend on what the model is asked, pass a function to
FunctionModel. It receives a ModelRequest and returns a reply, so one
function can serve a whole test suite:
from lllm import Prompt, Tactic, tool
from lllm.testing import FunctionModel, ModelRequest, ToolCall
@tool(description="Today's high in a city, in °C")
def high(city: str) -> int:
return {"Oslo": 4, "Rome": 18}[city]
def fake_model(request: ModelRequest):
if request.tool_results(): # (1)!
degrees = request.tool_results()[-1].split("---")[1].strip()
return f"About {degrees}°C today."
if "high" in request.tools:
city = request.text.split()[-1].rstrip("?")
return ToolCall("high", {"city": city})
return "Ask me about the weather."
ASK = Prompt(path="weather/ask", prompt="{question}", function_list=[high])
agent = Tactic.quick(
system_prompt="You report weather.", model="gpt-4o-mini"
)
agent.open("today")
agent.receive_prompt(ASK, {"question": "How warm is it in Rome?"})
with FunctionModel(fake_model) as model:
print(agent.respond().content)
#> About 18°C today.
print([request.tools for request in model.requests])
#> [['high'], ['high']]
tool_results()holds the tool messages since the last user message: none on the first call, the results after a tool round. Each result is framed with the call that produced it, the value between---lines.
ModelRequest |
Holds |
|---|---|
text |
The last message's content, as text |
system |
The system prompt |
tools |
Names of the tools on offer |
tool_results() |
Contents of the tool results since the last user message |
messages |
Every message, as the provider would receive it |
last_message |
The last message, as a dict |
response_format |
The response format the prompt asked for, if any |
model |
The model name |
Testing repairs
Script a reply that fails the prompt's parser or format, then a good one.
The agent sends the error back, and your code only sees the reply that
parsed:
from pydantic import BaseModel
from lllm import Prompt, Tactic
from lllm.testing import ScriptedModel
class Rating(BaseModel):
stars: int
RATE = Prompt(
path="review/rate", prompt="Rate 1-5: {review}", format=Rating
)
agent = Tactic.quick(system_prompt="You rate reviews.", model="gpt-4o-mini")
agent.open("rate")
agent.receive_prompt(RATE, {"review": "Fast shipping, crushed box."})
with ScriptedModel(["Four stars.", '{"stars": 4}']) as model:
session = agent.respond(return_session=True)
print(session.delivery.parsed, session.exception_retries_count)
#> {'stars': 4} 1
print(model.requests[1].text) # (1)!
#> Error: Expecting value: line 1 column 1 (char 0). Please fix.
- The repair message the model received, from the prompt's handler.
A script that never parses ends in AgentException (from
lllm.runtimes.native) after max_exception_retry repairs, which is how
to test that limit.
Testing a tactic end to end
Run the tactic with return_session=True and check the record: which
agents ran, the tools they called, the repairs, and the result. A tactic's
agents and dialogs last for one call, so to check its dialogs, read them in
an observer. Observability covers both. Here is a test
module for the tactic above, with pytest:
import pytest
from lllm.testing import FunctionModel, ScriptedModel
from moderate import moderate
@pytest.fixture(autouse=True)
def no_real_model(): # (1)!
def refuse(request):
raise AssertionError(f"Unexpected model call: {request.text!r}")
with FunctionModel(refuse):
yield
@pytest.mark.parametrize(
("reply", "outcome"),
[("Yes.", "published"), ("No.", "held"), ("I'm not sure.", "held")],
)
def test_verdicts(reply, outcome):
with ScriptedModel([reply]): # (2)!
assert moderate("A comment.") == outcome
def test_one_agent_call_per_comment():
with ScriptedModel(["Yes."]):
session = moderate("Hello!", return_session=True)
assert session.agent_call_count == 1
assert session.agent_sessions["moderator"][0].state == "success"
- A guard for the whole suite: any model call a test did not script fails it, instead of reaching a provider.
- Fake models nest. The innermost one answers, and the outer one takes over again when the block ends.
Good to know
- The fake models patch LiteLLM's completion call for the whole process,
so they also answer calls from other threads, such as
bcallworkers. Do not run tests with different fake models in parallel threads. - They answer chat-completion calls, the default. An agent configured with
api_type: responsestill calls the provider. - LiteLLM builds the responses, so the model name must be one it can route:
any real name, or a provider prefix such as
openai/my-model. - Token counts are LiteLLM's placeholder usage, the same for every call, so costs in tests are stable but not realistic.
- Plain-Python tactics need no fake model at all, and
Pydantic AI agents use Pydantic AI's own
TestModelandFunctionModel.
Every example in these docs runs this way: a FunctionModel answers each
request by matching it against a list of rules
(tests/docs_model.py),
and the docs tests check that each example prints what the page shows.