Runtimes
A runtime is whatever does the work behind a tactic: LLLM's native agents,
plain Python, a Pydantic AI agent, or a service on another machine. All of
them implement one small protocol, lllm.protocol.Tactic, so the code that
calls, serves, packages, and composes tactics never needs to know which
runtime it has.
Here are three runtimes behind the same boundary, called by the same loop:
from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
from lllm import PydanticAITactic, Tactic, as_tactic
NOTE = "Deploys now pause when the error rate crosses two percent."
def word_count(text: str) -> int: # (1)!
"""Count the words in a text."""
return len(text.split())
class Headline(Tactic): # (2)!
"""Write a headline for a note."""
name = "headline"
agent_group = ["editor"]
input_type = str
output_type = str
def call(self, task: str) -> str:
editor = self.agents["editor"]
editor.open("headline")
editor.receive(f"Write a headline for: {task}")
return editor.respond().content
headline = Headline(
{
"global": {"model_name": "gpt-4o-mini"},
"agent_configs": [{"name": "editor", "system_prompt": "You edit."}],
}
)
tagger = Agent(TestModel(custom_output_text="reliability"), name="tagger")
for tactic in [as_tactic(word_count), headline, PydanticAITactic(tagger)]:
info = tactic.info() # (3)!
print(f"{info.name}: {info.runtime_kind} -> {tactic.run(NOTE)!r}")
#> word_count: python -> 10
#> headline: native -> 'Deploys Pause When Errors Top 2%'
#> tagger: pydantic-ai -> 'reliability'
- A plain function.
as_tactic()reads its types from the annotations and its description from the docstring. - A native program:
call()runs an agent that is built fresh for every call. - Every tactic describes itself the same way.
runtime_kindis the only field that says what runs inside.
The loop treats the three the same way, and so does the rest of LLLM:
lllm serve serves any of them, a package lists any of them,
and tactic_as_function() hands any of them to an agent as a
tool. Choosing a runtime is a decision about one
tactic, and you can change it later without touching its callers.
The protocol
lllm.protocol.Tactic is a small generic class with no agent machinery. Its
public methods are the boundary:
| Method | Does |
|---|---|
run(value, *, context=None, return_trace=False, **kwargs) |
Validate the input, do the work, validate the output |
await arun(...) |
The same, async |
stream(value, ...), astream(...) |
Yield chunks, for tactics that stream |
aevents(...) |
Yield TacticEvents; a service's /stream sends these |
info() |
The contract: name, description, JSON schemas, capabilities, runtime_kind |
supports(name), capabilities() |
What it can do, such as stream or events |
run() enforces the boundary, and a runtime fills in one step of it:
Validation uses the tactic's input_type and output_type: anything
Pydantic can validate, such as a model, str, or list[str]. Every call
carries a CallContext with ids for tracing and per-request metadata; one is
created when the caller passes none. Typed I/O and
Tactics cover both from the caller's side.
LLLM ships these runtimes:
| Runtime | You write | runtime_kind |
|---|---|---|
| Native | an lllm.Tactic subclass with a call() program |
native |
| Plain Python | as_tactic(fn), or an lllm.Tactic with a _run() hook |
python |
| Pydantic AI | PydanticAITactic(agent) |
pydantic-ai |
Objects with run() or call() |
NativeTacticAdapter(obj) |
native |
| A service | RemoteTactic(url) |
http |
Wrappers such as ProxyTactic (hooks and call records) and
SandboxedTactic (limits at the call boundary) take a tactic of any of these
kinds, because they only use the protocol.
Plain functions
as_tactic() turns a function into a tactic. Types come from the
annotations, the name from the function, and the description from its
docstring:
from pydantic import BaseModel
from lllm import CallContext, as_tactic
class Order(BaseModel):
sku: str
quantity: int
class Quote(BaseModel):
sku: str
total: float
def quote(order: Order, context: CallContext) -> Quote:
"""Price an order at 2.50 per unit."""
print("trace:", context.trace_id)
#> trace: t-1
return Quote(sku=order.sku, total=order.quantity * 2.5)
tactic = as_tactic(quote)
order = {"sku": "A-1", "quantity": 4}
print(repr(tactic.run(order, context=CallContext(trace_id="t-1"))))
#> Quote(sku='A-1', total=10.0)
print(tactic.info().name, "|", tactic.info().description)
#> quote | Price an order at 2.50 per unit.
- The input type is the first parameter's annotation (a parameter named
contextis skipped), and the output type is the return annotation.input_type=andoutput_type=override them. - A function with a
contextparameter, or**kwargs, receives theCallContext. Extra keyword arguments torun()are passed through. - Async functions work with
arun()andrun().run()drives them withasyncio.run(), so inside a running event loop, usearun(). as_tactic()also takesname,description,package_ref,service_ref,examples, andmetadata, and returns aCallableTactic.
The native runtime
lllm.Tactic, the class the rest of this guide uses, is the native
runtime's program class, and it subclasses lllm.protocol.Tactic. There is
no adapter step: the class you write is the one that is served, packaged,
and handed to other runtimes. A class that writes call() is a native
program: each call runs on a fresh copy with fresh agents, inside a session
that records agent calls and cost. A subclass that implements only the
protocol hook _run() reports python. See Tactics.
Pydantic AI
PydanticAITactic(agent) wraps a Pydantic AI agent: run() calls the
agent's run_sync(), arun() calls run(), stream() uses
run_stream_sync(), and aevents() uses run_stream_events(). Its options,
and giving LLLM tactics to Pydantic AI agents as tools, are on
Pydantic AI.
Objects with run() or call()
Code you already have may not be a tactic but have the right shape: a
run() or call() method, and maybe arun(), acall(), or stream().
NativeTacticAdapter puts the boundary around it without changing it:
from lllm import CallContext, NativeTacticAdapter
class KeywordTagger:
"""Tag a text with words from a fixed vocabulary."""
name = "keyword-tagger"
input_type = str
output_type = list[str]
def __init__(self, vocabulary):
self.vocabulary = vocabulary
def run(self, text, *, limit=3, metadata=None):
print("trace:", metadata["lllm_trace_id"])
#> trace: t-2
words = text.lower().replace(".", "").split()
return [word for word in self.vocabulary if word in words][:limit]
tagger = NativeTacticAdapter(
KeywordTagger(["deploys", "errors", "latency", "rollback"]),
run_kwargs={"limit": 2}, # (1)!
)
note = "Rollback deploys when errors spike."
print(tagger.run(note, context=CallContext(trace_id="t-2")))
#> ['deploys', 'errors']
print(tagger.info().name, "|", tagger.info().description)
#> keyword-tagger | Tag a text with words from a fixed vocabulary.
- Added to every call. Keyword arguments to
run()are merged over them.
The adapter calls run() if the object has one, else call(), and uses
arun() or acall() for async calls when they exist (otherwise a worker
thread). It takes the types from the object's input_type or input_model
and output_type or output_model unless you pass them, and the name and
description from its name and class docstring. A method that accepts
context gets the CallContext; one that accepts metadata gets the
context's metadata with lllm_request_id, lllm_trace_id, and the other ids
added (turn this off with include_context_metadata=False).
Services
RemoteTactic(url) is a tactic backed by a served tactic: run() posts to
its /run, the stream methods read its /stream, and fetch_info() reads
its /info. Because it is a tactic, a remote service fits wherever a local
one does, including as an agent's tool. See Serving and the
HTTP service example.
Writing a runtime adapter
To bring another runtime to LLLM, subclass lllm.protocol.Tactic and
implement the hooks it needs. Here is a complete adapter for objects with
invoke() and stream() methods:
import asyncio
import inspect
from typing import Any
from lllm.protocol import Tactic
class InvokeTactic(Tactic[Any, Any]):
runtime_kind = "invoke" # (1)!
def __init__(
self, runnable, *, input_type=None, output_type=None, **kwargs
):
self.runnable = runnable
self.input_type = input_type
self.output_type = output_type
kwargs.setdefault("description", inspect.getdoc(type(runnable)))
super().__init__(**kwargs) # (2)!
def _run(self, input_value, *, context=None, **kwargs): # (3)!
return self.runnable.invoke(input_value, **kwargs)
def stream(self, input_value, *, context=None, **kwargs): # (4)!
yield from self.runnable.stream(input_value, **kwargs)
class Sentences:
"""Split a text into sentences."""
def invoke(self, text):
parts = [part.strip() for part in text.split(".")]
return [f"{part}." for part in parts if part]
def stream(self, text):
yield from self.invoke(text)
tactic = InvokeTactic(
Sentences(), name="sentences", input_type=str, output_type=list[str]
)
info = tactic.info()
print(info.name, info.runtime_kind, info.capabilities)
#> sentences invoke ('arun', 'run', 'stream')
print(info.description)
#> Split a text into sentences.
print(tactic.run("Deploys pause. Alerts fire."))
#> ['Deploys pause.', 'Alerts fire.']
async def main():
async for event in tactic.aevents("Deploys pause. Alerts fire."):
print(event.kind, event.data)
#> message Deploys pause.
#> message Alerts fire.
asyncio.run(main())
- A short label, shown in
info(),/info, and on hub cards. - Pass
name,description,package_ref,service_ref,examples, andmetadatathrough to the base class. _run()receives the validated input and the call'sCallContext. Return anything that validates intooutput_type.- Writing
stream()adds thestreamcapability.astream()andaevents()come for free: they validate the input, then runstream()on a worker thread.
| Hook | Write it when | Default |
|---|---|---|
_run() |
always, unless you write _arun() |
raises TacticUnsupportedError |
_arun() |
the runtime is natively async | runs _run() on a worker thread |
stream() |
the runtime can stream | raises TacticUnsupportedError |
astream() |
its stream is natively async | runs stream() on a worker thread |
aevents() |
it has its own event kinds | wraps each chunk as TacticEvent(kind="message") |
capabilities() |
support depends on the wrapped object | derived from the methods you override |
info() |
the runtime knows more, such as an output schema | built from the class and constructor |
Calling stream() directly skips input validation, so callers that start
from raw data should use astream() or aevents(), as services do.
How entrypoints become tactics
lllm serve module:attribute accepts more than a tactic. It passes the
value to coerce_tactic() (from lllm.runtimes), which accepts, in order:
- a
Tacticsubclass, which it instantiates, or aTacticinstance; - a factory, which it calls once at load: a class that can be constructed without arguments, a function with no parameters, or a function that can be called without arguments and is annotated to return a tactic;
- an object with a
run()method, whoserunit wraps withas_tactic(); - any other callable: a handler, wrapped with
as_tactic()and called once per input.
from lllm import CallableTactic, as_tactic
from lllm.runtimes import coerce_tactic
def greet(name: str = "world") -> str:
return f"Hello {name}"
def make_greeter() -> CallableTactic:
return as_tactic(greet, name="greeter")
print(coerce_tactic(greet).run("Ada"))
#> Hello Ada
print(coerce_tactic(make_greeter).info().name)
#> greeter
greet can be called without arguments too, but it is not annotated to
return a tactic, so it is a handler: it answers each input instead of being
called once at load.
Choosing a runtime
- Native when you are building the agentic system itself: you want to see and steer every turn, parse and repair outputs, fork dialogs, and track agents and cost in sessions.
- Plain Python for deterministic steps such as parsing, scoring, and lookups, and for wrapping code that has nothing to do with agents.
- Pydantic AI when an agent is already written for Pydantic AI, or you want its model, tool, and instrumentation ecosystem.
- Remote when the work runs elsewhere: another team's service, a GPU host, or a process you scale separately.
The choice is local to one tactic. A native program can call a Pydantic AI tactic, which can use a native tactic as a tool, and any of them can move behind a service without its callers changing.