Skip to content

Commit 10b1752

Browse files
authored
Support models that require tool strategy. (#103)
1 parent be95d64 commit 10b1752

File tree

6 files changed

+1402
-45
lines changed

6 files changed

+1402
-45
lines changed

splunklib/ai/README.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -554,6 +554,60 @@ async with Agent(
554554
result.structured_output.summary
555555
```
556556

557+
### Output schema generation details
558+
559+
When an `output_schema` is configured, the SDK automatically selects a strategy for generating
560+
structured output based on the capabilities of the underlying model:
561+
562+
- **Provider strategy** - used when the model natively supports structured output.
563+
564+
- **Tool strategy** - used as a fallback when the model does not natively support structured outputs.
565+
The LLM passes the structured output into a tool call, according to the tool input schema. The
566+
tool schema correspponds to the `output_schema` pydantic model as passed to the `Agent` constructor.
567+
In that case the returned `AIMessage` will contain the `structured_output_calls` field populated
568+
and a `StructuredOutputMessage` will be appended to the message list, since each tool call must
569+
have a corresponding tool response.
570+
571+
The strategy is selected automatically - no configuration is required.
572+
573+
#### Output schema generation failure
574+
575+
Output schema generation can fail for various reasons:
576+
577+
- The model produces output that does not conform to the schema (e.g. wrong type, missing field,
578+
invalid enum value).
579+
- The schema contains logic that cannot be fully expressed the encoded schema, which gets passed
580+
to the LLM - for example, a `model_validator`/`field_validator` that enforces a constraint.
581+
Because the model has no visibility into such constraints at generation time, it may produce
582+
values that pass schema validation but are then rejected by the validator at parse time.
583+
584+
```py
585+
class Output(BaseModel):
586+
min_score: float
587+
max_score: float = Field(descripiton="max_score must be less or equal than min_score")
588+
589+
@model_validator(mode="after")
590+
def max_must_exceed_min(self) -> "Output":
591+
if self.max_score <= self.min_score:
592+
raise ValueError("max_score must be greater than min_score")
593+
return self
594+
```
595+
- In case of **tool strategy** if the LLM model returned multiple structured output tool calls.
596+
597+
By default the output schema generation is re-tried, until the LLM generates a valid output.
598+
This happens differently depending on the output schema generation strategy.
599+
600+
- **Provider strategy** - the validation error is fed back to the model, which is asked to
601+
regenerate the output in the same agentic loop iteration.
602+
- **Tool strategy** - the validation error is returned as a tool response, prompting the model
603+
to retry the structured output tool call in the same agentic loop iteration.
604+
605+
On each failed attempt, a `StructuredOutputGenerationException` is raised inside the model
606+
middleware chain. If the exception propagates out of the last middleware, the SDK catches it and
607+
triggers the retry logic described above. A custom `model_middleware` can intercept this exception
608+
to observe, log, or override the retry behavior. A custom `model_middleware` can also raise
609+
the `StructuredOutputGenerationException` manually to reject structured output and force a re-generation.
610+
557611
### Subagents with structured output/input
558612

559613
In addition to output schemas, subagents can define input schemas. These schemas both constrain

0 commit comments

Comments
 (0)