Skip to content

Commit 55e246b

Browse files
praveen-formidoclaudepre-commit-ci[bot]
authored
UN-4427 [FEAT] Image output mode: per-model image limits instead of the 90-page / 14 MB caps (#2341)
* UN-4427 [FEAT] sdk1: model-aware image-request limits table and lookup API Image output mode sends every page image to a vision LLM in one request, gated so far by a global 90-page cap and a 14 MiB byte budget that match no provider (Bedrock 20 images, Azure 50, Anthropic 100/600, OpenAI 1,500; 20-100 MB request limits). This adds the data and the API the extraction and answer paths need to check against the limits of the LLM actually used. - utils/image_limits/limits.json: provider-keyed limits with per-field provenance (source URL, quote, verified_on, verified/unverified), built from the independently verified dataset; strict loader (_schema.py) that rejects unknown keys, unresolved sources, bad regexes and inherit targets; optional operator override file via VLM_IMAGE_LIMITS_FILE (per-provider replacement, no deep merge). - resolve.py: provider key = adapter get_provider() with the Bedrock Mantle exception; first-match overrides on the bare model, then cost_model, then extra ids; OpenRouter inherits the family's limits; resolution never reads litellm.model_cost, so the remote and bundled cost maps agree. Default count/byte values are advisory on pass-through providers. - policy.py: page budget = min(image count, context tokens, LLMWhisperer 999, operator ceilings) with the binding term recorded; raw-byte budget from the provider's wire limit; per-image byte/pixel checks including the Anthropic >20-images rule; plain-language messages that name the model, provider, limit and provenance and say whether pages were charged. - LLM.get_image_request_limits / get_image_input_token_budget / image_input_limits_for; the adapter's context_window is now kept. - get_page_images(image_limits=) and load_page_images(limits=) run the model-aware checks; ImageLimitExceededError carries a stable error code. Without a policy both keep today's fixed limits exactly, so this change is behaviour-neutral until callers pass one. Chosen defaults (each one constant): PROMPT_TOKENS_RESERVE=4000, context token term enforced, per-image limits enforced, TEXT_RESERVE_BYTES=64 KiB, ASSUMED_PAGE_BYTES=250 KB for a pre-conversion size advisory (never a rejection), PASS_THROUGH_DEFAULTS_ARE_ADVISORY=True, env vars read as optional ceilings by the new accessors (legacy accessors unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FEAT] Check image-mode documents against the consuming LLM's limits before conversion Image-mode extraction had no idea which LLM would read the pages, so the only pre-conversion check was the fixed 90-page cap. Every extraction now carries the LLM adapter ids that will consume it and is checked against those models' limits before LLMWhisperer bills the page conversion. - Executor: _handle_extract resolves llm_instance_ids into an ImageModePolicy (strictest of each limit across LLMs) and passes it to the x2text adapter, which runs the model-aware pre-submit page check and the post-download byte / per-image checks. Ids that cannot be resolved fail closed before conversion with IMAGE_LIMITS_UNRESOLVED (503, nothing charged). Image-mode limit errors from extraction keep their code as a "CODE: message" prefix, like the answer-time bridge. - Callers: Prompt Studio index (default profile), dynamic_extractor (the running profile) and the structure tool (tool LLM plus every prompt's LLM) send llm_instance_ids. - Upload: the Prompt Studio upload check uses the default profile's LLM limits (offline resolution) and says which profile it used; an unresolvable LLM falls back to the fixed cap. - Bridge: PageImageTooLargeError maps to the new IMAGE_PAGE_TOO_LARGE; the limit codes now come from sdk1 and are re-exported. - Rollout bound: extraction_bound_kwargs() keeps every check within the fixed 90 pages / 14 MiB that the cloud plugin still answers with (LEGACY_ANSWER_BOUNDS_ACTIVE). Strict providers now fail before billing; generous ones stay at 90 until the cloud plugin switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FEAT] Fetch the image-limits table from GitHub so data fixes ship without a release Installs now read limits.json from the main branch of Zipstack/unstract (VLM_IMAGE_LIMITS_URL; empty disables it for air-gapped installs) and keep the bundled copy as the always-available fallback. The operator override file still wins over both. The fetch is bounded: one attempt per process per refresh window (VLM_IMAGE_LIMITS_REFRESH_SECONDS, default 3600, failures included), a 3-second budget and 1 MB cap per download, and other threads keep serving the last good table while one fetches. The remote file passes the same strict schema as the bundled one, must carry the schema_version this sdk1 reads, and may not contain override patterns over 300 characters. Any failure logs a warning and keeps the last good copy. The table records its origin (bundled/remote, location, verified_on, sha256 prefix). The log names it, and format_limit_source appends the version to user-facing errors and the run-log summary. Test suites set VLM_IMAGE_LIMITS_URL="" so they never reach the network. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Reject inheritance loops and keep malformed overrides from escaping the fallback - The loader enforces one-level inherit: the target must exist, must not be the provider itself, and must not inherit in turn. The same check runs on the merged table after an override, so a cycle across the shipped and override layers is rejected too. The resolver also tracks visited providers, so a lookup can never recurse forever. - apply_override_file checks the shapes it merges (sources, providers) before touching them, and invalid JSON is a schema error; the override fallback also catches unexpected errors, so a malformed override is logged and ignored instead of failing image extraction. Addresses Greptile review on #2341. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-4427 [MISC] Fix docstring format flagged by ruff (D205) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Print image-mode byte limits in one unit with one decimal Size messages rounded limits to whole megabytes. Gemma 4's 3.5 MB limit showed as "4 MB", and with the legacy 14 MiB ceiling binding the message read "total 14.7 MB ... over the 15 MB limit". It mixed decimal MB with a MiB ceiling and rounded up past the total. - A message now prints its total and its limit in the limit's own unit, with one decimal: decimal MB for provider limits, MiB for the operator and legacy ceilings (VLM_IMAGE_ANSWER_MAX_TOTAL_MB has always been MiB). - ImageModePolicy.describe() reports the ceiling when it is below the model's limit, instead of the model's limit that does not bind. For example, "14.0 MiB of page images per request" rather than "512 MB per request" on OpenAI during the rollout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Warn on the default token estimate, add a no-fetch table read and the cost-model id From the review of cloud PR C (unstract-cloud#1869): - Token term: the table's only tokens-per-page figure is the unverified top-level default (1,500). Enforcing it capped 128K-context models such as gpt-4o at 82 pages, below the 90 they answered before and below what real 150-DPI pages fit (~765-1,105 tokens). It is now reported as an advisory (ENFORCE_DEFAULT_TOKEN_ESTIMATE = False); a model-specific tokens_per_image in the table is still enforced. - get_limits_table(refresh=False), and LLM.image_input_limits_for(..., refresh_table=False), never fetch from GitHub: they serve the last good copy the process holds, else the bundled one. This is for request paths such as the cloud profile serializer. The LLM limit accessors take an optional table. - LLM.get_cost_model_name(): the real model id (cost_model, e.g. azure/gpt-4o) instead of the routed azure/<deployment>, for vision checks and pricing. - Docs: the env vars replace the 90 / 14 MiB bound while LEGACY_ANSWER_BOUNDS_ACTIVE is on, and only tighten once it is off. The rollout comment no longer says the cloud plugin uses fixed limits. The fetch timeout applies per phase, not as one total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Refresh the limits table in the background and keep bundled providers (O1, O2, O17, O22) O1: get_limits_table() never waits on GitHub. A due refresh starts on a daemon thread behind the existing non-blocking lock; callers keep serving the last good copy. Each attempt has a real overall deadline (the download runs on its own thread, so DNS and connect are covered) and no longer follows redirects. Docstrings that promised "no network" now describe this. VLM_IMAGE_LIMITS_URL (empty disables, for air-gapped installs), VLM_IMAGE_LIMITS_REFRESH_SECONDS, VLM_IMAGE_LIMITS_FILE and VLM_IMAGE_LIMITS_ENFORCE are listed in the backend and workers sample.env. O2: a remote table that drops a provider the bundled table has is rejected (last good copy kept, reason logged), and resolution never turns a bundled provider into pass-through because the effective table lacks it. O17: a contended test drives the real background thread. O22: local_max_input_tokens no longer claims to read a "local" registry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Make the token term warn-only, stream advisories and name the right model (O3, O4, O10, O12, O13, O16, O18, O20, O24, O25) O4 (user decision): the context-window term never rejects, whether its tokens-per-page figure is the default or a model's own entry; it is reported per LLM as an advisory. Count and byte limits stay enforced. O12: ImageModePolicy.with_advisory_sink / emit_advisories send advisories and the size hint to the user's run log as well as the worker log. O10/O20: each per-image limit keeps the LLM it came from, messages and describe() name that model, and the many-images rule stays paired per model instead of minimising threshold and pixels separately. O13: VLM_IMAGE_LIMITS_ENFORCE=false selects the fixed 90-page / 14 MiB cap (fixed_cap_bound_kwargs) instead of model-aware limits. O18: an invalid ceiling env value logs ERROR and keeps the legacy default instead of removing the ceiling. O3/O16/O24/O25: strictest-of-many tests run over both LLM orders plus a two-LLM per-image test; a test pins the legacy clamp to the env overrides; tests no longer read ambient env; the obsolete 100-image cap test is gone. The image-mode summary gains a page-count parser (used by the Prompt Studio cache-hit check). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Re-check cached image extractions, add the enforce switch and say when a retry helps (O3, O7, O12, O13, O19, O21) O7: a Prompt Studio extraction cache hit in image mode is re-checked against the current profile LLM's page budget, using the page count in the stored image-mode summary (the cache key ignores the LLM). O13: the upload and cache-hit checks and the executor honour VLM_IMAGE_LIMITS_ENFORCE=false (fixed cap, no LLM lookup). O12: the executor attaches a run-log sink, so advisories reach the user at WARN. O19: IMAGE_LIMITS_UNRESOLVED says whether retrying can help (lookup outage vs a broken adapter config); only the message reaches the user. O21: extraction limit errors use the answer bridge's "CODE: message" shape; codes.py says consumers must find the code anywhere in the text. O3: the executor strictest-LLM test runs over both orders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Fix the Claude limit data, merge override defaults per field and state the lookup rule once (O5, O6, O8, O14, O15, O23) O5: the 200k-context Claude regex accepts '-' or '.' between version digits (OpenRouter's claude-sonnet-4.5) on anthropic, vertex_ai and azure_ai, and bedrock_mantle uses the same 200k rule instead of a Haiku-only override. O6: Claude 4.7+ (Anthropic's high-resolution tier) carries tokens_per_image 2,850 on anthropic, vertex_ai, azure_ai and bedrock_mantle, derived from the documented 28 px patch formula (unverified; feeds the warn-only advisory). O8: an override file's default block is merged field by field over the shipped default and may not null a defaultable field. O14/O15: the README explains LEGACY_ANSWER_BOUNDS_ACTIVE, the warn-only token term, the enforce switch and the background refresh, and states the override-matching rule once; resolve.py and limits.json _meta point to it. O23: _meta no longer cites a verification.md that does not exist. Data-only limits.json change: schema_version and schema keys unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Test the redirect setting through the real client factory (O1) The redirect test replaced the whole client factory with a mock client, so it passed with follow_redirects=True too (mutation check). It now keeps the real factory and swaps only the transport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Lock remote kinds, cap remote raises at 2x and honour the enforce switch inside sdk1 (O2, O13, N4, O19) - O2: a remote limits table is rejected (last good copy kept, reasons logged) when a bundled provider changes kind, or an enforced image-count / request-byte limit rises above 2x the bundled value, in a provider's defaults, a same-pattern override, the default block, or for any bundled example model as it resolves (catches shadowing overrides, removed overrides and retargeted inherits). Lowering and additions are accepted; the operator override file is not subject to the rule. - O13: VLM_IMAGE_LIMITS_ENFORCE=false is honoured inside ImageModePolicy.from_limits, compute_page_budget and extraction_bound_kwargs (new ImageModePolicy.fixed_cap), so the cloud answer plugin and every other consumer get the fixed 90-page / 14 MiB cap. Call-site checks stay: they also skip the LLM lookup. - N4: advisories report every LLM's advisory image count and request size, not only the binding LLM's. - O19: only a connection error, timeout or 5xx from the adapter lookup is reported as retryable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Clamp remote image limits at lookup time instead of rejecting the table (O2, P3, P4) O2: a remote limits.json is no longer judged as a whole. Every lookup on a remote table also resolves the same inputs against the bundled table: the bundled kind wins (advisory flags follow it) and max_images_per_request / max_request_bytes are min(remote, 2x bundled), compared as raw bytes. This catches a new override shadowing any model, not just the bundled examples, and an over-raised value no longer freezes an install out of every other fix in the file. Fields and entries from the operator override file are not clamped (per-field layers now say which entry supplied each field). Each clamp logs one WARNING per provider/model/field and is named in the limit's provenance. The table-level loosening and dropped-provider rejections are removed; a missing provider resolves from the bundled entry. P3: the Prompt Studio cache-hit check applies the fixed page cap when VLM_IMAGE_LIMITS_ENFORCE=false, as the upload check does. P4: from_limits logs the switched-off warning once per process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Keep operator override files valid when the remote table drops a provider, and log the bundled fallback once (Q1, Q2) Q1: an override file whose entry inherits from a provider (or cites a source) that a remote copy dropped no longer fails validation and is no longer ignored. apply_override_file validates inherit targets and source keys against the base table plus the bundled one, and the lookup resolves such an inherit parent from the bundled entry, as a missing provider already does. Operator fields stay unclamped; inherited fields keep the shipped layer and the remote clamp. Q2: the "provider missing from the remote table" WARNING is logged once per provider per process (reset by clear_limits_cache), as is a bad VLM_IMAGE_LIMITS_REFRESH_SECONDS value, which is read on every refreshing lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Log an invalid VLM_IMAGE_LIMITS_ENFORCE value once per process (R1) The switch is read on every lookup, so a bad value logged an ERROR per page set and per profile render. Each distinct bad value is now reported once, like the switched-off warning above it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [MISC] Simplify the image-limits changes without behaviour change Drop the unused Confidence type alias, fold the private _bundled_table into the public bundled_table it was wrapped by, and share one once-per-process helper between first_clamp_report and first_missing_provider_report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Keep adapter credentials out of image-limit errors and logs (H1, M2) The adapter-validation error quotes the stored metadata (pydantic input_value=, api_key included). The extraction-time IMAGE_LIMITS_UNRESOLVED message and the Prompt Studio upload log now report only the exception type with a fixed sentence, and the cause chain is dropped so the executor's traceback log cannot carry it either. A non-SdkError while resolving limits is reported as an internal error (logged with its traceback) instead of telling the user to fix their adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Fetch the image-limits table only when a URL is configured (H2) On-prem must never phone home. VLM_IMAGE_LIMITS_URL now defaults to unset: the bundled table is used, nothing is fetched and no refresh warning is logged. SaaS opts in by setting it to PUBLISHED_LIMITS_URL (the raw limits.json on main), which replaces DEFAULT_LIMITS_URL. README, docstrings, both sample.env files and the configuration tests follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Run the workers executor unit tests in CI (H3) No rig lane collected workers/executor/tests, so the image-limit resolution and vlm_image_answer bridge tests never ran in CI. unit-workers now includes the path (junit: 1488 tests, 54 of them from executor.tests, 0 failures). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Re-read the override file on change, advise splitting under the many-images rule, check only prompt LLMs in deployments (M1, M3, M4) M1: the VLM_IMAGE_LIMITS_FILE result was cached per base table for the process life, failure included, so an air-gapped install never saw a fixed file. It is now re-read when its (mtime, size) changes, and a failure is retried on change or after OVERRIDE_RETRY_SECONDS. M3: when the many-images pixel rule applied, the message said splitting does not help; it now advises at most N pages per request. The code restatements point at the one message instead of repeating it. M4: confirmed no image-mode path sends page images to the tool-level LLM (single-pass is rejected, challenge/evaluation skipped, summarize reads text), so deployments resolve limits from the prompt LLMs only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Re-check cached extracts at Index, correct the clamp and limits-table docs (M5, M6, M7) M5: Manage Documents -> Index reused a cached image-mode extract without the cache-hit check prompt runs apply; build_index_payload now runs _check_cached_image_extraction (outside the marker fallback, so the rejection reaches the user). M6: README says a model newly listed under a provider the bundled table has is clamped against the bundled resolution (rule 7); the OpenRouter anthropic/ note now says what it resolves to (32 MB). No data change. M7: stale comments in page_image_loader (fixed cap vs operator ceilings), the limits.json default rationale caveat, fetch_remote_bytes' deadline and a policy test comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Treat empty page/size env vars as unset, share the fixed-cap message, drop the unused adapter resolver (Lows) - VLM_IMAGE_ANSWER_PAGE_CAP / _MAX_TOTAL_MB: an empty value means unset (no ERROR, no forced 90 cap); a bad value is logged once per value. - fixed_cap_message in policy.py replaces four copies of the fixed-cap text (page loader, LLMWhisperer pre-check, PS cache hit and upload); the wording is unchanged. - resolve_image_limits_for_adapter had no production caller: deleted; its tests now go through LLM.image_input_limits_for. - The PS cache-hit ENFORCE=false test now asserts the fixed cap rejects 91 pages, and is named for what it checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [MISC] Require an owner's review for the image limits table limits.json is runtime data for deployments that set VLM_IMAGE_LIMITS_URL (SaaS): a merge to main changes which image-mode documents they accept within the refresh window, without a release. Add a CODEOWNERS entry so such changes request review from the maintainers and the data's author. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] PR review toolkit pass: clamp the per-image limits, skip the limits lookup for highlight, pin the cache-hit summary format - A remote limits table may loosen max_image_bytes, max_image_px, many_images_threshold and many_images_max_px only to 2x the bundled value, like the image count and request bytes (byte limits compared as raw bytes). A raised or removed many_images_threshold counts as loosening. Tightening stays unbounded. - The executor resolves the image policy only off the highlight path, so highlight in image mode reports the adapter's own error instead of a limits-lookup failure, with no platform call. - The structure tool sends the LLM ids under IndexingConstants.LLM_INSTANCE_IDS. - Tests pin today's image-mode summary text and its cache-hit rejection. - Comment fixes: limits.json size, clamped_from meaning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [FIX] Assert the source URL's host, not a prefix (CodeQL py/incomplete-url-substring-sanitization) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * UN-4427 [DOCS] README: the remote clamp also covers per-image limits The maintainer checklist still said per-image limits are not clamped, but GUARDED_FIELDS includes the per-image byte and pixel limits and the many-images rule. Name the fields that really are unclamped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 21c07bb commit 55e246b

43 files changed

Lines changed: 10737 additions & 121 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/CODEOWNERS‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# CODEOWNERS
2+
#
3+
# The vision-LLM image limits table is runtime data: when a deployment sets
4+
# VLM_IMAGE_LIMITS_URL (SaaS does), a merge to main changes which image-mode
5+
# documents every such deployment accepts within the refresh window, without a
6+
# release. Changes to it need an owner's review. See
7+
# unstract/sdk1/src/unstract/sdk1/utils/image_limits/README.md.
8+
/unstract/sdk1/src/unstract/sdk1/utils/image_limits/limits.json @Zipstack/unstract-maintainers @praveen-formido

‎backend/conftest.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,15 @@
1414
testenv); this file only takes effect in local/IDE runs.
1515
"""
1616

17+
import os
1718
from pathlib import Path
1819

1920
import pytest
2021
from dotenv import load_dotenv
2122

23+
# Keep image-limits lookups on the bundled table (no fetch from GitHub).
24+
os.environ.setdefault("VLM_IMAGE_LIMITS_URL", "")
25+
2226
# `-s` is set in pyproject so prints surface — log when test.env is absent
2327
# to make a mis-located file debuggable instead of silently empty.
2428
if not load_dotenv(Path(__file__).parent / "test.env", override=False):

‎backend/prompt_studio/prompt_studio_core_v2/constants.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,9 @@ class IndexingConstants:
169169
EMBEDDING_INSTANCE_ID = "embedding_instance_id"
170170
VECTOR_DB_INSTANCE_ID = "vector_db_instance_id"
171171
X2TEXT_INSTANCE_ID = "x2text_instance_id"
172+
# LLM adapter instance ids that will read an image-mode extraction; the
173+
# executor checks their image-request limits before the billed conversion.
174+
LLM_INSTANCE_IDS = "llm_instance_ids"
172175
FILE_PATH = "file_path"
173176
CHUNK_SIZE = "chunk_size"
174177
CHUNK_OVERLAP = "chunk_overlap"

‎backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py‎

Lines changed: 151 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,17 @@
9191
from unstract.sdk1.execution.context import ExecutionContext
9292
from unstract.sdk1.file_storage.constants import StorageType
9393
from unstract.sdk1.file_storage.env_helper import EnvHelper
94+
from unstract.sdk1.llm import LLM
95+
from unstract.sdk1.utils.image_limits import (
96+
codes as image_limit_codes,
97+
)
98+
from unstract.sdk1.utils.image_limits import (
99+
compute_page_budget,
100+
extraction_bound_kwargs,
101+
fixed_cap_message,
102+
model_aware_limits_enabled,
103+
page_cap_message,
104+
)
94105
from unstract.sdk1.utils.indexing import IndexingUtils
95106
from unstract.sdk1.utils.tool import ToolUtils
96107

@@ -636,6 +647,13 @@ def build_index_payload(
636647
"check_extraction_status raised; falling back to full extraction",
637648
exc_info=True,
638649
)
650+
if reused_extracted_text:
651+
# The cache key ignores the LLM, so a cached image-mode extraction
652+
# made for another LLM must still fit this profile's (outside the
653+
# fallback above: a rejection must reach the user).
654+
PromptStudioHelper._check_cached_image_extraction(
655+
default_profile, reused_extracted_text, filename
656+
)
639657

640658
util = PromptIdeBaseTool(log_level=LogLevel.INFO, org_id=org_id)
641659
doc_id_key = IndexingUtils.generate_index_key(
@@ -654,6 +672,9 @@ def build_index_payload(
654672

655673
extract_params = {
656674
IKeys.X2TEXT_INSTANCE_ID: str(default_profile.x2text.id),
675+
IKeys.LLM_INSTANCE_IDS: PromptStudioHelper._image_limit_llm_ids(
676+
default_profile
677+
),
657678
IKeys.FILE_PATH: file_path,
658679
IKeys.ENABLE_HIGHLIGHT: tool.enable_highlight,
659680
IKeys.OUTPUT_FILE_PATH: extract_file_path,
@@ -1480,34 +1501,139 @@ def uploads_use_image_output_mode(profile_manager: ProfileManager | None) -> boo
14801501
return False
14811502

14821503
@staticmethod
1483-
def validate_upload_page_count_for_image_mode(file_data: Any, file_type: str) -> None:
1504+
def _image_limit_llm_ids(profile_manager: ProfileManager | None) -> list[str]:
1505+
"""The LLM ids whose image limits apply to this profile's extraction."""
1506+
llm = getattr(profile_manager, "llm", None)
1507+
llm_id = getattr(llm, "id", None)
1508+
return [str(llm_id)] if llm_id else []
1509+
1510+
@staticmethod
1511+
def _upload_page_budget(profile_manager: ProfileManager | None) -> Any:
1512+
"""The profile LLM's page budget, or None to use the fixed cap.
1513+
1514+
Resolution is local (the adapter's own validation) and never waits on
1515+
the network: the limits table is served from memory, and a due
1516+
refresh runs in the background. Any failure, or
1517+
``VLM_IMAGE_LIMITS_ENFORCE=false``, falls back to the fixed cap rather
1518+
than blocking the upload: extraction re-checks against the LLM before
1519+
anything is converted.
1520+
"""
1521+
llm = getattr(profile_manager, "llm", None)
1522+
if llm is None or not model_aware_limits_enabled():
1523+
return None
1524+
try:
1525+
limits, tokens = LLM.image_input_limits_for(
1526+
str(llm.adapter_id), llm.metadata or {}
1527+
)
1528+
except Exception as e:
1529+
# The error type only, no traceback: the adapter's validation
1530+
# error quotes the stored metadata, credentials included.
1531+
logger.warning(
1532+
"Could not resolve image limits for the default profile's LLM "
1533+
"(%s); using the fixed page cap for the upload check",
1534+
type(e).__name__,
1535+
)
1536+
return None
1537+
bounds = extraction_bound_kwargs()
1538+
return compute_page_budget(
1539+
limits,
1540+
context_tokens=tokens,
1541+
page_ceiling=bounds["page_ceiling"],
1542+
legacy_cap=bounds["legacy_cap"],
1543+
)
1544+
1545+
@staticmethod
1546+
def _check_cached_image_extraction(
1547+
profile_manager: ProfileManager, extracted_text: str, filename: str
1548+
) -> None:
1549+
"""Re-check a cached image-mode extraction against the profile's LLM.
1550+
1551+
The extraction cache is keyed on the document, profile and x2text
1552+
config, not the LLM. After a profile switches LLM (e.g. Gemini to
1553+
Bedrock Claude, 20 images) a cache hit would skip the page-budget
1554+
check and the prompt would fail at the provider. Uses the page count
1555+
recorded in the image-mode summary; no-op outside image mode or when
1556+
the count or the LLM's limits are unknown (answer time re-checks).
1557+
With ``VLM_IMAGE_LIMITS_ENFORCE=false`` the fixed page cap applies,
1558+
as at upload.
1559+
"""
1560+
if not PromptStudioHelper.uploads_use_image_output_mode(profile_manager):
1561+
return
1562+
page_count = LLMWhispererHelper.page_count_from_image_output_summary(
1563+
extracted_text
1564+
)
1565+
if page_count is None:
1566+
return
1567+
if not model_aware_limits_enabled():
1568+
page_cap = configured_page_cap()
1569+
if page_count <= page_cap:
1570+
return
1571+
raise ExtractionAPIError(
1572+
f"Failed to extract '{filename}'. "
1573+
f"{image_limit_codes.IMAGE_PAGE_CAP_EXCEEDED}: "
1574+
+ fixed_cap_message(
1575+
page_count,
1576+
page_cap,
1577+
switch_to="switch this profile to a text output mode",
1578+
)
1579+
)
1580+
budget = PromptStudioHelper._upload_page_budget(profile_manager)
1581+
if budget is None or page_count <= budget.max_pages:
1582+
return
1583+
raise ExtractionAPIError(
1584+
f"Failed to extract '{filename}'. "
1585+
f"{image_limit_codes.IMAGE_PAGE_CAP_EXCEEDED}: "
1586+
+ page_cap_message(page_count, budget, charged=None)
1587+
+ " The document was extracted earlier for a different LLM; "
1588+
"switch this profile back, or use one whose LLM accepts more pages."
1589+
)
1590+
1591+
@staticmethod
1592+
def validate_upload_page_count_for_image_mode(
1593+
file_data: Any, file_type: str, profile_manager: ProfileManager | None = None
1594+
) -> None:
14841595
"""Reject, at upload, a PDF with more pages than image mode can answer.
14851596
14861597
Call only when ``uploads_use_image_output_mode`` is True. Image mode
14871598
sends every page to the LLM in one request, so a document over the
1488-
page cap can never be answered — and converting it would be billed.
1489-
Catching it at upload means the user never builds prompts around a
1490-
document that will fail at indexing.
1491-
1492-
Uses the SDK's own page counter and cap, so a document accepted here
1493-
also passes the extraction-time pre-check. The file is read in place,
1494-
not copied into memory. A PDF whose page count can't be read is let
1495-
through; extraction re-checks it.
1599+
LLM's page budget can never be answered — and converting it would be
1600+
billed. Catching it at upload means the user never builds prompts
1601+
around a document that will fail at indexing.
1602+
1603+
``profile_manager`` is the project's default profile: its LLM's
1604+
image-request limits set the budget (the same computation the
1605+
extraction-time pre-check runs). Without a resolvable LLM the fixed
1606+
cap applies. The file is read in place, not copied into memory. A
1607+
PDF whose page count can't be read is let through; extraction
1608+
re-checks it.
14961609
"""
14971610
if file_type != "application/pdf":
14981611
return
1499-
page_cap = configured_page_cap()
15001612
page_count = LLMWhispererHelper.pdf_page_count(file_data)
15011613
file_data.seek(0)
1502-
if page_count is None or page_count <= page_cap:
1614+
if page_count is None:
1615+
return
1616+
budget = PromptStudioHelper._upload_page_budget(profile_manager)
1617+
if budget is not None:
1618+
if page_count <= budget.max_pages:
1619+
return
1620+
profile_name = getattr(profile_manager, "profile_name", "") or "default"
1621+
raise ImageModePageLimitExceeded(
1622+
detail=(
1623+
page_cap_message(page_count, budget, charged=None)
1624+
+ " This check uses the project's default profile "
1625+
f"('{profile_name}'); other profiles may allow a different "
1626+
"number of pages."
1627+
)
1628+
)
1629+
page_cap = configured_page_cap()
1630+
if page_count <= page_cap:
15031631
return
15041632
raise ImageModePageLimitExceeded(
1505-
detail=(
1506-
"This document has too many pages for image output mode. It "
1507-
f"has {page_count} pages, and the limit is {page_cap} because "
1508-
"every page is sent to the LLM in one request. Split the "
1509-
"document into smaller files, or switch the project's default "
1510-
"profile to a text output mode."
1633+
detail=fixed_cap_message(
1634+
page_count,
1635+
page_cap,
1636+
switch_to="switch the project's default profile to a text output mode",
15111637
)
15121638
)
15131639

@@ -2703,6 +2829,11 @@ def dynamic_extractor(
27032829
try:
27042830
extracted_text = fs_instance.read(path=extract_file_path, mode="r")
27052831
logger.info("Extracted text found. Reading from file..")
2832+
# The cache key ignores the LLM, so a cached image-mode
2833+
# extraction made for another LLM must still fit this one.
2834+
PromptStudioHelper._check_cached_image_extraction(
2835+
profile_manager, extracted_text, filename
2836+
)
27062837
return extracted_text
27072838
except FileNotFoundError as e:
27082839
logger.warning(
@@ -2712,6 +2843,9 @@ def dynamic_extractor(
27122843
extracted_text = None
27132844
payload = {
27142845
IKeys.X2TEXT_INSTANCE_ID: x2text,
2846+
IKeys.LLM_INSTANCE_IDS: PromptStudioHelper._image_limit_llm_ids(
2847+
profile_manager
2848+
),
27152849
IKeys.FILE_PATH: file_path,
27162850
IKeys.ENABLE_HIGHLIGHT: enable_highlight,
27172851
IKeys.USAGE_KWARGS: usage_kwargs.copy(),

‎backend/prompt_studio/prompt_studio_core_v2/tests/test_build_index_payload.py‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@
2424
from typing import Any
2525
from unittest.mock import MagicMock, patch
2626

27+
import pytest
28+
2729
from prompt_studio.prompt_studio_core_v2 import prompt_studio_helper as _psh_mod
2830

2931
PromptStudioHelper = _psh_mod.PromptStudioHelper
@@ -194,6 +196,60 @@ def test_check_extraction_status_raises_is_swallowed(self, caplog) -> None:
194196
)
195197

196198

199+
_LLMW_ADAPTER_ID = "llmwhisperer|a5e6b8af-3e1f-4a80-b006-d017e8e67f93"
200+
_BEDROCK_ID = "bedrock|8d18571f-5e96-4505-bd28-ad0379c64064"
201+
_BEDROCK_META = {
202+
"model": "anthropic.claude-sonnet-4-5-20250929-v1:0",
203+
"aws_region_name": "us-east-1",
204+
}
205+
206+
207+
def _image_mode_bedrock_profile() -> MagicMock:
208+
profile = _make_profile()
209+
profile.x2text.adapter_id = _LLMW_ADAPTER_ID
210+
profile.x2text.metadata = {"output_mode": "image"}
211+
profile.llm.adapter_id = _BEDROCK_ID
212+
profile.llm.metadata = _BEDROCK_META
213+
return profile
214+
215+
216+
def _cached_summary(pages: int) -> str:
217+
return _psh_mod.LLMWhispererHelper.build_image_output_summary([None] * pages)
218+
219+
220+
class TestCacheHitImageLimits:
221+
"""Index reuses a cached extract only if it fits the profile's LLM (M5).
222+
223+
The extraction cache ignores the LLM: a document extracted for a
224+
generous LLM must not be indexed for one that cannot answer it.
225+
"""
226+
227+
@pytest.fixture(autouse=True)
228+
def _no_env_cap(self, monkeypatch: pytest.MonkeyPatch) -> None:
229+
monkeypatch.delenv("VLM_IMAGE_ANSWER_PAGE_CAP", raising=False)
230+
monkeypatch.delenv("VLM_IMAGE_LIMITS_ENFORCE", raising=False)
231+
232+
def test_cached_extract_over_the_llm_budget_is_rejected(self) -> None:
233+
with pytest.raises(_psh_mod.ExtractionAPIError) as excinfo:
234+
_dispatch_build(
235+
check_return=True,
236+
read_return=_cached_summary(21),
237+
profile=_image_mode_bedrock_profile(),
238+
)
239+
detail = str(excinfo.value.detail)
240+
assert "IMAGE_PAGE_CAP_EXCEEDED" in detail
241+
assert "doc.pdf" in detail and "20 page images" in detail
242+
243+
def test_cached_extract_within_the_budget_is_reused(self) -> None:
244+
context, _cb, _fs, _check = _dispatch_build(
245+
check_return=True,
246+
read_return=_cached_summary(20),
247+
profile=_image_mode_bedrock_profile(),
248+
)
249+
index_params = context.executor_params["index_params"]
250+
assert index_params[IKeys.EXTRACTED_TEXT] == _cached_summary(20)
251+
252+
197253
class TestOwnerAccessPlumbing:
198254
"""UN-3739: the REQUESTER must reach the owner-access check.
199255

0 commit comments

Comments
 (0)