UN-4427 [FEAT] Image output mode: per-model image limits instead of the 90-page / 14 MB caps by praveen-formido · Pull Request #2341 · Zipstack/unstract

praveen-formido · 2026-10-14T17:30:15Z

What

Replace image output mode's global 90-page cap and 14 MiB page-image budget with limits looked up for the LLM actually used, from a maintained table of vision-LLM limits (image count, request size, per-image size/pixels, context tokens), keyed by provider with model overrides.
Check those limits as early as possible: at Prompt Studio upload, before LLMWhisperer conversion (nothing charged), and after download before anything is stored.
Fetch the limits table from GitHub (limits.json on main) so data fixes reach installs without an Unstract release; the copy bundled in sdk1 is the fallback.

Why

Real provider limits differ widely (Bedrock 20 images per request, Azure OpenAI 50, Anthropic 100/600 and 32 MB, OpenAI 1,500, Gemini 3,600), so Bedrock and Azure users got raw provider errors on documents well under 90 pages, and generous providers were held back at 90.
litellm's registry has no image-count or request-size fields, so Unstract maintains the data. Decided with Arun, Shuveb, Johny and Hari on 2026-10-09 (no windowing; provider-categorised limits; clear errors instead of hard caps).

How

sdk1 (unstract/sdk1/src/unstract/sdk1/utils/image_limits/)

limits.json: providers with defaults and ordered model_overrides; every number carries a source URL, verified-on date and confidence. Verified against official provider docs on 2026-10-09 (57 fields verified, 30 marked unverified, 44 unknown left null). Strict loader: unknown keys are rejected, regexes compiled.
Lookup keyed by the adapter's provider, with Bedrock Mantle routed by its bedrock_mantle/ prefix; never by the litellm prefix alone (MiniMax can send anthropic/<id>). Azure model overrides are best-effort (deployment names); Anthropic's 100-vs-600 split comes from our own list, not litellm's context window.
Page budget = lowest of image-count limit, context-token estimate, LLMWhisperer's 999 pages, and an optional operator ceiling. Byte budget accounts for base64 where the provider counts encoded bytes. Per-page byte and pixel checks.
Remote refresh: fetched from https://raw.githubusercontent.com/Zipstack/unstract/main/unstract/sdk1/src/unstract/sdk1/utils/image_limits/limits.json (env VLM_IMAGE_LIMITS_URL; empty disables), short timeout, cached per process and refreshed about hourly (VLM_IMAGE_LIMITS_REFRESH_SECONDS). Same strict validation plus a schema_version check; a bad or incompatible file is rejected and the last good copy stays in use. Precedence: VLM_IMAGE_LIMITS_FILE override → remote → bundled.

Workers and backend

Extraction now receives the LLM adapter instance id(s) (llm_instance_ids) from Prompt Studio indexing, Prompt Studio prompt runs and deployments (tool LLM plus every prompt LLM; the strictest wins). The executor resolves limits before LLMWhisperer converts (and bills).
If limits can't be resolved for given LLM ids, extraction stops before conversion with IMAGE_LIMITS_UNRESOLVED (nothing charged, retryable).
New IMAGE_PAGE_TOO_LARGE for a single page over a per-image limit. Extraction errors keep their code as a CODE: message prefix.
The Prompt Studio upload check uses the default profile's LLM limits and names that profile.

Rollout safety

The cloud vlm-image-answer plugin still answers with the fixed 90 / 14 MiB until its PR lands, so LEGACY_ANSWER_BOUNDS_ACTIVE keeps every check here within those values: this PR can only make limits stricter. Strict providers now fail before billing; generous providers stay at 90. A follow-up OSS PR turns that off and removes the old constants once the cloud change ships.

Defaults chosen for open questions (each behind one constant)

Question	Default	Where
Unknown / self-hosted models	20 images / 20 MB; advisory (logged) on pass-through providers, enforced on hosted ones	`limits.json` `default`; `PASS_THROUGH_DEFAULTS_ARE_ADVISORY`
LLM unknown at extraction	Only the 999 and operator ceilings; the answer step re-checks	`ImageModePolicy.ceiling_only`
Limits can't be resolved	Fail closed before conversion	`IMAGE_LIMITS_UNRESOLVED`
Context-token check	Hard check; 4,000-token reserve; 1,500 tokens/page	`ENFORCE_CONTEXT_TOKEN_TERM`, `PROMPT_TOKENS_RESERVE`, `tokens_per_image`
Per-image byte/pixel limits	Reject (no downscaling)	`ENFORCE_PER_IMAGE_LIMITS`
Size estimate before conversion	Warning only, at 250 KB/page	`ASSUMED_PAGE_BYTES`
Upload check scope	Default profile only	`validate_upload_page_count_for_image_mode`
Azure legacy 10-image deployments	Not handled (50 for all)	`limits.json`
Cost gate	None	—
OpenRouter	Inherit the family's direct-API limits	`limits.json` `inherit`
Limits hosting	Remote from `main`, hourly, bundled fallback	`VLM_IMAGE_LIMITS_URL`, `VLM_IMAGE_LIMITS_REFRESH_SECONDS`, `VLM_IMAGE_LIMITS_FILE`

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Text output modes: unaffected; no extra platform call, no change in behaviour.
Image output mode: limits can only get stricter while LEGACY_ANSWER_BOUNDS_ACTIVE is on. Documents on strict providers (Bedrock, Azure, Mistral, Nova 2 Lite) that previously reached the provider and failed there now fail earlier with a clear message, before conversion where possible.
New failure mode: IMAGE_LIMITS_UNRESOLVED if the platform service can't return the LLM adapter config at extraction time; it happens before conversion (nothing charged) and is retryable.
Network: each process makes one GitHub request per refresh window; failures fall back to the bundled copy. Air-gapped installs should set VLM_IMAGE_LIMITS_URL="" to avoid an hourly failed attempt (up to 3 s, never blocking image mode).
Schema changes to limits.json: an older install rejects a newer incompatible file and stays on its bundled copy (no crash). Changes that add fields need a schema_version bump (see the package README).

Database Migrations

None.

Env Config

Optional: VLM_IMAGE_LIMITS_URL (default: the raw GitHub URL above; empty disables remote fetch), VLM_IMAGE_LIMITS_REFRESH_SECONDS (default 3600), VLM_IMAGE_LIMITS_FILE (operator override file, same schema).
VLM_IMAGE_ANSWER_PAGE_CAP / VLM_IMAGE_ANSWER_MAX_TOTAL_MB keep their current meaning in this PR; they become optional ceilings in the follow-up.

Relevant Docs

Plan: Image output mode: model-aware image limits — Implementation Plan
Maintenance procedure: unstract/sdk1/src/unstract/sdk1/utils/image_limits/README.md

Related Issues or PRs

UN-4427 (this work), UN-2646 (image output mode), UN-2646 [FIX] Image mode: fail oversized documents early (upload and indexing); fix re-extraction of names with [ ] #2324 (earlier image-mode fixes).
Follow-ups: cloud vlm-image-answer plugin + backend hooks use these limits; OSS cleanup of the old constants; LW-460 (LLMWhisperer stops at a caller-supplied size limit without billing).

Dependencies Versions

None (litellm stays at 1.104.0).

Notes on Testing

sdk1: 1,012 passed (new: test_image_limits_{table,resolve,policy,remote}.py, test_llm_image_limits.py, plus loader and helper additions). Includes a test that lookup does not depend on litellm's cost map, and remote-fetch tests with no real network (success, timeout, HTTP error, invalid JSON, schema-invalid, incompatible schema_version, bad regex, refresh window, override precedence, disabled URL).
python3 -m tests.rig run unit-backend: 1,229 passed. unit-workers: 1,433 passed.
Cloud compatibility: ran the cloud rig group unit-vlm-image-answer against this branch on a merged OSS + cloud tree: 25 passed.
Not yet tested live on a deployed stack.

Screenshots

n/a

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

…p API Image output mode sends every page image to a vision LLM in one request, gated so far by a global 90-page cap and a 14 MiB byte budget that match no provider (Bedrock 20 images, Azure 50, Anthropic 100/600, OpenAI 1,500; 20-100 MB request limits). This adds the data and the API the extraction and answer paths need to check against the limits of the LLM actually used. - utils/image_limits/limits.json: provider-keyed limits with per-field provenance (source URL, quote, verified_on, verified/unverified), built from the independently verified dataset; strict loader (_schema.py) that rejects unknown keys, unresolved sources, bad regexes and inherit targets; optional operator override file via VLM_IMAGE_LIMITS_FILE (per-provider replacement, no deep merge). - resolve.py: provider key = adapter get_provider() with the Bedrock Mantle exception; first-match overrides on the bare model, then cost_model, then extra ids; OpenRouter inherits the family's limits; resolution never reads litellm.model_cost, so the remote and bundled cost maps agree. Default count/byte values are advisory on pass-through providers. - policy.py: page budget = min(image count, context tokens, LLMWhisperer 999, operator ceilings) with the binding term recorded; raw-byte budget from the provider's wire limit; per-image byte/pixel checks including the Anthropic >20-images rule; plain-language messages that name the model, provider, limit and provenance and say whether pages were charged. - LLM.get_image_request_limits / get_image_input_token_budget / image_input_limits_for; the adapter's context_window is now kept. - get_page_images(image_limits=) and load_page_images(limits=) run the model-aware checks; ImageLimitExceededError carries a stable error code. Without a policy both keep today's fixed limits exactly, so this change is behaviour-neutral until callers pass one. Chosen defaults (each one constant): PROMPT_TOKENS_RESERVE=4000, context token term enforced, per-image limits enforced, TEXT_RESERVE_BYTES=64 KiB, ASSUMED_PAGE_BYTES=250 KB for a pre-conversion size advisory (never a rejection), PASS_THROUGH_DEFAULTS_ARE_ADVISORY=True, env vars read as optional ceilings by the new accessors (legacy accessors unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… limits before conversion Image-mode extraction had no idea which LLM would read the pages, so the only pre-conversion check was the fixed 90-page cap. Every extraction now carries the LLM adapter ids that will consume it and is checked against those models' limits before LLMWhisperer bills the page conversion. - Executor: _handle_extract resolves llm_instance_ids into an ImageModePolicy (strictest of each limit across LLMs) and passes it to the x2text adapter, which runs the model-aware pre-submit page check and the post-download byte / per-image checks. Ids that cannot be resolved fail closed before conversion with IMAGE_LIMITS_UNRESOLVED (503, nothing charged). Image-mode limit errors from extraction keep their code as a "CODE: message" prefix, like the answer-time bridge. - Callers: Prompt Studio index (default profile), dynamic_extractor (the running profile) and the structure tool (tool LLM plus every prompt's LLM) send llm_instance_ids. - Upload: the Prompt Studio upload check uses the default profile's LLM limits (offline resolution) and says which profile it used; an unresolvable LLM falls back to the fixed cap. - Bridge: PageImageTooLargeError maps to the new IMAGE_PAGE_TOO_LARGE; the limit codes now come from sdk1 and are re-exported. - Rollout bound: extraction_bound_kwargs() keeps every check within the fixed 90 pages / 14 MiB that the cloud plugin still answers with (LEGACY_ANSWER_BOUNDS_ACTIVE). Strict providers now fail before billing; generous ones stay at 90 until the cloud plugin switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… ship without a release Installs now read limits.json from the main branch of Zipstack/unstract (VLM_IMAGE_LIMITS_URL; empty disables it for air-gapped installs) and keep the bundled copy as the always-available fallback. The operator override file still wins over both. The fetch is bounded: one attempt per process per refresh window (VLM_IMAGE_LIMITS_REFRESH_SECONDS, default 3600, failures included), a 3-second budget and 1 MB cap per download, and other threads keep serving the last good table while one fetches. The remote file passes the same strict schema as the bundled one, must carry the schema_version this sdk1 reads, and may not contain override patterns over 300 characters. Any failure logs a warning and keeps the last good copy. The table records its origin (bundled/remote, location, verified_on, sha256 prefix). The log names it, and format_limit_source appends the version to user-facing errors and the run-log summary. Test suites set VLM_IMAGE_LIMITS_URL="" so they never reach the network. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

greptile-apps · 2026-10-14T17:37:14Z

via Greptile

Confidence Score: 5/5

[High impact] The PR appears safe to merge; both previous findings are fixed and no new actionable issues remain.

Summary

This PR adds per-model image limits and checks them before paid conversion and before storing page images.

Image output mode looks up limits for each LLM model.
Prompt Studio checks uploads against the default profile’s LLM limits.
Extraction checks the reading LLMs before converting a document.
Image-mode requests check page size and pixels for each model.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Upload or extraction request] --> B[Resolve consuming model limits]
  C[Remote table or bundled fallback] --> B
  B --> D[Check page count]
  D --> E[Paid page conversion]
  E --> F[Check image sizes and dimensions]
  F --> G[Store page images]

_{Reviews (2) · Last reviewed commit: "UN-4427 [MISC] Fix docstring format flag..." · Reviewed by Greptile}

…rom escaping the fallback - The loader enforces one-level inherit: the target must exist, must not be the provider itself, and must not inherit in turn. The same check runs on the merged table after an override, so a cycle across the shipped and override layers is rejected too. The resolver also tracks visited providers, so a lookup can never recurse forever. - apply_override_file checks the shapes it merges (sources, providers) before touching them, and invalid JSON is a schema error; the override fallback also catches unexpected errors, so a malformed override is logged and ignored instead of failing image extraction. Addresses Greptile review on #2341. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sonarqubecloud · 2026-10-14T17:47:26Z

Quality Gate passed

Issues
42 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-10-14T17:54:12Z

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Skipped	Duration (s)
✅	`e2e-api-deployment`	e2e	3	0	0	5.5
✅	`e2e-coowners`	e2e	1	0	0	0.9
✅	`e2e-etl`	e2e	1	0	0	9.6
✅	`e2e-login`	e2e	2	0	0	0.8
✅	`e2e-prompt-studio`	e2e	1	0	0	6.7
✅	`e2e-smoke`	e2e	2	0	0	1.1
✅	`e2e-workflow`	e2e	1	0	0	13.6
✅	`frontend`	unit	625	0	0	14.2
✅	`integration-backend`	integration	638	0	26	59.4
✅	`integration-connectors`	integration	1	0	7	9.3
✅	`integration-workers`	integration	164	0	1	50.4
❌	`ui`	e2e	0	1	0	0.0
✅	`unit-backend`	unit	1577	0	1	38.2
✅	`unit-connectors`	unit	82	0	0	10.1
✅	`unit-core`	unit	281	0	0	2.4
✅	`unit-platform-service`	unit	18	0	0	2.3
✅	`unit-rig`	unit	120	0	0	3.4
✅	`unit-runner`	unit	12	0	0	2.1
✅	`unit-sdk1`	unit	1019	0	0	32.8
✅	`unit-tool-sidecar`	unit	8	0	0	1.2
✅	`unit-workers`	unit	1433	0	1	112.4
	TOTAL		5989	1	36	376.4

Critical paths

⚠️ Critical paths not yet covered

workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

auth-login — covered by e2e-login
adapter-register-llm — covered by integration-backend
workflow-author — covered by integration-backend
co-owner-manage — covered by integration-backend, e2e-coowners
workflow-create-execute — covered by e2e-workflow
api-deployment-provision — covered by integration-backend
api-deployment-auth — covered by integration-backend
api-deployment-run — covered by e2e-api-deployment
mcp-server-auth — covered by integration-backend
mcp-platform-auth — covered by integration-backend
platform-key-whoami — covered by integration-backend
prompt-studio-author — covered by integration-backend
prompt-studio-fetch-response — covered by e2e-prompt-studio
connector-register-test — covered by integration-backend
pipeline-etl-execute — covered by e2e-etl
usage-aggregate-read — covered by integration-backend
usage-token-tracking — covered by e2e-api-deployment
callback-result-delivery — covered by e2e-api-deployment

praveen-formido and others added 3 commits October 14, 2026 22:54

greptile-apps Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread unstract/sdk1/src/unstract/sdk1/utils/image_limits/_schema.py

Comment thread unstract/sdk1/src/unstract/sdk1/utils/image_limits/_schema.py Outdated

praveen-formido and others added 3 commits October 14, 2026 23:13

[pre-commit.ci] auto fixes from pre-commit.com hooks

e081516

for more information, see https://pre-commit.ci

UN-4427 [MISC] Fix docstring format flagged by ruff (D205)

119a2d5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

praveen-formido assigned praveen-formido and chandrasekharan-zipstack Oct 16, 2026

praveen-formido requested review from hari-kuriakose and ritwik-g October 16, 2026 10:16

praveen-formido unassigned chandrasekharan-zipstack Oct 16, 2026

praveen-formido requested a review from chandrasekharan-zipstack October 16, 2026 10:17

Navigation Menu

Repository navigation

UN-4427 [FEAT] Image output mode: per-model image limits instead of the 90-page / 14 MB caps - #2341

UN-4427 [FEAT] Image output mode: per-model image limits instead of the 90-page / 14 MB caps#2341
praveen-formido wants to merge 6 commits into
mainfrom
UN-4427-model-aware-image-limits

praveen-formido commented Oct 14, 2026

Uh oh!

greptile-apps Bot commented Oct 14, 2026 •

edited

Loading

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

praveen-formido commented Oct 14, 2026

What

Why

How

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Database Migrations

Env Config

Relevant Docs

Related Issues or PRs

Dependencies Versions

Notes on Testing

Screenshots

Checklist

Uh oh!

greptile-apps Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Oct 14, 2026

Quality Gate passed

Uh oh!

github-actions Bot commented Oct 14, 2026

Unstract test results

Per-group results

Critical paths

⚠️ Critical paths not yet covered

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Oct 14, 2026 •

edited

Loading