We're using TensorZero with Vertex AI's OpenAI-compatible endpoint (type = "openai" provider pointing at Vertex AI's /openapi path). Everything works great for text and images, but sending PDFs breaks.

What happens

When we send a PDF as image_url with a data:application/pdf;base64,... data URI, TZ's OpenAI provider detects it's not an image, resolves it, and re-serializes it as a type: "file" content block.

The problem is that Vertex AI's OpenAI-compatible endpoint doesn't support type: "file":

Unrecognized 'type' field in an object element of an array 'content' field; found: 'file'.

What we tested

We tested every format we could find, both directly against Vertex AI and through TZ:

The only format Vertex AI accepts for PDFs is data:application/pdf;base64,... inside image_url blocks — and TZ converts that to type: "file" before it reaches Vertex AI.

Community confirmation that this format works: https://discuss.ai.google.dev/t/openai-compatibility-for-pdf-file/77388

Why we use type = "openai" for Vertex AI

Vertex AI's OpenAI-compatible endpoint handles response_format: json_schema with complex nested schemas better than the native Gemini API. We rely on this for structured extraction with Instructor, so gcp_vertex_gemini isn't an option for us right now.

Idea

Would it be possible to add a per-provider option that keeps non-image files as image_url data URIs instead of converting them to type: "file"? Something like:

[models.my_model.providers.vertex_openai_compat]
type = "openai"
api_base = "https://aiplatform.googleapis.com/..."
non_image_file_behavior = "image_url"  # default: "file"

The change would be in prepare_file_message() — in the catch-all branch that currently creates OpenAIContentBlock::File, it would instead create OpenAIContentBlock::ImageUrl with the data:{mime};base64,... URL when this option is set.

Related

• Support 'application/pdf' file input and add support to OpenAI provider #2285 added type: "file" to the OpenAI provider — works for real OpenAI, not Vertex AI
• Allow Google AI Studio Gemini to accept PDF uploads #3854 fixed PDF support for the gcp_vertex_gemini provider using InlineData — works great

let me know if this makes sense. I can raise a PR for this if you are open to this approach or happy to learn about your opinions on this.