Skip to content

fix: decode base64 embeddings as little-endian floats - #3711

Open
HostX0 wants to merge 1 commit into
openai:mainfrom
HostX0:fix/embedding-base64-byte-order
Open

fix: decode base64 embeddings as little-endian floats#3711
HostX0 wants to merge 1 commit into
openai:mainfrom
HostX0:fix/embedding-base64-byte-order

Conversation

@HostX0

@HostX0 HostX0 commented Aug 21, 2026

Copy link
Copy Markdown
  • I understand that this repository is auto-generated and my pull request may not be merged

Root cause

The base64 embedding representation contains little-endian float32 bytes, but the handwritten response parser used native byte order in both its NumPy and stdlib paths. On big-endian Python platforms, that silently produces incorrect vector values. The existing fixture was also native-endian, so it followed the test host instead of the wire format and could not expose the defect.

Fix

  • Decode the NumPy path with the explicit <f4 dtype.
  • Byte-swap the stdlib array("f") path on big-endian hosts.
  • Construct fixture bytes with struct.pack("<3f", ...) so the test represents the wire format on every host.
  • Cover the stdlib big-endian branch without requiring big-endian CI hardware.

The branch is rebased onto current main and preserves #3757: NumPy availability is still checked once per response, while each encoded vector is decoded only once.

Validation

Before the rebase, the focused embeddings suite passed under both Pydantic environments (58 tests each), along with targeted Ruff, Pyright, and Mypy checks. After rebasing onto #3757, I re-ran syntax checks and an isolated execution harness against the actual parser module covering:

  • stdlib little-endian decoding for multiple vectors
  • simulated stdlib big-endian byte swapping
  • NumPy decoding with the explicit <f4 dtype
  • one NumPy availability check per response

The full current suite is left to GitHub Actions because this local macOS workspace does not have the repository's locked uv development environment installed.

@HostX0
HostX0 marked this pull request as ready for review August 21, 2026 13:38
@HostX0
HostX0 requested a review from a team as a code owner August 21, 2026 13:38
@HostX0
HostX0 force-pushed the fix/embedding-base64-byte-order branch from 6ad2fec to ac63c61 Compare August 29, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant