Skip to content

test(coderd/x/chatd/chattest): table the OpenAI fake positive tests and fail on error streams - #30948

Draft
ibetitsmike wants to merge 2 commits into
mainfrom
mike/chattest-openai-fake-positive-tests
Draft

ibetitsmike wants to merge 2 commits into
mainfrom
mike/chattest-openai-fake-positive-tests

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

The six OpenAI fake positive tests in openai_test.go become three table tests, TestOpenAI_StreamingText, TestOpenAI_NonStreamingText and TestOpenAI_ToolCalls, each with a Completions API row and a Responses API row. The rows also check more than the old tests did: streaming rows fail on any error part and assert the exact text deltas, and non-streaming rows assert the returned text. The old Responses streaming test passed on a stream that held only an error, and both non-streaming tests checked only NoError and NotNil.

Why it is safe

  • Every old assertion maps to a row: the exact deltas, one tool execution, the follow-up request after it, NoError and NotNil.
  • I ran 12 mutants of the fake with go test -overlay. Seven pass the old test that targets the mutated behavior and fail the new rows: an error-only Responses stream, an error part after the Completions text, an error instead of response.completed, empty non-streaming text in either format, and doubled or dropped Responses deltas. The other five (an error-only Completions stream, doubled Completions deltas, a truncated OpenAITextChunks, and a dropped tool call in either format) fail both the old tests and the new rows.
  • The fake is unchanged; the stronger checks pass on main.

Review focus

The Responses streaming row now uses the same two-choice text chunks as the Completions row, so both formats share one expected delta list.

Validation

  • go test ./coderd/x/chatd/chattest/ -count=1: ok
  • go vet ./coderd/x/chatd/chattest/: ok
  • golangci-lint run --fix=false ./coderd/x/chatd/chattest/: 0 issues
  • pre-commit hook: passed

Stack

The stack makes the chattest fake tests fail on wrong output, then drops the fakes' write-only state and shares their response-shape check.

  1. test(coderd/x/chatd/chattest): table the OpenAI fake positive tests and fail on error streams #30948 table the OpenAI fake positive tests and fail on error streams (this PR)
  2. test(coderd/x/chatd/chattest): pin the fake response-shape errors #30949 pin the fake response-shape errors
  3. refactor(coderd/x/chatd/chattest): drop write-only fake state and share the shape check #30950 drop write-only fake state and share the shape check

Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.

@ibetitsmike
ibetitsmike added this pull request to stack #30951 October 18, 2026 15:37
@ibetitsmike ibetitsmike changed the title mike/chattest openai fake positive tests test(coderd/x/chatd/chattest): table the OpenAI fake positive tests and fail on error streams Oct 18, 2026
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-18T17:14:20.444582Z 56b7131 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 023da30728

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, posted this comment for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 56b7131b44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant