feat: add integration test for max_output_tokens by gyliu513 · Pull Request #4825 · ogx-ai/ogx

gyliu513 · 2026-02-03T22:38:14Z

What does this PR do?

Related to #4817

Test Plan

(llama-stack) gualiu@gualiu-mac llama-stack % uv run ./scripts/integration-tests.sh \
      --stack-config server:ci-tests \
      --inference-mode record-if-missing \
      --setup gpt \
      --suite responses \
      --pattern 'TestOpenAIResponses'
=== Llama Stack Integration Test Runner ===
Stack Config: server:ci-tests
Setup: gpt
Inference Mode: record-if-missing
Test Suite: responses
Test Subdirs:
Test Pattern: TestOpenAIResponses

Checking llama packages
llama-stack                              0.2.2.dev1723+g3ec57a7f6 /Users/gualiu/go/src/github.com/llamastack/llama-stack
llama-stack-api                          0.2.2.dev1723+g3ec57a7f6 /Users/gualiu/go/src/github.com/llamastack/llama-stack/src/llama_stack_api
llama-stack-client                       0.4.4
ollama                                   0.6.1
=== Applying Setup Environment Variables ===
Setting SQLITE_STORE_DIR: /var/folders/fp/4p_qynps32sg30hs3m3bs5pc0000gn/T/tmp.v9JzrvrTB9
Setting stack config type: server
Setting MCP host: localhost (library/server mode)
Setting up environment variables:
export LLAMA_STACK_TEST_TEXT_MODEL='openai/gpt-4o'
export LLAMA_STACK_TEST_EMBEDDING_MODEL='openai/text-embedding-3-small'
export LLAMA_STACK_TEST_EMBEDDING_DIMENSION='1536'

Will use port: 8321
=== Starting Llama Stack Server ===
Waiting for Llama Stack Server to start on port 8321...
✅ Llama Stack Server started successfully

=== Running Integration Tests ===
Test subdirs to run:
+ STACK_CONFIG_ARG=
+ [[ -n server:ci-tests ]]
+ STACK_CONFIG_ARG=--stack-config=server:ci-tests
+ [[ false == \f\a\l\s\e ]]
+ pytest -s -v tests/integration/ --stack-config=server:ci-tests --inference-mode=record-if-missing -k 'not( builtin_tool or safety_with_image or code_interpreter or test_rag ) and TestOpenAIResponses' --setup=gpt --suite=responses --color=yes --embedding-model=sentence-transformers/nomic-ai/nomic-embed-text-v1.5 --color=yes --capture=tee-sys
=================================================================================================== test session starts ====================================================================================================
platform darwin -- Python 3.12.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/gualiu/go/src/github.com/llamastack/llama-stack/.venv/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.7', 'Platform': 'macOS-26.2-arm64-arm-64bit', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.9.0', 'html': '4.1.1', 'socket': '0.7.0', 'asyncio': '1.1.0', 'json-report': '1.5.0', 'timeout': '2.4.0', 'metadata': '3.1.1', 'cov': '6.2.1', 'nbval': '0.11.0'}}
rootdir: /Users/gualiu/go/src/github.com/llamastack/llama-stack
configfile: pyproject.toml
plugins: anyio-4.9.0, html-4.1.1, socket-0.7.0, asyncio-1.1.0, json-report-1.5.0, timeout-2.4.0, metadata-3.1.1, cov-6.2.1, nbval-0.11.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 97 items / 94 deselected / 3 selected

tests/integration/responses/test_openai_response.py::TestOpenAIResponses::test_openai_response_with_max_output_tokens[txt=openai/gpt-4o]
instantiating llama_stack_client
Port 8321 is already in use, assuming server is already running...
llama_stack_client instantiated in 0.025s
PASSED                                                                      [ 33%]
tests/integration/responses/test_openai_response.py::TestOpenAIResponses::test_openai_response_with_small_max_output_tokens[txt=openai/gpt-4o] PASSED                                                                [ 66%]
tests/integration/responses/test_openai_response.py::TestOpenAIResponses::test_openai_response_max_output_tokens_below_minimum[txt=openai/gpt-4o] PASSED                                                             [100%]

=================================================================================================== slowest 10 durations ===================================================================================================
0.14s call     tests/integration/responses/test_openai_response.py::TestOpenAIResponses::test_openai_response_with_max_output_tokens[txt=openai/gpt-4o]
0.03s setup    tests/integration/responses/test_openai_response.py::TestOpenAIResponses::test_openai_response_with_max_output_tokens[txt=openai/gpt-4o]
0.02s call     tests/integration/responses/test_openai_response.py::TestOpenAIResponses::test_openai_response_with_small_max_output_tokens[txt=openai/gpt-4o]

(6 durations < 0.005s hidden.  Use -vv to show these durations.)
======================================================================================= 3 passed, 94 deselected, 2 warnings in 0.28s =======================================================================================
+ exit_code=0
+ set +x
✅ All tests completed successfully

=== Integration Tests Complete ===
Stopping Llama Stack Server...
Killing Llama Stack Server processes: 50190
50190
Llama Stack Server stopped

gyliu513 · 2026-02-03T22:42:52Z

@mattf I was trying to add a integration test with openai client now, can you help check if this is the right direction? Thanks

gyliu513 · 2026-02-03T22:44:39Z

+            input=request.input,
+            model=request.model,
+            prompt=request.prompt,
+            instructions=request.instructions,
+            previous_response_id=request.previous_response_id,
+            conversation=request.conversation,
+            store=request.store,
+            stream=request.stream,
+            temperature=request.temperature,
+            text=request.text,
+            tool_choice=request.tool_choice,
+            tools=request.tools,
+            include=request.include,
+            max_infer_iters=request.max_infer_iters,
+            guardrails=request.guardrails,
+            parallel_tool_calls=request.parallel_tool_calls,
+            max_tool_calls=request.max_tool_calls,
+            max_output_tokens=request.max_output_tokens,
+            reasoning=request.reasoning,
+            metadata=request.metadata,


This can make sure when you are adding a new parameter here, do not need to consider the order of the parameters.

yeah this makes sense, I have been hitting this too

mattf

this is worth including just for the param naming!

imho, tests like this could be done w/ the openresponses cli.

gyliu513 · 2026-02-06T14:51:09Z

imho, tests like this could be done w/ the openresponses cli.

@mattf can you elaborate more on this? I did not see openresponses cli from https://github.com/openresponses

mattf · 2026-02-06T14:54:10Z

imho, tests like this could be done w/ the openresponses cli.

can you elaborate more on this? I did not see openresponses cli from https://github.com/openresponses

https://github.com/openresponses/openresponses?tab=readme-ov-file#cli

gyliu513 · 2026-02-06T15:29:19Z

@mattf +1 with you for the compliance integration test, I think after all response API finished, we can add a new integration file named as test_openresponses_compliance.py and this will call openresponse cli to do compliance test. With that, I think for #4817, we do not need to introduce OpenAI Client for Responses API Integration Test, comments?

mattf · 2026-02-07T13:23:10Z

@mattf +1 with you for the compliance integration test, I think after all response API finished, we can add a new integration file named as test_openresponses_compliance.py and this will call openresponse cli to do compliance test. With that, I think for #4817, we do not need to introduce OpenAI Client for Responses API Integration Test, comments?

no need to wait. the test can be informative and non-gating.

gyliu513 requested review from ashwinb, bbrowning, cdoern, ehhuang, franciscojavierarceo, leseb, mattf and raghotham as code owners February 3, 2026 22:38

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Feb 3, 2026

gyliu513 commented Feb 3, 2026

View reviewed changes

gyliu513 mentioned this pull request Feb 4, 2026

feat: add support for 'presence_penalty' param to Responses API #4830

Merged

feat: add integration test for max_output_tokens

5487faa

gyliu513 force-pushed the arg branch from b9a9fff to 5487faa Compare February 5, 2026 17:30

mattf approved these changes Feb 6, 2026

View reviewed changes

gyliu513 mentioned this pull request Feb 6, 2026

Use openresponse cli to do integration test for openai compliance #4848

Closed

Merge branch 'main' into arg

1a6de1a

mattf merged commit 5eff27a into ogx-ai:main Feb 10, 2026
47 checks passed

gyliu513 deleted the arg branch February 10, 2026 16:09

gyliu513 mentioned this pull request Feb 27, 2026

Use OpenAI Client for Responses API Integration Test #4817

Closed

2 tasks

gyliu513 mentioned this pull request Mar 11, 2026

Using max_output_tokens in responses API is throwing error #5089

Closed

2 tasks

Navigation Menu

Repository navigation

feat: add integration test for max_output_tokens - #4825

feat: add integration test for max_output_tokens#4825
mattf merged 2 commits into
ogx-ai:mainfrom
gyliu513:arg

gyliu513 commented Feb 3, 2026

Uh oh!

gyliu513 commented Feb 3, 2026

Uh oh!

gyliu513 Feb 3, 2026

Uh oh!

cdoern Feb 4, 2026

Uh oh!

mattf left a comment

Uh oh!

gyliu513 commented Feb 6, 2026

Uh oh!

mattf commented Feb 6, 2026

Uh oh!

gyliu513 commented Feb 6, 2026

Uh oh!

mattf commented Feb 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

gyliu513 commented Feb 3, 2026

What does this PR do?

Test Plan

Uh oh!

gyliu513 commented Feb 3, 2026

Uh oh!

gyliu513 Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

cdoern Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

mattf left a comment

Choose a reason for hiding this comment

Uh oh!

gyliu513 commented Feb 6, 2026

Uh oh!

mattf commented Feb 6, 2026

Uh oh!

gyliu513 commented Feb 6, 2026

Uh oh!

mattf commented Feb 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants