Skip to content

Commit fe6546b

Browse files
vdusekvdusek
authored
refactor: Generate all models and TypedDicts from OpenAPI spec (#765)
Closes: #758 ## Summary Every Pydantic model and TypedDict now comes from the OpenAPI spec — no more hand-maintained shapes alongside the generated ones. - Deleted the hand-written `src/apify_client/_models.py` and `src/apify_client/_typeddicts.py`. - Renamed `_models_generated.py` → `_models.py`, `_typeddicts_generated.py` → `_typeddicts.py`. The `_generated` suffix is no longer needed now that there is no parallel hand-written file. - Regenerated models pick up the new `WebhookRepresentation` schema added in apify/apify-docs#2469. ### Knock-on changes - `WebhookRepresentationList` class is gone — replaced by a single `encode_webhooks_to_base64()` helper in `_utils.py`. Callers in `actor.py` / `task.py` simplified to `webhooks=encode_webhooks_to_base64(webhooks)`. - `_wait_for_finish` already returns a plain dict and each caller validates it against the concrete `Run` / `Build` model, so the hand-written `ActorJob` / `ActorJobResponse` shapes are no longer needed. - `RequestInput` → `RequestDraft`, `RequestDeleteInput` → `RequestDraftDelete` (and matching `*Dict` names) — these were hand-rolled aliases for shapes that are already in the spec under the canonical names. The v3 upgrade guide is updated to match. - Bumped `RESOURCE_INPUT_TYPEDDICTS` in `scripts/postprocess_generated_models.py` to seed the typed-dict reachability walk with `RequestDraft`, `RequestDraftDelete`, and `WebhookRepresentation`. - Updated lint per-file-ignores from `_*_generated.py` to `_{models,typeddicts}.py` to match the new filenames. ## Pairs with - apify/apify-docs#2469 — adds the `WebhookRepresentation` schema component and references it from the `webhooks` query parameter. The docs PR should land first; this PR's regenerated models depend on it.
1 parent 4d7b67d commit fe6546b

63 files changed

Lines changed: 4359 additions & 4424 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.rules.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ uv run poe type-check # Run ty type checker
1919
uv run poe unit-tests # Run unit tests
2020
uv run poe check-docstrings # Verify async docstrings match sync
2121
uv run poe fix-docstrings # Auto-fix async docstrings
22-
uv run poe generate-models # Regenerate _models_generated.py and _typeddicts_generated.py from live OpenAPI spec
22+
uv run poe generate-models # Regenerate _models.py and _typeddicts.py from live OpenAPI spec
2323
uv run poe generate-models-from-file <path> # Regenerate from a local OpenAPI spec file
2424

2525
# Run a single test
@@ -73,7 +73,7 @@ Docstrings are written on sync clients and **automatically copied** to async cli
7373

7474
### Data Models
7575

76-
`src/apify_client/_models_generated.py` and `src/apify_client/_typeddicts_generated.py` are **auto-generated** — do not edit them manually. The hand-maintained `src/apify_client/_models.py` and `src/apify_client/_typeddicts.py` hold only shapes that are not exposed by the OpenAPI spec (or that need local logic); import generated types directly from the `_*_generated` modules.
76+
`src/apify_client/_models.py` and `src/apify_client/_typeddicts.py` are **auto-generated** — do not edit them manually. Every Pydantic model and TypedDict comes from the OpenAPI spec.
7777

7878
- Generated by `datamodel-code-generator` from the OpenAPI spec at `https://docs.apify.com/api/openapi.json` (config in `pyproject.toml` under `[tool.datamodel-codegen]`, aliases in `datamodel_codegen_aliases.json`)
7979
- After generation, `scripts/postprocess_generated_models.py` is run to apply additional fixes

‎docs/02_concepts/code/03_nested_async.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
from apify_client import ApifyClientAsync
2-
from apify_client._models_generated import ActorJobStatus
2+
from apify_client._models import ActorJobStatus
33

44
TOKEN = 'MY-APIFY-TOKEN'
55

‎docs/02_concepts/code/03_nested_sync.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
from apify_client import ApifyClient
2-
from apify_client._models_generated import ActorJobStatus
2+
from apify_client._models import ActorJobStatus
33

44
TOKEN = 'MY-APIFY-TOKEN'
55

‎docs/04_upgrading/upgrading_to_v3.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ run.status
5555
Models also use `populate_by_name=True`, which means you can use either the Python field name or the camelCase alias when **constructing** a model:
5656

5757
```python
58-
from apify_client._models_generated import Run
58+
from apify_client._models import Run
5959

6060
# Both work when constructing models
6161
Run(default_dataset_id='abc') # Python field name
@@ -86,7 +86,7 @@ rq_client.add_request({
8686
After (v3) — both forms are accepted:
8787

8888
```python
89-
from apify_client._models import RequestInput
89+
from apify_client._models import RequestDraft
9090

9191
# Option 1: dict (still works)
9292
rq_client.add_request({
@@ -96,7 +96,7 @@ rq_client.add_request({
9696
})
9797

9898
# Option 2: Pydantic model (new)
99-
rq_client.add_request(RequestInput(
99+
rq_client.add_request(RequestDraft(
100100
url='https://example.com',
101101
unique_key='https://example.com',
102102
method='GET',

‎pyproject.toml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ indent-style = "space"
152152
"**/docs/03_guides/code/05_custom_http_client_{async,sync}.py" = [
153153
"ARG002", # Unused method argument
154154
]
155-
"src/apify_client/_*_generated.py" = [
155+
"src/apify_client/_{models,typeddicts}.py" = [
156156
"D", # Everything from the pydocstyle
157157
"E501", # Line too long
158158
"ERA001", # Commented-out code
@@ -213,7 +213,7 @@ context = 7
213213
# https://koxudaxi.github.io/datamodel-code-generator/
214214
[tool.datamodel-codegen]
215215
input_file_type = "openapi"
216-
output = "src/apify_client/_models_generated.py"
216+
output = "src/apify_client/_models.py"
217217
target_python_version = "3.11"
218218
output_model_type = "pydantic_v2.BaseModel"
219219
use_schema_description = true
@@ -285,7 +285,7 @@ cwd = "website"
285285
shell = """
286286
uv run datamodel-codegen --url https://docs.apify.com/api/openapi.json \
287287
&& uv run datamodel-codegen --url https://docs.apify.com/api/openapi.json \
288-
--output src/apify_client/_typeddicts_generated.py \
288+
--output src/apify_client/_typeddicts.py \
289289
--output-model-type typing.TypedDict \
290290
--no-use-closed-typed-dict \
291291
&& python scripts/postprocess_generated_models.py
@@ -295,7 +295,7 @@ uv run datamodel-codegen --url https://docs.apify.com/api/openapi.json \
295295
shell = """
296296
uv run datamodel-codegen --input $input_file \
297297
&& uv run datamodel-codegen --input $input_file \
298-
--output src/apify_client/_typeddicts_generated.py \
298+
--output src/apify_client/_typeddicts.py \
299299
--output-model-type typing.TypedDict \
300300
--no-use-closed-typed-dict \
301301
&& python scripts/postprocess_generated_models.py

‎scripts/postprocess_generated_models.py‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
"""Post-process datamodel-codegen output to fix known issues and prune the TypedDict file.
22
3-
Applied to `_models_generated.py`:
3+
Applied to `_models.py`:
44
- Fix discriminator field names that use camelCase instead of snake_case (known issue with
55
discriminators on schemas referenced from array items).
66
- Deduplicate the inlined `Type(StrEnum)` that comes from ErrorResponse.yaml; rewire to `ErrorType`.
77
- Add `@docs_group('Models')` to every model class (plus the required import).
88
9-
Applied to `_typeddicts_generated.py`:
9+
Applied to `_typeddicts.py`:
1010
- Keep only the TypedDicts actually used as resource-client method inputs (plus their transitive
1111
dependencies). The file is generated in full by datamodel-codegen; the trimming happens here.
1212
- Rename every kept class to add a `Dict` suffix so it doesn't clash with the Pydantic model name
@@ -27,8 +27,8 @@
2727

2828
REPO_ROOT = Path(__file__).resolve().parent.parent
2929
PACKAGE_DIR = REPO_ROOT / 'src' / 'apify_client'
30-
MODELS_PATH = PACKAGE_DIR / '_models_generated.py'
31-
TYPEDDICTS_PATH = PACKAGE_DIR / '_typeddicts_generated.py'
30+
MODELS_PATH = PACKAGE_DIR / '_models.py'
31+
TYPEDDICTS_PATH = PACKAGE_DIR / '_typeddicts.py'
3232

3333
# Map of camelCase discriminator values to their snake_case equivalents.
3434
# Add new entries here as needed when the OpenAPI spec introduces new discriminators.
@@ -37,16 +37,19 @@
3737
}
3838

3939
# TypedDicts accepted as inputs by resource-client methods. These are the roots of the reachability
40-
# walk over `_typeddicts_generated.py`: anything not reachable from here (directly or transitively)
40+
# walk over `_typeddicts.py`: anything not reachable from here (directly or transitively)
4141
# is dropped so only the TypedDicts that are part of the public input surface — plus their nested
4242
# shapes — survive. Names are the raw datamodel-codegen outputs (no `Dict` suffix yet); the suffix
4343
# is added later by `rename_with_dict_suffix`. Update this set whenever a new `<Name>Dict | <Name>`
4444
# union is introduced on a resource-client method signature.
4545
RESOURCE_INPUT_TYPEDDICTS: frozenset[str] = frozenset(
4646
{
4747
'Request', # RequestQueueClient.update_request
48+
'RequestDraft', # RequestQueueClient.add_request, batch_add_requests
49+
'RequestDraftDelete', # RequestQueueClient.batch_delete_requests
4850
'TaskInput', # Actor/Task start/call/update default input
4951
'WebhookCreate', # Actor/Task start/call webhook list element
52+
'WebhookRepresentation', # Actor/Task start/call ad-hoc webhook list element
5053
}
5154
)
5255

@@ -254,7 +257,7 @@ def rename_with_dict_suffix(content: str, names: set[str]) -> str:
254257

255258

256259
def postprocess_models(path: Path) -> bool:
257-
"""Apply `_models_generated.py`-specific fixes. Returns True if the file changed."""
260+
"""Apply `_models.py`-specific fixes. Returns True if the file changed."""
258261
original = path.read_text()
259262
fixed = fix_discriminators(original)
260263
fixed = deduplicate_error_type_enum(fixed)
@@ -266,7 +269,7 @@ def postprocess_models(path: Path) -> bool:
266269

267270

268271
def postprocess_typeddicts(path: Path) -> bool:
269-
"""Apply `_typeddicts_generated.py`-specific fixes. Returns True if the file changed."""
272+
"""Apply `_typeddicts.py`-specific fixes. Returns True if the file changed."""
270273
original = path.read_text()
271274
pruned, kept = prune_typeddicts(original, RESOURCE_INPUT_TYPEDDICTS)
272275
renamed = rename_with_dict_suffix(pruned, kept)
@@ -291,13 +294,13 @@ def main() -> None:
291294
changed.append(MODELS_PATH)
292295
print(f'Fixed generated models in {MODELS_PATH}')
293296
else:
294-
print('No fixes needed for _models_generated.py')
297+
print('No fixes needed for _models.py')
295298

296299
if postprocess_typeddicts(TYPEDDICTS_PATH):
297300
changed.append(TYPEDDICTS_PATH)
298301
print(f'Pruned and renamed TypedDicts in {TYPEDDICTS_PATH}')
299302
else:
300-
print('No fixes needed for _typeddicts_generated.py')
303+
print('No fixes needed for _typeddicts.py')
301304

302305
if changed:
303306
run_ruff(changed)

‎src/apify_client/_consts.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
from datetime import timedelta
44

5-
from apify_client._models_generated import ActorJobStatus
5+
from apify_client._models import ActorJobStatus
66

77
DEFAULT_API_URL = 'https://api.apify.com'
88
"""Default base URL for the Apify API."""

0 commit comments

Comments
 (0)