Add E2E tests for CSV/JSON import overwrite and skip by premtsd-code · Pull Request #11818 · appwrite/appwrite

premtsd-code · 2026-04-08T07:14:21Z

Summary

Adds E2E tests for overwrite and skip params on CSV and JSON import endpoints
Tests mutual exclusion validation (both true → 400)
Tests skip mode: duplicate rows silently skipped, original data preserved
Tests overwrite mode: existing rows updated with new data, total count unchanged
Points utopia-php/database and utopia-php/migration to csv-import-upsert branches

Depends on:

Add ignore param to createDocuments for silent duplicate handling utopia-php/database#850
Add overwrite and skip options to Appwrite destination utopia-php/migration#168

Test plan

testCSVImportOverwriteAndSkip — passes locally (31 assertions)
testJSONImportOverwriteAndSkip — passes locally (32 assertions)

- Add testCSVImportOverwriteAndSkip and testJSONImportOverwriteAndSkip - Test mutual exclusion validation (overwrite+skip returns 400) - Test skip: duplicates silently skipped, originals unchanged - Test overwrite: existing rows updated with new data - Point database and migration deps to csv-import-upsert branches

github-actions · 2026-04-08T07:29:31Z

🔄 PHP-Retry Summary

Flaky tests detected across commits:

Commit 6e47274 - 3 flaky tests

Test	Retries	Total Time	Details
`TablesDBCustomClientTest::testOneToOneRelationship`	1	240.29s	Logs
`TablesDBCustomServerTest::testInvalidDocumentStructure`	1	240.34s	Logs
`TablesDBCustomServerTest::testOrQueries`	1	240.64s	Logs

Commit 5ec0ed7 - 1 flaky test

Test	Retries	Total Time	Details
`FunctionsScheduleTest::testCreateScheduledAtExecution`	1	128.38s	Logs

Commit fcd32e1 - 7 flaky tests

Test	Retries	Total Time	Details
`UsageTest::testFunctionsStats`	1	10.19s	Logs
`UsageTest::testPrepareSitesStats`	1	6ms	Logs
`UsageTest::testEmbeddingsTextUsageDoesNotBreakProjectUsage`	1	5ms	Logs
`LegacyCustomClientTest::testOrQueries`	1	240.37s	Logs
`LegacyTransactionPermissionsCustomClientTest::testCannotDeleteNonExistentDocument`	1	240.31s	Logs
`VectorsDBConsoleClientTest::testGetCollectionLogs`	1	7ms	Logs
`DatabasesConsoleClientTest::testGetCollectionLogs`	1	11ms	Logs

greptile-apps · 2026-04-08T07:34:22Z

Greptile Summary

This PR adds E2E tests for overwrite and skip import options on CSV and JSON migration endpoints, along with the controller and worker changes to support these options. The controller correctly validates mutual exclusion at the API layer for all affected endpoints, and the worker plumbs the options through to DestinationAppwrite via setOverwrite/setSkip. The new test fixtures (documents-duplicates.csv and .json) are minimal and well-formed.

Confidence Score: 4/5

Not safe to merge into 1.9.x until upstream dev-branch pins in composer.json are replaced with stable releases.

The production code changes (controller and worker) are logically sound and backward-compatible. However, the open blocker from a prior review thread — utopia-php/database and utopia-php/migration pinned to transient dev branches — still applies and is a clear merge blocker for a stable branch. The new P1 finding (assertEventually masking migration failures) is a test reliability issue that should be fixed before this PR lands to avoid silent CI failures in future regressions.

composer.json (dev-branch pins) and tests/e2e/Services/Migrations/MigrationsBase.php (assertEventually fast-fail, statusCounters assertions).

Vulnerabilities

No security concerns identified. The new overwrite and skip parameters are boolean flags validated server-side with mutual exclusion checks, and they default to false. No user-controlled values are passed unsanitised to storage or query layers.

Important Files Changed

Filename	Overview
app/controllers/api/migrations.php	Adds `overwrite` and `skip` params to all migration source endpoints (Appwrite, Firebase, Supabase, NHost, CSV import, JSON import) with mutual-exclusion validation. Logic is consistent; options are stored in the migration document for the worker to consume.
src/Appwrite/Platform/Workers/Migrations.php	Worker now reads `overwrite`/`skip` from migration `options` and calls `setOverwrite`/`setSkip` on `DestinationAppwrite`. Guard correctly scoped to `DestinationAppwrite` instances; CSV/JSON imports resolve to this destination so the path is exercised.
tests/e2e/Services/Migrations/MigrationsBase.php	Two new test methods cover mutual exclusion, skip mode, and overwrite mode. Several assertion gaps remain (noted in open review threads): column-creation responses unchecked, `statusCounters` not verified on skip/overwrite runs, and only one duplicate row's value is verified after skip.
tests/resources/csv/documents-duplicates.csv	Four-row fixture: 2 IDs matching the initial dataset (to test duplicate handling) + 2 new IDs. Data values are within the declared column constraints (age 18–65).
tests/resources/json/documents-duplicates.json	JSON equivalent of the CSV fixture; mirrors the same four records and constraints.
composer.json	Pins `utopia-php/database` and `utopia-php/migration` to unreleased dev branches (open upstream PRs), which is a blocker for merging into `1.9.x`.

_{Reviews (6): Last reviewed commit: "Pass overwrite and skip options from mig..." | Re-trigger Greptile}

greptile-apps · 2026-04-08T07:34:28Z

+        $this->assertEventually(function () use ($skipMigration) {
+            $m = $this->client->call(Client::METHOD_GET, '/migrations/' . $skipMigration['body']['$id'], array_merge([
+                'content-type' => 'application/json',
+                'x-appwrite-project' => $this->getProject()['$id'],
+            ], $this->getHeaders()));
+            $this->assertEquals('finished', $m['body']['stage']);
+            $this->assertEquals('completed', $m['body']['status']);
+        }, 30_000, 500);
+
+        // Verify total is 102 (100 original + 2 new), not 104
+        $rows = $this->client->call(Client::METHOD_GET, '/tablesdb/' . $databaseId . '/tables/' . $tableId . '/rows', array_merge([
+            'content-type' => 'application/json',
+            'x-appwrite-project' => $this->getProject()['$id'],
+        ], $this->getHeaders()), [
+            'queries' => [Query::limit(150)->toString()]
+        ]);
+        $this->assertEquals(102, $rows['body']['total']);
+
+        // Verify original row was NOT overwritten
+        $row = $this->client->call(Client::METHOD_GET, '/tablesdb/' . $databaseId . '/tables/' . $tableId . '/rows/hxfcwpcas5xokpwe', array_merge([
+            'content-type' => 'application/json',
+            'x-appwrite-project' => $this->getProject()['$id'],
+        ], $this->getHeaders()));
+        $this->assertEquals('Diamond Mendez', $row['body']['name']);


Skip-mode statusCounters not verified

After the skip migration completes, the test checks the aggregate row count and one specific row's name, but does not assert statusCounters to confirm exactly 2 rows were skipped and 2 new rows were inserted. Without this, a regression where skip does nothing would still pass the existing assertions.

Similarly, only hxfcwpcas5xokpwe is checked for the original value - the second duplicate gw8nxwf6esn3tfwf is not verified. The same gap exists in the JSON equivalent at lines 1761-1784.

Consider adding:

$this->assertEquals(2, $m['body']['statusCounters'][Resource::TYPE_ROW]['skipped'] ?? 0); $this->assertEquals(2, $m['body']['statusCounters'][Resource::TYPE_ROW]['success']);

and a second row check for gw8nxwf6esn3tfwf.

github-actions · 2026-04-08T07:34:54Z

✨ Benchmark results

Requests per second: 1,803
Requests with 200 status code: 324,506
P99 latency: 0.100245716

⚡ Benchmark Comparison

Metric	This PR	Latest version
RPS	1,803	1,364
200	324,506	245,566
P99	0.100245716	0.149546395

…branches

# Conflicts: # composer.lock

…n adapter

premtsd-code added 2 commits April 8, 2026 08:12

Merge 1.9.x and resolve composer conflicts

6e47274

This comment has been minimized.

Sign in to view

greptile-apps bot reviewed Apr 8, 2026

View reviewed changes

premtsd-code added 6 commits April 8, 2026 08:47

Assert column creation responses in overwrite/skip tests

5ec0ed7

Add overwrite and skip params to all migration endpoints

34ce1cd

Point utopia-php/database to csv-import-upsert-v2 branch

eac9858

Update composer.lock for database csv-import-upsert-v2 and migration …

be464d6

…branches

Merge remote-tracking branch 'origin/1.9.x' into csv-import-upsert

2490784

# Conflicts: # composer.lock

Pass overwrite and skip options from migration document to destinatio…

fcd32e1

…n adapter

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add E2E tests for CSV/JSON import overwrite and skip#11818

Add E2E tests for CSV/JSON import overwrite and skip#11818
premtsd-code wants to merge 8 commits into1.9.xfrom
csv-import-upsert

premtsd-code commented Apr 8, 2026

Uh oh!

github-actions bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

greptile-apps bot commented Apr 8, 2026 •

edited

Loading

Vulnerabilities

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps bot Apr 8, 2026

Uh oh!

github-actions bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

premtsd-code commented Apr 8, 2026

Summary

Test plan

Uh oh!

github-actions bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔄 PHP-Retry Summary

Uh oh!

This comment has been minimized.

greptile-apps bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Vulnerabilities

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ Benchmark results

⚡ Benchmark Comparison

Uh oh!

This comment has been minimized.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions bot commented Apr 8, 2026 •

edited

Loading

greptile-apps bot commented Apr 8, 2026 •

edited

Loading

github-actions bot commented Apr 8, 2026 •

edited

Loading