Follow-up to #3343, #3357 and #3359.
#3359 restores verified tests for requesting a reco counts report. It cannot verify what the request returns you to, because that needs waiting for a report to generate.
The gap
Real Vuforia takes between a few seconds and one hour to generate a report. TestDownloadReport in tests/mock_vws/test_reco_counts_report.py is therefore mock-only for a reason that #3359 does not fix: even with a database ID credential in hand, a normal test run cannot wait that long, and the credentialed CI jobs run on every push.
So these remain unverified no matter what #3359 does:
- the CSV itself: the column names, the header row, and the row format for a target with recognitions;
- the headers of the download response, including whether the content type really is
text/csv;
- the documented
404-until-ready behaviour, and what the response looks like before the report is ready;
- whether the download really needs no authorization, as the mock assumes.
The one part of the download that #3359 can verify is the shape of the presigned_url value, since that comes back immediately: its host, path and query parameters. Real Vuforia returns a cloud storage URL, which normally carries X-Amz-* query parameters; the mock returns a URL served by itself with no query parameters at all. That check belongs in #3359 and has been moved there.
Approach
This needs a mechanism other than the normal test suite. Options, roughly in order of cost:
- A one-off script under
admin/, in the spirit of admin/create_secrets_files.py, which a maintainer runs by hand: request a report, poll the presigned URL until it resolves, and print the status codes, headers and body at each stage. Record the findings in differences-to-vws.rst and fix the mock where it is wrong. Cheapest, and enough to settle the questions above once.
- A scheduled workflow, separate from the per-push CI, which does the same thing on a slow cadence and fails if the shape changes. Guards against drift, at the cost of a job that can run for an hour.
- A test marked to run only in such a scheduled job, with a long poll timeout.
Option 1 is probably right unless we want ongoing drift detection, since this endpoint is unlikely to change often.
Note on the row format
Confirming the row format is harder than the rest. The test databases have no recognitions, so a real report for one of them is likely to be header-only too — which would confirm the file's shape but tell us nothing about how a row is written. Producing a non-empty real report means performing queries against a real database and then waiting for Vuforia's recognition counts to catch up, and those counts are known to lag well past the length of a test run (see #3356). Getting the header row confirmed is worthwhile on its own; the row format may have to stay a guess.
Acceptance criteria
- The download half of the endpoint has been exercised against real Vuforia at least once, by whichever mechanism is chosen.
- The mock's CSV content, download response headers and not-ready behaviour either match what was observed or are corrected.
differences-to-vws.rst is updated to drop the claims which are no longer unverified, and to state plainly whichever ones remain guesses.
Follow-up to #3343, #3357 and #3359.
#3359 restores verified tests for requesting a reco counts report. It cannot verify what the request returns you to, because that needs waiting for a report to generate.
The gap
Real Vuforia takes between a few seconds and one hour to generate a report.
TestDownloadReportintests/mock_vws/test_reco_counts_report.pyis therefore mock-only for a reason that #3359 does not fix: even with a database ID credential in hand, a normal test run cannot wait that long, and the credentialed CI jobs run on every push.So these remain unverified no matter what #3359 does:
text/csv;404-until-ready behaviour, and what the response looks like before the report is ready;The one part of the download that #3359 can verify is the shape of the
presigned_urlvalue, since that comes back immediately: its host, path and query parameters. Real Vuforia returns a cloud storage URL, which normally carriesX-Amz-*query parameters; the mock returns a URL served by itself with no query parameters at all. That check belongs in #3359 and has been moved there.Approach
This needs a mechanism other than the normal test suite. Options, roughly in order of cost:
admin/, in the spirit ofadmin/create_secrets_files.py, which a maintainer runs by hand: request a report, poll the presigned URL until it resolves, and print the status codes, headers and body at each stage. Record the findings indifferences-to-vws.rstand fix the mock where it is wrong. Cheapest, and enough to settle the questions above once.Option 1 is probably right unless we want ongoing drift detection, since this endpoint is unlikely to change often.
Note on the row format
Confirming the row format is harder than the rest. The test databases have no recognitions, so a real report for one of them is likely to be header-only too — which would confirm the file's shape but tell us nothing about how a row is written. Producing a non-empty real report means performing queries against a real database and then waiting for Vuforia's recognition counts to catch up, and those counts are known to lag well past the length of a test run (see #3356). Getting the header row confirmed is worthwhile on its own; the row format may have to stay a guess.
Acceptance criteria
differences-to-vws.rstis updated to drop the claims which are no longer unverified, and to state plainly whichever ones remain guesses.