Unverified mock behaviour is indistinguishable from verified behaviour #3445

#3444 lists what to do about unverified Model Target behaviour once the credentials are rotated. Writing it made clear that Model Target is one instance of a class which runs through the whole project, so this issue covers the class and #3444 becomes the first instance of it.

The class

The mock asserts things about real Vuforia which have never been observed from real Vuforia, and nothing in the repository distinguishes those from the things which have.

A test which runs only against the mock backends still passes, still counts towards coverage, and still reads like verification. TestErrorResponses::test_invalid_dataset_request asserts 33 exact Vuforia error strings and not one of them has ever been compared with Vuforia's. That is not a defect in the test — the credentials are revoked (#3441) — but there is no way to see it. The split in #3441 had to be counted by hand, one xfail at a time, and it was only counted because someone went looking.

Three distinct reasons a claim can be unverified, which currently look identical:

1. Temporarily unverifiable — the credentials or scope are missing, but the behaviour is observable in principle. Model Target creation validation (Every Model Target test which needs an access token is xfailed because the OAuth2 credentials are revoked #3441, Request advanced Model Target scope and convert advanced mock-only test #3202); advanced and State-Based dataset routes.
2. Inherently unverifiable — the behaviour cannot be provoked against real Vuforia at all. A VuMark target held in PROCESSING (targets cannot be created through the API), a dataset caught mid-generation, a configured generation failure. These are correctly mock-only forever, and tests/mock_vws/test_vumark_generation_api.py explains itself well — it is the model the others should follow.
3. Never attempted — the mock implements something from Vuforia's public documentation and nobody has checked the documentation is accurate.

Category 3 is the one with no tracking at all. The instances I found:

• Request quota exhaustion. RequestQuotaReachedError says in its own docstring that it "has not been verified against a real database with an exhausted quota". The status code and body shape come from Vuforia's documentation and the common VWS error shape.
• Request rate limits. docs/source/differences-to-vws.rst records documented limits of 15/45/10 requests per second and 1 request per minute, then says "The documented numbers have not been verified against a real database". The mock applies no limit by default, so nothing exercises them.
• The reco counts report 404. "The 404 has not been verified, because no request for a real report has caught one before it was generated."
• NGINX error handling. The docs note Vuforia returns 400 for a header or cookie larger than 8 KiB. The mock does not implement it and no test asserts either behaviour.
• Every Model Target assumption in Work list for once the Model Target credentials are rotated #3444, section 1.

Why it matters

Each of these is a place where a user of the mock writes code against behaviour which may not exist. The failure mode is quiet: the mock's tests pass, the user's tests pass, and the divergence only appears in production against real Vuforia. It is the same failure mode #3400 describes for the cross-cutting endpoint fixtures — the mechanism which would catch a problem is not pointed at the thing.

The Model Target case also shows how it decays. get_access_token calls pytest.xfail() imperatively, so 102 tests stopped verifying anything and CI stayed green. xfail_strict = true does not apply to imperative pytest.xfail(), only to the marker, so this can never XPASS and will hide the next revocation exactly as it hid this one.

Suggested resolution

Not one change; three.

1. Make the categories explicit. Every mock-only test should say which of the three reasons applies, in a way that can be read mechanically rather than from prose. The existing fixtures already encode part of this — mock_only_vuforia and model_target_mock_only_vuforia — but tests which construct MockVWS() directly bypass them, and there are 63 such call sites across nine files. A marker such as @pytest.mark.mock_only(reason=...) with a constrained reason, or separate fixtures per category, would make "how much of this suite verifies anything" a query rather than an audit.

2. Make the count visible. Something should report the verified/unverified split per API, and fail when it moves the wrong way. The revoked credentials should have produced one loud failure, not 102 quiet xfails.

3. Separate the three kinds of statement in differences-to-vws.rst. The document currently mixes deliberate differences ("The mock returns a fixed minimal image"), unimplemented behaviour ("This has error handling which is not duplicated in the mock") and unverified guesses ("has not been verified against a real database") in the same voice. A reader cannot tell which is which, and the third kind is the only one which is a bug waiting to happen. Mark the unverified claims so they can be listed, and so that removing one is a visible event.

Acceptance criteria

• An inventory exists of every claim the mock makes which has not been observed from real Vuforia, each tagged with which of the three reasons applies.
• Every mock-only test states its category in a machine-readable way, including those which construct MockVWS() directly.
• The verified/unverified split per API is reported somewhere that changes are noticed, and a drop fails rather than passes quietly.
• differences-to-vws.rst distinguishes deliberate differences from unverified assumptions.
• No imperative pytest.xfail() silently converts a verified test into an unverified one.

Instances

• Work list for once the Model Target credentials are rotated #3444 — Model Target, blocked on Every Model Target test which needs an access token is xfailed because the OAuth2 credentials are revoked #3441.
• Cross-cutting endpoint tests do not cover the reco counts or Model Target endpoints #3400 — the same shape of gap in the cross-cutting endpoint fixtures.
• Request advanced Model Target scope and convert advanced mock-only test #3202 — advanced Model Target scope.
• Request quota exhaustion, request rate limits, the reco counts 404 and NGINX error handling have no issue of their own yet; whether they get one should follow from the inventory.