Fixes #790

Any chance we can expand this for arrays, so we can filter for empty/non-empty arrays? Or should we create a separate issue for the next iteration?

Any chance we can expand this for arrays, so we can filter for empty/non-empty arrays? Or should we create a separate issue for the next iteration?

@alangmartini Could you create another issue for this? I will address that in another PR.

@kishorenc PR is ready for your review.

A bit of manual + automated review but I have gone through each issue carefully and verified that proposed fixes are logical. I have attached a patch. @happy-san please review and confirm.

2777_review_patch.patch

Issues found

1. optional_index metadata was updated from partial update payloads instead of the final merged document state. In mixed update batches this could incorrectly mark an unchanged optional field as missing, breaking field: _exists / field: !_exists.
2. optional_index: true was accepted on index: false fields. That schema produced no backing existence index, so _exists / !_exists could return incorrect results.
3. _exists parsing used substring matching, so ordinary string filters containing _exists (for example title: pre_exists_post) were misparsed as existence filters.
4. field::field_from_json() did not preserve optional_index, so JSON-to-field reconstruction silently dropped the flag.
5. Lazy _exists iterators could not be reset or materialized correctly. compute_iterators() and related paths treated the missing-id list as the final result set, so lazy _exists searches could return wrong hits.

Fix summary

• Existence bookkeeping now uses each record's final document state (new_doc for updates, doc for inserts) and updates the missing-field index in one place.
• Schema validation now rejects optional_index unless the field is both optional: true and index: true.
• Existence parsing now only triggers on exact _exists / !_exists tokens, while normal string filters continue to work.
• field::field_from_json() now carries optional_index through correctly.
• Lazy _exists iteration now has dedicated reset and materialization logic, so _exists uses the complement of the missing-id list consistently in iterator and search paths.
• Added focused regression tests for mixed-batch updates, string literals containing _exists, invalid optional_index schemas, JSON field reconstruction, lazy iterator materialization, and lazy search hits.

@ozanarmagan I have listed the tests that will surface each issue:

1. Have a schema like,

{
  "fields": [
    {
      "name": "field",
      "type": "string"
    },
    {
      "name": "optional_field",
      "type": "string",
      "optional": true,
      "optional_index": true
    }
  ]
}

Add a document like,

{
  "field": "foo",
  "optional_field": "bar"
}

Check that filter_by: optional_field: !_exists should match no document.
Update the document with:

{
  "field": "baz"
}

filter_by: optional_field: !_exists should still match no document.
2. A simple test that should fail when creating a field with optional_index: true and index: false should work.
3. Using the schema ,

{
  "fields": [
    {
      "name": "field_exists",
      "type": "string"
    },
    {
      "name": "field",
      "type": "string"
    }
  ]
}

try passing a filter like field_exists: foo or field: value_exists.
4. A test like https://github.com/typesense/typesense/blob/v31/test/collection_join_test.cpp#L6013-L6021 will surface this issue. We must add this test when a new option is added in the field.
5. Add

    iter_exists.reset();
    ASSERT_EQ(filter_result_iterator_t::valid, iter_exists.validity);

    for (uint32_t i = 0; i < validate_ids.size(); i++) {
        ASSERT_EQ(filter_result_iterator_t::valid, iter_exists.validity);
        ASSERT_EQ(expected[i], iter_exists.is_valid(validate_ids[i]));

        if (expected[i] == 1) {
            iter_exists.next();
        }
        ASSERT_EQ(seq_ids[i], iter_exists.seq_id);
    }
    ASSERT_EQ(filter_result_iterator_t::invalid, iter_exists.validity);

after this line

The file I have attached to my previous comment already has the patch that contains both the code and the tests for these issues.