Skip to content

fix(idex): preserve integer placeholder type in monthly count products#3229

Open
davidt0x wants to merge 1 commit into
IMAP-Science-Operations-Center:devfrom
davidt0x:fix/istp_idex_monthly_count_placeholders
Open

fix(idex): preserve integer placeholder type in monthly count products#3229
davidt0x wants to merge 1 commit into
IMAP-Science-Operations-Center:devfrom
davidt0x:fix/istp_idex_monthly_count_placeholders

Conversation

@davidt0x
Copy link
Copy Markdown
Collaborator

Summary

Preserve integer placeholder typing in the monthly IDEX count products.

This PR is a small part of the broader ISTP / metadata cleanup tracked in issue #2577. In that larger effort, this branch addresses one specific remaining file-level metadata mismatch in the monthly IDEX outputs by making the written count-variable types agree with their declared CDF metadata.

Scope:

  • imap_idex_l2b_sci-1mo
  • imap_idex_l2c_rectangular-map-1mo

Why

The monthly IDEX products intentionally blank several derived outputs while the science team continues validating the fitting routines and derived values.

Before this change, the placeholder block applied np.nan to both:

  • floating-point rate variables
  • integer count variables

That promoted the monthly count variables to floating-point in the written CDFs, while their metadata remained integer-style. The result was an ISTP/SPDF mismatch in the emitted files for:

  • counts_by_charge
  • counts_by_mass
  • counts_by_charge_map
  • counts_by_mass_map

So within the broader metadata cleanup under #2577, this PR fixes the IDEX monthly count-placeholder type mismatch at the source.

What Changed

Preserve integer placeholder typing for monthly counts

For the monthly count variables only, replace the NaN placeholder behavior with the standard int64 CDF fill sentinel:

  • counts_by_charge
  • counts_by_mass
  • counts_by_charge_map
  • counts_by_mass_map

These variables now:

  • stay np.int64
  • use -9223372036854775808 as the placeholder fill value

Leave monthly rate placeholders unchanged

The monthly rate placeholders are intentionally unchanged and still use NaN:

  • rate_by_charge
  • rate_by_mass
  • rate_by_charge_map
  • rate_by_mass_map

That remains valid because those variables are already float-typed and their metadata matches the written data.

Update tests accordingly

The monthly IDEX regression tests now:

  • assert that the count placeholders are filled with the integer fill sentinel
  • keep the existing NaN assertions for the rate placeholders
  • verify that the written monthly count variables serialize as integer CDF types with the expected integer FILLVAL

Testing

Validated locally with:

pytest -q imap_processing/tests/idex/test_idex_l2b.py
pre-commit run --files \
  imap_processing/idex/idex_l2b.py \
  imap_processing/tests/idex/test_idex_l2b.py

Keep the temporary June-release placeholder behavior for the monthly IDEX
products, but stop coercing the count variables to floating-point `NaN`.

The monthly L2B and L2C outputs intentionally blank derived products
while the IDEX team continues validating the fitting routines and
derived values. However, the placeholder block was applying `np.nan` to
both the floating-point rate variables and the integer count variables.

That caused the count variables to be promoted to floating point in the
written CDFs:
- `counts_by_charge`
- `counts_by_mass`
- `counts_by_charge_map`
- `counts_by_mass_map`

Their metadata remained integer-style, so the SPDF validator reported
type/fill/format mismatches in the monthly files.

Fix this by preserving the intended integer type for the monthly count
placeholders:
- replace the `np.nan` placeholder for the monthly count variables with
  the standard `int64` CDF fill sentinel
  (`-9223372036854775808`)
- keep the count arrays as `np.int64`
- leave the monthly rate placeholders unchanged as `NaN`, since their
  float metadata already matches the written data

Also update the monthly IDEX tests so they:
- assert the count placeholders are fully set to the integer fill value
- keep the existing `NaN` expectations for the rate placeholders
- verify the written monthly CDF count variables serialize as `CDF_INT8`
  and carry the standard integer `FILLVAL`
- use context-managed `cdflib.CDF(...)` opens in the new metadata checks

Validated locally with:
- `/home/david/.virtualenvs/imap/bin/python -m pytest -q imap_processing/tests/idex/test_idex_l2b.py`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant