fix(idex): preserve integer placeholder type in monthly count products#3229
Open
davidt0x wants to merge 1 commit into
Open
Conversation
Keep the temporary June-release placeholder behavior for the monthly IDEX products, but stop coercing the count variables to floating-point `NaN`. The monthly L2B and L2C outputs intentionally blank derived products while the IDEX team continues validating the fitting routines and derived values. However, the placeholder block was applying `np.nan` to both the floating-point rate variables and the integer count variables. That caused the count variables to be promoted to floating point in the written CDFs: - `counts_by_charge` - `counts_by_mass` - `counts_by_charge_map` - `counts_by_mass_map` Their metadata remained integer-style, so the SPDF validator reported type/fill/format mismatches in the monthly files. Fix this by preserving the intended integer type for the monthly count placeholders: - replace the `np.nan` placeholder for the monthly count variables with the standard `int64` CDF fill sentinel (`-9223372036854775808`) - keep the count arrays as `np.int64` - leave the monthly rate placeholders unchanged as `NaN`, since their float metadata already matches the written data Also update the monthly IDEX tests so they: - assert the count placeholders are fully set to the integer fill value - keep the existing `NaN` expectations for the rate placeholders - verify the written monthly CDF count variables serialize as `CDF_INT8` and carry the standard integer `FILLVAL` - use context-managed `cdflib.CDF(...)` opens in the new metadata checks Validated locally with: - `/home/david/.virtualenvs/imap/bin/python -m pytest -q imap_processing/tests/idex/test_idex_l2b.py`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Preserve integer placeholder typing in the monthly IDEX count products.
This PR is a small part of the broader ISTP / metadata cleanup tracked in issue #2577. In that larger effort, this branch addresses one specific remaining file-level metadata mismatch in the monthly IDEX outputs by making the written count-variable types agree with their declared CDF metadata.
Scope:
imap_idex_l2b_sci-1moimap_idex_l2c_rectangular-map-1moWhy
The monthly IDEX products intentionally blank several derived outputs while the science team continues validating the fitting routines and derived values.
Before this change, the placeholder block applied
np.nanto both:That promoted the monthly count variables to floating-point in the written CDFs, while their metadata remained integer-style. The result was an ISTP/SPDF mismatch in the emitted files for:
counts_by_chargecounts_by_masscounts_by_charge_mapcounts_by_mass_mapSo within the broader metadata cleanup under #2577, this PR fixes the IDEX monthly count-placeholder type mismatch at the source.
What Changed
Preserve integer placeholder typing for monthly counts
For the monthly count variables only, replace the
NaNplaceholder behavior with the standardint64CDF fill sentinel:counts_by_chargecounts_by_masscounts_by_charge_mapcounts_by_mass_mapThese variables now:
np.int64-9223372036854775808as the placeholder fill valueLeave monthly rate placeholders unchanged
The monthly rate placeholders are intentionally unchanged and still use
NaN:rate_by_chargerate_by_massrate_by_charge_maprate_by_mass_mapThat remains valid because those variables are already float-typed and their metadata matches the written data.
Update tests accordingly
The monthly IDEX regression tests now:
NaNassertions for the rate placeholdersFILLVALTesting
Validated locally with: