SWE: L1B to L2 flag addition by vineetbansal · Pull Request #3059 · IMAP-Science-Operations-Center/imap_processing

vineetbansal · 2026-04-21T00:03:10Z

Change Summary

Addresses issue #2950

Overview

Added a per-epoch SWE L1B quality flag indicating when in-flight calibration uses the final calibration interval (agreed to be set "far" into the future).

File changes

imap_processing/cdf/config/imap_swe_l1b_variable_attrs.yaml - Added CDF metadata for the new inflight_cal_flags variable.
imap_processing/quality_flags.py - Added an SweL1bFlags enum for SWE L1B quality flags.
imap_processing/swe/l1b/swe_l1b.py - Returns per-epoch inflight calibration flags and adds them to the L1B dataset.

Testing

imap_processing/tests/swe/lut/imap_swe_l1b-in-flight-cal_20240510_20260716_v000.csv - Added an additional calibration time entry needed to test “last interval” behavior. The second to last entry encompasses acquisition time tested in existing tests, so no changes are needed to existing tests.
imap_processing/tests/swe/test_swe_l1b.py - Added a new unit test for the flag logic.
imap_processing/tests/swe/test_swe_l2.py - Added an assertion to an existing test that L1B epochs are not flagged for the L2 test data (since all tested data is outside the last cal interval)

…last-interval interpolation was used for in-flight calibration of counts.

tech3371 · 2026-04-21T15:36:29Z

@laspsandoval can you double check if I-ALiRT will be impacted by this change because I remember you were using this function for it too

Copilot

Pull request overview

Adds a new SWE L1B per-epoch quality flag that indicates when in-flight calibration is being interpolated using the final (“far future”) calibration interval, with accompanying metadata and unit test updates.

Changes:

Introduces SweL1bFlags and adds inflight_cal_flags to the SWE L1B dataset as a per-epoch bitmask.
Updates SWE in-flight calibration application to return both corrected counts and per-epoch flags.
Extends SWE test LUT and adds/updates tests to exercise and assert the new flag behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
imap_processing/swe/l1b/swe_l1b.py	Returns per-epoch in-flight cal flags from calibration application and writes them into the L1B dataset.
imap_processing/quality_flags.py	Adds the `SweL1bFlags` IntFlag enum for SWE L1B.
imap_processing/cdf/config/imap_swe_l1b_variable_attrs.yaml	Adds CDF metadata for the new `inflight_cal_flags` variable.
imap_processing/tests/swe/lut/imap_swe_l1b-in-flight-cal_20240510_20260716_v000.csv	Adds an additional calibration entry to enable “last interval” test coverage.
imap_processing/tests/swe/test_swe_l1b.py	Updates existing calibration test for new return signature and adds a new unit test for flag logic.
imap_processing/tests/swe/test_swe_l2.py	Adds an assertion that L1B epochs are not flagged for the L2 test data.

Comments suppressed due to low confidence (1)

imap_processing/tests/swe/test_swe_l1b.py:87

test_in_flight_calibration_factor still calls apply_in_flight_calibration() with counts shaped (N_ESA_STEPS, N_ANGLE_SECTORS, N_CEMS) and acquisition_time shaped (N_ESA_STEPS, N_ANGLE_SECTORS), but the function now returns per-epoch flags and documents an epoch axis. Updating this test to use an explicit single-epoch shape (e.g., add a leading epoch dimension and then index [0] for assertions) will keep it aligned with the updated API and avoid silently exercising the wrong flag-shape behavior.

    calibrated_count, _ = apply_in_flight_calibration(
        one_full_cycle_data,
        acquisition_time,
        in_flight_cal_files,
    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tmplummer

This looks good to me. Just two things worth checking on.

laspsandoval

Updates need to happen so this doesn't break I-ALiRT code.

tmplummer

I believe that @vineetbansal has responded to all review comments and addressed needed changes. Thanks!

tech3371

Minor comments!

tech3371 · 2026-04-24T15:05:00Z

@vineetbansal once this PR is merge, it will be good to check with Ruth or Jon if that's file contains what they expected. Let me or @jtniehof know if you need help.

My initial recommendation is to go to this line, https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dev/imap_processing/tests/swe/test_swe_l2.py#L449 and save L2 file to your local machine and sent to Ruth and Menlo for validation. That test will produce file with your changes in it. If they say it looks good, then we can push code to prod.

tech3371

minor rename of CDF variable based on discussion in email thread.

I like this idea and will make adding other flags easier in the future if we make variable name general.

tech3371 · 2026-04-28T17:29:39Z

  FORMAT: I5
  VALIDMAX: 65535

+inflight_cal_flags:


Suggested change

inflight_cal_flags:

data_quality_flags:

tech3371 · 2026-04-28T17:30:13Z

+    In-flight calibration quality flags. Bit 2 (LAST_CAL_INTERVAL): set if any
+    acquisition time in this cycle was extrapolated using the last two entries
+    in the calibration table.
+  FIELDNAM: In-Flight Calibration Flags


Suggested change

FIELDNAM: In-Flight Calibration Flags

FIELDNAM: SWE data quality flags

tech3371 · 2026-04-28T17:31:44Z

+    In-flight calibration quality flags. Bit 2 (LAST_CAL_INTERVAL): set if any
+    acquisition time in this cycle was extrapolated using the last two entries
+    in the calibration table.


Vineet, can you update description and VAR_NOTES, similar to what I did for SWAPI, https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dev/imap_processing/cdf/config/imap_swapi_variable_attrs.yaml#L52.

tech3371 · 2026-04-28T17:32:59Z

  VAR_TYPE: support_data
  DICT_KEY: SPASE>Support>SupportQuantity:Temporal,Qualifier:Array
+
+inflight_cal_flags:


same update here

tech3371 · 2026-04-28T17:34:04Z

        attrs=cdf_attrs.get_variable_attributes("esa_energy"),
    )

+    science_dataset["inflight_cal_flags"] = xr.DataArray(


Suggested change

science_dataset["inflight_cal_flags"] = xr.DataArray(

science_dataset["data_quality_flags"] = xr.DataArray(

vineetbansal · 2026-04-29T17:25:18Z

@tech3371 - I've made the changes. I'm using data_quality instead of data_quality_flags like you suggested because that's the name of an existing flag for codice that they're using, so I wanted to avoid making a new name. Hope that's okay - if you really want data_quality_flags let me know and I'll make the change.

Also added VAR_NOTES.

tech3371 · 2026-04-29T17:37:11Z

@tech3371 - I've made the changes. I'm using data_quality instead of data_quality_flags like you suggested because that's the name of an existing flag for codice that they're using, so I wanted to avoid making a new name. Hope that's okay - if you really want data_quality_flags let me know and I'll make the change.

Also added VAR_NOTES.

What you had is good. I was going to generic name too!

vineetbansal added 2 commits April 20, 2026 19:53

Added a flag for SWE l1b processing (epoch level) indicating whether …

fad863a

…last-interval interpolation was used for in-flight calibration of counts.

Merge branch 'dev' into vb/issue2950

ba845ab

vineetbansal mentioned this pull request Apr 21, 2026

SWE: L1B to L2 flag addition #2950

Closed

vineetbansal added 2 commits April 20, 2026 20:05

pre-commit checked

ba1f5e8

using bitwise comparison to allow future additions to SweL1bFlags

16974e0

tech3371 requested review from Copilot, laspsandoval and tmplummer April 21, 2026 15:35

Copilot started reviewing on behalf of tech3371 April 21, 2026 15:36 View session

tech3371 self-requested a review April 21, 2026 15:38

tech3371 assigned vineetbansal Apr 21, 2026

tech3371 added the Ins: SWE Related to the SWE instrument label Apr 21, 2026

Copilot AI reviewed Apr 21, 2026

View reviewed changes

tweaks for CoPilot feedback on PR 3059

df53bde

tmplummer reviewed Apr 21, 2026

View reviewed changes

Comment thread imap_processing/quality_flags.py

Comment thread imap_processing/cdf/config/imap_swe_l2_variable_attrs.yaml Outdated

laspsandoval reviewed Apr 21, 2026

View reviewed changes

Comment thread imap_processing/swe/l1b/swe_l1b.py

added SPASE entry for inflight_cal_flags for swe l2

b138ad7

tmplummer approved these changes Apr 22, 2026

View reviewed changes

tech3371 approved these changes Apr 24, 2026

View reviewed changes

Comment thread imap_processing/cdf/config/imap_swe_l1b_variable_attrs.yaml Outdated

Comment thread imap_processing/swe/l1b/swe_l1b.py Outdated

Comment thread imap_processing/quality_flags.py Outdated

minor changes based on PR feedback

bda5893

tmplummer reviewed Apr 24, 2026

View reviewed changes

Comment thread imap_processing/cdf/config/imap_swe_l1b_variable_attrs.yaml Outdated

tmplummer reviewed Apr 24, 2026

View reviewed changes

Comment thread imap_processing/cdf/config/imap_swe_l2_variable_attrs.yaml Outdated

interpolated -> extrapolated (last cal line is made up)

476f0d2

tech3371 approved these changes Apr 28, 2026

View reviewed changes

Renamed variable to data_quality; VAR_NOTES introduced

2eaf4a5

tech3371 merged commit f2be49d into IMAP-Science-Operations-Center:dev Apr 29, 2026
14 checks passed

maxinelasp pushed a commit to maxinelasp/imap_processing that referenced this pull request May 1, 2026

SWE: L1B to L2 flag addition (IMAP-Science-Operations-Center#3059)

e38d7ac

lacoak21 pushed a commit to lacoak21/imap_processing that referenced this pull request May 4, 2026

SWE: L1B to L2 flag addition (IMAP-Science-Operations-Center#3059)

591258d

	FIELDNAM: In-Flight Calibration Flags
	FIELDNAM: SWE data quality flags

	science_dataset["inflight_cal_flags"] = xr.DataArray(
	science_dataset["data_quality_flags"] = xr.DataArray(

Conversation

vineetbansal commented Apr 21, 2026

Change Summary

Overview

File changes

Testing

Uh oh!

tech3371 commented Apr 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tmplummer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

laspsandoval left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tmplummer left a comment

Choose a reason for hiding this comment

Uh oh!

tech3371 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tech3371 commented Apr 24, 2026

Uh oh!

Uh oh!

Uh oh!

tech3371 left a comment

Choose a reason for hiding this comment

Uh oh!

tech3371 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

tech3371 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

tech3371 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

tech3371 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

tech3371 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

vineetbansal commented Apr 29, 2026

Uh oh!

tech3371 commented Apr 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants