Skip to content

IDEX l1b: filter upstream dependencies correctly before processing#3094

Merged
lacoak21 merged 2 commits into
IMAP-Science-Operations-Center:devfrom
lacoak21:idex_missing_impact_times
Apr 28, 2026
Merged

IDEX l1b: filter upstream dependencies correctly before processing#3094
lacoak21 merged 2 commits into
IMAP-Science-Operations-Center:devfrom
lacoak21:idex_missing_impact_times

Conversation

@lacoak21
Copy link
Copy Markdown
Contributor

@lacoak21 lacoak21 commented Apr 28, 2026

Change Summary

closes #3092

Overview

This issue is caused by this problem #2843.

Currently IDEX packets

  • contain ~1weeks worth of data but can be more in some cases
  • They can contain events from months previous

This is a pretty major problem because our infrastructure is not built to deal with this.

The issue:

Since IDEX l1b processing needs spice kernels, I added a buffer time in the upstream dependency querying logic for l1b jobs only. This ensures that we have spice coverage but also provides the job with extraneous l1a files. I thought I was handling this correctly by choosing the l1a file with the greatest epoch value. This was returning the incorrect l1a file for processing because we should really be using the file with the start date of the job. I thought these two would always be the same but Jamey noticed a case where a file with an older start date has a more recent epoch value! This is due to the jumbling of IDEX events (I think) because of the issue described in #2843. Once this issue is resolved, this will no longer be a problem but as a small fix, I can pass the correct file to l1b processing by finding the one with the same start date.

EXAMPLE:
The max epoch value for the following files will illuminate the issue

  • imap_idex_l1a_sci-1week_20251209_v005.cdf' max epoch = np.int64(818510673907323520)
  • imap_idex_l1a_sci-1week_20251218_v007.cdf' max epoch = np.int64(818509752656208384)
  • imap_idex_l1a_sci-1week_20251217_v005.cdf' max epoch = np.int64(819201872691961088

File with start date 20251217 has a greater max epoch than teh file with 20251218

File changes

imap_processing/cli.py

Fix file sorting logic

Testing

This was tested with the fie Jamey mentioned in this email.

@lacoak21 lacoak21 added this to the April 2026 milestone Apr 28, 2026
@lacoak21 lacoak21 self-assigned this Apr 28, 2026
@lacoak21 lacoak21 added the Ins: IDEX Related to the IDEX instrument label Apr 28, 2026
@lacoak21 lacoak21 merged commit 1449521 into IMAP-Science-Operations-Center:dev Apr 28, 2026
14 checks passed
@lacoak21 lacoak21 deleted the idex_missing_impact_times branch April 28, 2026 21:21
maxinelasp pushed a commit to maxinelasp/imap_processing that referenced this pull request May 1, 2026
lacoak21 added a commit to lacoak21/imap_processing that referenced this pull request May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ins: IDEX Related to the IDEX instrument

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG - IDEX events disappeared

2 participants