Skip to content

SWE and SWAPI and CoDICE: Filter extra data at day boundary for L2#3194

Merged
tech3371 merged 3 commits into
IMAP-Science-Operations-Center:devfrom
tech3371:#3182_day_bound
May 14, 2026
Merged

SWE and SWAPI and CoDICE: Filter extra data at day boundary for L2#3194
tech3371 merged 3 commits into
IMAP-Science-Operations-Center:devfrom
tech3371:#3182_day_bound

Conversation

@tech3371
Copy link
Copy Markdown
Contributor

@tech3371 tech3371 commented May 13, 2026

Change Summary

closes #3182 and #3062

Overview

Write generic function to filter out out of day boundary for instruments who have buffered data, especially needed for CoDICE, SWE and SWAPI.

File changes

Testing

@tech3371 tech3371 changed the title #3182 day bound SWE and SWAPI: Filter extra data at day boundary at L2 May 13, 2026
tech3371
tech3371 commented May 13, 2026
View reviewed changes
Comment thread imap_processing/spice/time.py
return vectorized_et_to_frac_doy(et)


def str_yyyymmdd_to_ttj2000ns(date_str: str) -> np.int64:
Copy link
Copy Markdown
Contributor Author

@tech3371 tech3371 May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lacoak21 I took your code from IDEX PR :)

Copy link
Copy Markdown
Collaborator

@vmartinez-cu vmartinez-cu May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a helpful utility : )

@vmartinez-cu
Copy link
Copy Markdown
Collaborator

vmartinez-cu commented May 14, 2026

Add CoDICE to your PR title for clarity since your update also includes changing the CoDICE L1A products

@tech3371
Copy link
Copy Markdown
Contributor Author

tech3371 commented May 14, 2026 •
edited
Loading

I tested and check that all those three instruments are getting filter correctly and we are. Here is data I used for testing CLI.

from imap_processing.cdf.utils import load_cdf


# file_path = "/Users/tech3371/git/imap_processing/data/imap/codice/l1a/2026/04/imap_codice_l1a_hi-omni_20260403_v005.cdf"
# dataset = load_cdf(file_path, to_datetime=True)
# print(dataset["epoch"].data)
# Before filtering for day boundary for CoDICE:
# ['2026-04-02T23:58:08.995452416' '2026-04-02T23:59:08.551240576'
#  '2026-04-03T00:00:08.107028608' ... '2026-04-03T23:55:10.683111552'
#  '2026-04-03T23:56:10.238899584' '2026-04-03T23:57:09.794687744']
# After filtering
# ['2026-04-03T00:00:08.107028608' '2026-04-03T00:01:07.662816768'
#  '2026-04-03T00:02:07.214224640' ... '2026-04-03T23:55:10.683111552'
#  '2026-04-03T23:56:10.238899584' '2026-04-03T23:57:09.794687744']

# SWE data day boundary filtering:
file_path = "/Users/tech3371/git/imap_processing/data/imap/swe/l1a/2026/05/imap_swe_l1a_sci_20260513_v001.cdf"
dataset = load_cdf(file_path, to_datetime=True)
print(dataset["epoch"].data)
# ['2026-05-13T00:00:14.007089024' '2026-05-13T00:00:29.007076096'
#  '2026-05-13T00:00:45.007062272' ... '2026-05-13T13:15:28.965835648'
#  '2026-05-13T13:15:43.965822592' '2026-05-13T13:15:59.965808768']

# SWAPI data day boundary filtering:
file_path = "/Users/tech3371/git/imap_processing/data/imap/swapi/l1a/2026/05/imap_swapi_l1a_hk_20260509_v001.cdf"
dataset = load_cdf(file_path, to_datetime=True)
print(dataset["epoch"].data)

# ['2026-05-13T00:00:14.007089024' '2026-05-13T00:00:29.007076096'
#  '2026-05-13T00:00:45.007062272' ... '2026-05-13T13:15:28.965835648'
#  '2026-05-13T13:15:43.965822592' '2026-05-13T13:15:59.965808768']
# ['2026-05-09T00:00:43.305854080' '2026-05-09T00:01:43.305802240'
#  '2026-05-09T00:02:43.305750272' ... '2026-05-09T23:57:43.231290752'
#  '2026-05-09T23:58:43.231238784' '2026-05-09T23:59:43.231186944']

@tech3371 tech3371 changed the title SWE and SWAPI: Filter extra data at day boundary at L2 SWE and SWAPI and CoDICE: Filter extra data at day boundary at L2 May 14, 2026
@tech3371 tech3371 changed the title SWE and SWAPI and CoDICE: Filter extra data at day boundary at L2 SWE and SWAPI and CoDICE: Filter extra data at day boundary for L2 May 14, 2026
@tech3371 tech3371 mentioned this pull request May 14, 2026
Merged
@tech3371 tech3371 requested a review from maxinelasp May 14, 2026 20:21
@tech3371 tech3371 merged commit b989f15 into IMAP-Science-Operations-Center:dev May 14, 2026
14 checks passed
@vmartinez-cu
Copy link
Copy Markdown
Collaborator

vmartinez-cu commented May 14, 2026

The logic and tests you shared make sense. It might be worth sharing a CDF with the CoDICE team to ensure that it addresses their concerns in ticket #3062

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxinelasp maxinelasp maxinelasp approved these changes

@sdhoyt sdhoyt Awaiting requested review from sdhoyt

@vmartinez-cu vmartinez-cu Awaiting requested review from vmartinez-cu

@laspsandoval laspsandoval Awaiting requested review from laspsandoval

@lacoak21 lacoak21 Awaiting requested review from lacoak21

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SWE and SWAPI: Filter extra data at day boundary at L2

3 participants

@tech3371 @vmartinez-cu @maxinelasp