Skip to content

BUG - GLOWS L1B SPICE "insufficent ephemeris data" #2900

@maxinelasp

Description

@maxinelasp

Description of the issue

Some GLOWS jobs are failing due to insufficient ephemeris data.

Once

I am wondering if this is an issue related to Tim's comment on #2871 about timestamps?

This blocks development of GLOWS L3!

Steps to reproduce the issue

L1B histogram timestamps effected:

  2026-03-25T00:00:00  
  2026-03-25T00:00:00  
  2026-03-25T00:00:00  
  2026-03-24T00:00:00  
  2026-01-27T00:00:00  
  2026-03-10T00:00:00  
  2026-03-16T00:00:00  
  2026-02-27T00:00:00  
  2026-01-28T00:00:00  
  2026-03-23T00:00:00  
  2026-03-24T00:00:00  
  2026-03-25T00:00:00  
  2026-02-28T00:00:00  
  2026-03-19T00:00:00  
  2026-03-14T00:00:00  
  2026-03-05T00:00:00  
  2026-03-06T00:00:00  
  2026-03-04T00:00:00  
  2026-03-22T00:00:00  
  2026-03-03T00:00:00  
  2026-03-21T00:00:00  
  2026-03-15T00:00:00  
  2026-03-02T00:00:00  
  2026-03-20T00:00:00  
  2026-01-29T00:00:00  
  2026-03-01T00:00:00  
  2026-01-27T00:00:00  
  2026-03-10T00:00:00  
  2026-03-11T00:00:00  
  2026-03-12T00:00:00  
  2026-03-18T00:00:00  
  2026-03-17T00:00:00  
  2026-03-13T00:00:00  
  2026-03-09T00:00:00  
  2026-03-08T00:00:00  
  2026-03-07T00:00:00  
  2026-02-15T00:00:00  
  2026-02-16T00:00:00  
  2026-02-25T00:00:00  
  2026-02-13T00:00:00  
  2026-02-17T00:00:00  
  2026-02-14T00:00:00  
  2026-02-24T00:00:00  
  2026-02-21T00:00:00  
  2026-02-23T00:00:00  
  2026-02-26T00:00:00  
  2026-02-22T00:00:00  
  2026-02-19T00:00:00  
  2026-02-18T00:00:00  
  2026-02-03T00:00:00  
  2026-02-20T00:00:00  
  2026-02-12T00:00:00  
  2026-02-06T00:00:00  
  2026-02-08T00:00:00  
  2026-02-07T00:00:00  
  2026-01-30T00:00:00  
  2026-02-11T00:00:00  
  2026-02-09T00:00:00  
  2026-02-10T00:00:00  
  2026-02-05T00:00:00  
  2026-02-04T00:00:00  
  2026-01-25T00:00:00  
  2026-02-01T00:00:00  
  2026-01-26T00:00:00  
  2026-01-23T00:00:00  
  2026-01-20T00:00:00  
  2026-01-21T00:00:00  
  2026-01-22T00:00:00  
  2026-01-31T00:00:00  
  2026-01-19T00:00:00  
  2026-01-18T00:00:00  
  2026-01-17T00:00:00  
  2025-12-31T00:00:00  
  2026-01-16T00:00:00  
  2026-01-01T00:00:00  
  2026-01-14T00:00:00  
  2025-12-25T00:00:00  
  2026-01-13T00:00:00  
  2025-12-22T00:00:00  
  2025-12-21T00:00:00  
  2025-12-28T00:00:00  
  2025-12-27T00:00:00  
  2025-12-20T00:00:00  
  2025-12-24T00:00:00  
  2025-12-30T00:00:00  
  2025-12-23T00:00:00  
  2025-12-06T00:00:00  
  2025-12-26T00:00:00  
  2025-12-07T00:00:00  
  2025-12-29T00:00:00  
  2025-12-05T00:00:00  
  2025-12-15T00:00:00  
  2025-12-09T00:00:00  
  2025-12-18T00:00:00  
  2025-12-10T00:00:00  
  2025-12-11T00:00:00  
  2025-12-14T00:00:00  
  2025-12-08T00:00:00  
  2025-12-13T00:00:00  
  2025-12-12T00:00:00  

Expected vs Actual behavior

No response

Code Snippet (If applicable)

Processing job details
Processing log
Job properties

2026-04-01 01:36:28 - INFO:imap_processing.cli:Interpreting dependency argument as a JSON file: imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json
2026-04-01 01:36:28 - INFO:imap_data_access.io:Downloading file imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json from https://api.imap-mission.com/api-key/download/imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json to /app/data/imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json
2026-04-01 01:36:29 - INFO:imap_data_access.io:File /app/data/imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json downloaded successfully
2026-04-01 01:36:29 - INFO:imap_processing.cli:IMAP Processing Version: 1.0.25.post18.dev0+b0f1dc4
2026-04-01 01:36:29 - INFO:imap_processing.cli:Processing Glows level l1b
2026-04-01 01:36:29 - INFO:imap_processing.cli:Beginning preprocessing (download dependencies)
2026-04-01 01:36:29 - INFO:imap_data_access.io:Downloading file imap/spice/spin/imap_2026_083_2026_084_01.spin from https://api.imap-mission.com/api-key/download/imap/spice/spin/imap_2026_083_2026_084_01.spin to /app/data/imap/spice/spin/imap_2026_083_2026_084_01.spin
2026-04-01 01:36:29 - INFO:imap_data_access.io:File /app/data/imap/spice/spin/imap_2026_083_2026_084_01.spin downloaded successfully
2026-04-01 01:36:29 - INFO:imap_data_access.io:Downloading file imap/spice/spin/imap_2026_084_2026_085_01.spin from https://api.imap-mission.com/api-key/download/imap/spice/spin/imap_2026_084_2026_085_01.spin to /app/data/imap/spice/spin/imap_2026_084_2026_085_01.spin
2026-04-01 01:36:30 - INFO:imap_data_access.io:File /app/data/imap/spice/spin/imap_2026_084_2026_085_01.spin downloaded successfully
2026-04-01 01:36:30 - INFO:imap_data_access.io:Downloading file imap/spice/spin/imap_2026_085_2026_086_01.spin from https://api.imap-mission.com/api-key/download/imap/spice/spin/imap_2026_085_2026_086_01.spin to /app/data/imap/spice/spin/imap_2026_085_2026_086_01.spin
2026-04-01 01:36:31 - INFO:imap_data_access.io:File /app/data/imap/spice/spin/imap_2026_085_2026_086_01.spin downloaded successfully
2026-04-01 01:36:31 - INFO:imap_data_access.io:Downloading file imap/spice/lsk/naif0012.tls from https://api.imap-mission.com/api-key/download/imap/spice/lsk/naif0012.tls to /app/data/imap/spice/lsk/naif0012.tls
2026-04-01 01:36:32 - INFO:imap_data_access.io:File /app/data/imap/spice/lsk/naif0012.tls downloaded successfully
2026-04-01 01:36:32 - INFO:imap_data_access.io:Downloading file imap/spice/fk/imap_130.tf from https://api.imap-mission.com/api-key/download/imap/spice/fk/imap_130.tf to /app/data/imap/spice/fk/imap_130.tf
2026-04-01 01:36:32 - INFO:imap_data_access.io:File /app/data/imap/spice/fk/imap_130.tf downloaded successfully
2026-04-01 01:36:32 - INFO:imap_data_access.io:Downloading file imap/spice/fk/imap_science_120.tf from https://api.imap-mission.com/api-key/download/imap/spice/fk/imap_science_120.tf to /app/data/imap/spice/fk/imap_science_120.tf
2026-04-01 01:36:33 - INFO:imap_data_access.io:File /app/data/imap/spice/fk/imap_science_120.tf downloaded successfully
2026-04-01 01:36:33 - INFO:imap_data_access.io:Downloading file imap/spice/sclk/imap_sclk_0145.tsc from https://api.imap-mission.com/api-key/download/imap/spice/sclk/imap_sclk_0145.tsc to /app/data/imap/spice/sclk/imap_sclk_0145.tsc
2026-04-01 01:36:33 - INFO:imap_data_access.io:File /app/data/imap/spice/sclk/imap_sclk_0145.tsc downloaded successfully
2026-04-01 01:36:33 - INFO:imap_data_access.io:Downloading file imap/spice/spk/de440.bsp from https://api.imap-mission.com/api-key/download/imap/spice/spk/de440.bsp to /app/data/imap/spice/spk/de440.bsp
2026-04-01 01:36:35 - INFO:imap_data_access.io:File /app/data/imap/spice/spk/de440.bsp downloaded successfully
2026-04-01 01:36:35 - INFO:imap_data_access.io:Downloading file imap/spice/spk/imap_recon_20250925_20260325_v01.bsp from https://api.imap-mission.com/api-key/download/imap/spice/spk/imap_recon_20250925_20260325_v01.bsp to /app/data/imap/spice/spk/imap_recon_20250925_20260325_v01.bsp
2026-04-01 01:36:37 - INFO:imap_data_access.io:File /app/data/imap/spice/spk/imap_recon_20250925_20260325_v01.bsp downloaded successfully
2026-04-01 01:36:37 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_2025_358_2026_085_002.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_2025_358_2026_085_002.ah.bc to /app/data/imap/spice/ck/imap_2025_358_2026_085_002.ah.bc
2026-04-01 01:36:47 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_2025_358_2026_085_002.ah.bc downloaded successfully
2026-04-01 01:36:47 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_2026_084_2026_091_001.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_2026_084_2026_091_001.ah.bc to /app/data/imap/spice/ck/imap_2026_084_2026_091_001.ah.bc
2026-04-01 01:36:48 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_2026_084_2026_091_001.ah.bc downloaded successfully
2026-04-01 01:36:48 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc to /app/data/imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc
2026-04-01 01:36:49 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc downloaded successfully
2026-04-01 01:36:49 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc to /app/data/imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc
2026-04-01 01:36:49 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc downloaded successfully
2026-04-01 01:36:49 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc to /app/data/imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc
2026-04-01 01:36:50 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc downloaded successfully
2026-04-01 01:36:50 - INFO:imap_data_access.io:Downloading file imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf from https://api.imap-mission.com/api-key/download/imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf to /app/data/imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf
2026-04-01 01:36:50 - INFO:imap_data_access.io:File /app/data/imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf downloaded successfully
2026-04-01 01:36:50 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat
2026-04-01 01:36:51 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat downloaded successfully
2026-04-01 01:36:51 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat
2026-04-01 01:36:51 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat downloaded successfully
2026-04-01 01:36:51 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat
2026-04-01 01:36:52 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat downloaded successfully
2026-04-01 01:36:52 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json to /app/data/imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json
2026-04-01 01:36:52 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json downloaded successfully
2026-04-01 01:36:52 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat
2026-04-01 01:36:53 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat downloaded successfully
2026-04-01 01:36:53 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json to /app/data/imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json
2026-04-01 01:36:53 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json downloaded successfully
2026-04-01 01:36:53 - INFO:imap_processing.cli:Furnishing kernels: ['naif0012.tls', 'imap_130.tf', 'imap_science_120.tf', 'imap_sclk_0145.tsc', 'de440.bsp', 'imap_recon_20250925_20260325_v01.bsp', 'imap_2025_358_2026_085_002.ah.bc', 'imap_2026_084_2026_091_001.ah.bc', 'imap_dps_2025_359_2026_084_001.ah.bc', 'imap_dps_2025_359_2026_087_003.ah.bc', 'imap_dps_2026_085_2026_089_001.ah.bc']
2026-04-01 01:36:53 - INFO:imap_processing.spice.spin:Using the following spin-tables in processing: ['imap_2026_083_2026_084_01.spin', 'imap_2026_084_2026_085_01.spin', 'imap_2026_085_2026_086_01.spin']
2026-04-01 01:36:53 - INFO:imap_processing.cli:Beginning actual processing
/usr/local/lib/python3.12/site-packages/imap_processing/ancillary/ancillary_dataset_combiner.py:366: UserWarning: loadtxt: input contained no data: "/app/data/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat"
  data = np.loadtxt(filepath, comments="#")
/usr/local/lib/python3.12/site-packages/imap_processing/ancillary/ancillary_dataset_combiner.py:389: UserWarning: Input line 1 contained no data and will not be counted towards `max_rows=50000`.  This differs from the behaviour in NumPy <=1.22 which counted lines rather than rows.  If desired, the previous behaviour can be achieved by using `itertools.islice`.
Please see the 1.23 release notes for an example on how to do this.  If you wish to ignore this warning, use `warnings.filterwarnings`.  This warning is expected to be removed in the future and is given only once per `loadtxt` call.
  data = np.loadtxt(filepath, comments="#", dtype=str)
Traceback (most recent call last):
  File "/usr/local/bin/imap_cli", line 8, in <module>
Processing GLOWS l1b
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/imap_processing/cli.py", line 1705, in main
    instrument.process()
  File "/usr/local/lib/python3.12/site-packages/imap_processing/cli.py", line 447, in process
    products = self.do_processing(dependencies)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/imap_processing/cli.py", line 727, in do_processing
    glows_l1b(
  File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b.py", line 78, in glows_l1b
    output_dataarrays = process_histogram(
                        ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b.py", line 304, in process_histogram
    l1b_fields = xr.apply_ufunc(
                 ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/computation/apply_ufunc.py", line 1267, in apply_ufunc
    return apply_dataarray_vfunc(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/computation/apply_ufunc.py", line 312, in apply_dataarray_vfunc
    result_var = func(*data_vars)
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/computation/apply_ufunc.py", line 820, in apply_variable_ufunc
    result_data = func(*input_data)
                  ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2518, in __call__
    return self._call_as_normal(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2511, in _call_as_normal
    return self._vectorize_call(func=func, args=vargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2592, in _vectorize_call
    res = self._vectorize_call_with_signature(func, args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2630, in _vectorize_call_with_signature
    results = func(*(arg[index] for arg in args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b.py", line 300, in create_histogram_l1b
    return HistogramL1B(  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 20, in __init__
  File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b_data.py", line 850, in __post_init__
    self.update_spice_parameters()
  File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b_data.py", line 956, in update_spice_parameters
    imap_state = geometry.imap_state(
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/imap_processing/spice/geometry.py", line 141, in imap_state
    state, _ = spiceypy.spkezr(
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/spiceypy/spiceypy.py", line 148, in with_errcheck
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/spiceypy/spiceypy.py", line 12917, in spkezr
    check_for_spice_error(None)
  File "/usr/local/lib/python3.12/site-packages/spiceypy/spiceypy.py", line 132, in check_for_spice_error
    raise dynamically_instantiate_spiceyerror(
spiceypy.utils.exceptions.SpiceSPKINSUFFDATA: 
================================================================================
Toolkit version: CSPICE_N0067
SPICE(SPKINSUFFDATA) --
Insufficient ephemeris data has been loaded to compute the state of -43 (IMAP) relative to 10 (SUN) at the ephemeris epoch 2026 MAR 25 10:02:53.932.
spkezr_c --> SPKEZR --> SPKEZ --> SPKGEO
================================================================================

Additional notes, affected areas, and suggested fixes

No response

Metadata

Metadata

Assignees

Labels

Data Release 1Deadline: June 15, 2026Ins: GLOWSRelated to the GLOWS instrumentPlannedTasks originally committed for the monthRelease BlockerNeeds to be addressed before (next) public releasebugSomething isn't working

Type

No fields configured for Bug.

Projects

Status

Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions