Skip to content

fix: Replace static RSA FIPS cipher suites with DHE-RSA for IBM Power ppc64le - #6728

Closed
jyejare wants to merge 439 commits into
feast-dev:masterfrom
opendatahub-io:ciphersuites_again
Closed

fix: Replace static RSA FIPS cipher suites with DHE-RSA for IBM Power ppc64le#6728
jyejare wants to merge 439 commits into
feast-dev:masterfrom
opendatahub-io:ciphersuites_again

Conversation

@jyejare

@jyejare jyejare commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:

The Feast offline server fails to start on FIPS-enabled IBM Power ppc64le clusters with:

SSL routines::library has no ciphers

The _FIPS_CIPHER_SUITES list in offline_server.py contained two static RSA cipher suites (AES128-GCM-SHA256 and AES256-GCM-SHA384) that are not available on IBM Power ppc64le in FIPS mode. When gRPC encounters any invalid cipher in the colon-separated GRPC_SSL_CIPHER_SUITES string, it rejects the entire list, preventing the TLS context from being created.

This PR replaces those two ciphers with their DHE-RSA equivalents:

Removed (static RSA, no forward secrecy) Replaced with (DHE-RSA, forward secrecy)
AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256
AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384

The updated cipher list (6 suites) now uses exclusively forward-secrecy key exchanges (ECDHE-RSA, ECDHE-ECDSA, DHE-RSA) with AES-GCM, all of which are FIPS-approved and confirmed available on both x86_64 and IBM Power ppc64le FIPS systems.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/RHOAIENG-82588

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests

Made with Cursor

m-rafeeq and others added 30 commits January 17, 2026 00:27
Signed-off-by: Srihari <svenkata@redhat.com>
Signed-off-by: Srihari <svenkata@redhat.com>
Signed-off-by: Srihari <svenkata@redhat.com>
Fix Ray Offline Store Notebook tests memory issues
ntkathole and others added 26 commits July 14, 2026 11:30
Add disconnected readiness workflow
Merge remote-tracking branch 'upstream/master'
* feat: add upstream and downstream cherry-pick sync workflows

Extend the label-based cherry-pick automation beyond stable:
- sync_upstream.yml: on `sync-to-upstream` label, creates cross-repo
  PR against feast-dev/feast master
- sync_downstream_branch.yml: on `sync-to-rhoai-*` labels, creates
  PRs against downstream release branches (e.g., rhoai-3.5, rhoai-3.4-ea.1)

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address coderabbitai review feedback on sync workflows

- Add persist-credentials: false on checkout to limit PAT scope
- Add least-privilege permissions (contents: read) to detect-targets job
- Move event-derived values into env: vars to prevent template injection
- Reorder branch-existence check before git fetch (was unreachable)
- Fix doc inconsistency between label table and usage examples

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com>

---------

Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…#166)

* feat: Multi-arch publish for feast operator image

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: Allow users to have protected project on shared registry

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

* fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock (feast-dev#6648)

feast_metrics.start_metrics_server() starts a feature-freshness thread
in the Gunicorn master process, before Gunicorn forks its worker(s).
That thread's first action, with no delay, is update_feature_freshness()
-> store.list_feature_views(), which lazily builds the registry for the
first time. For registry backends needing a lazy DBAPI import (e.g. the
SQL registry importing pymysql via SQLAlchemy's create_engine()), this
means a thread in the master can be mid-import, holding CPython's
per-module import lock, at the exact moment Gunicorn forks a worker.

POSIX fork() only duplicates the calling thread into the child process;
every other thread in the parent, including this one, simply ceases to
exist in the worker. If the fork lands while that thread holds a
module's import lock, the lock stays permanently held in the new
worker, since there is no longer any thread that can finish the import
and release it. The worker's own later attempt to build its registry
then deadlocks forever with no error - the process just hangs at
"Waiting for application startup." This is intermittent by nature: it
only manifests if the fork lands inside that narrow timing window.

Resource monitoring already avoids this correctly (start_resource_monitoring=
not uses_gunicorn plus the post_worker_init hook calling
init_worker_monitoring()), but the freshness thread was not given the
same treatment. This applies the identical pattern: start_metrics_server
gains a start_freshness_monitoring flag (deferred exactly like resource
monitoring's), and FeastServeApplication's post_worker_init hook now
also calls the new init_worker_freshness_monitoring(store) after the
fork, instead of feast_metrics.py starting it unconditionally beforehand.

Fixes feast-dev#6647

Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com>
Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>

---------

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com>
Co-authored-by: ntkathole <nikhilkathole2683@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Carlos Sánchez <i52saarc@uco.es>
Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>
…rash on IBM Power

RHOAIENG-78595

# What this PR does / why we need it:

The Feast offline container crashes in CrashLoopBackOff on FIPS-enabled
OpenShift clusters running IBM Power (ppc64le). The existing FIPS cipher
suite fix (RHOAIENG-70153) set GRPC_SSL_CIPHER_SUITES in `start_server()`,
but pyarrow.flight was already imported at module level. On IBM Power, gRPC
reads this env var during shared-library initialization, so the late
configuration had no effect.

This fix moves the FIPS cipher configuration to module level — before the
pyarrow.flight import — so the env var is present when gRPC initializes its
SSL context. The `_configure_grpc_fips()` call in `start_server()` is
retained as a safety net.

# Which issue(s) this PR fixes:

Fixes RHOAIENG-78595

# Checks
- [x] I've made sure the tests are passing.
- [x] My PR title follows conventional commits format

## Testing Strategy
- [x] Unit tests

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: aipcc-bot <aipcc-bot@redhat.com>
# What this PR does / why we need it:
Move FIPS log message below logger initialization so it is emitted at INFO
level instead of being silently dropped under the default WARNING threshold.
Replace importlib.reload-based import ordering test with a subprocess-based
test that exercises fresh Python import from scratch, ensuring
pyarrow.flight is not cached in sys.modules and the GRPC_SSL_CIPHER_SUITES
ordering check is genuine.

# Which issue(s) this PR fixes:
Fixes RHOAIENG-78595

# Checks
- [x] I've made sure the tests are passing.
- [x] My PR title follows conventional commits format

## Testing Strategy
- [x] Unit tests

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
…d errors

The gh pr create commands used --label flags referencing labels that don't
exist on target repositories (feast-dev/feast, red-hat-data-services/feast),
causing sync workflows to fail with "label not found in stream".

Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com>
chore: Rebase with upstream master(Auto Sync Failed)
… ppc64le

The _FIPS_CIPHER_SUITES list contained two static RSA cipher suites
(AES128-GCM-SHA256, AES256-GCM-SHA384) that are not available on IBM
Power ppc64le in FIPS mode. When gRPC encounters any invalid cipher in
the colon-separated GRPC_SSL_CIPHER_SUITES string, it rejects the
entire list, causing the TLS context creation to fail with
"SSL routines::library has no ciphers".

Replace them with DHE-RSA-AES128-GCM-SHA256 and
DHE-RSA-AES256-GCM-SHA384, which are FIPS-approved on IBM Power
ppc64le and also provide forward secrecy.

Signed-off-by: Jitendra Yejare <jyejare@redhat.com>
Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@jyejare
jyejare requested a review from a team as a code owner August 11, 2026 11:58
@jyejare

jyejare commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

Closing - PR was opened against the wrong base. Will re-create against opendatahub-io/feast.

@jyejare jyejare closed this Aug 11, 2026
@codecov-commenter

codecov-commenter commented Aug 11, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 90.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.79%. Comparing base (39d408d) to head (ee1d6bb).

Files with missing lines Patch % Lines
sdk/python/feast/metrics.py 0.00% 1 Missing ⚠️
.../feast/permissions/auth/kubernetes_token_parser.py 94.73% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #6728   +/-   ##
=======================================
  Coverage   46.79%   46.79%           
=======================================
  Files         415      415           
  Lines       50395    50406   +11     
  Branches     7215     7217    +2     
=======================================
+ Hits        23581    23589    +8     
- Misses      25162    25164    +2     
- Partials     1652     1653    +1     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.12% <90.00%> (+<0.01%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/offline_server.py 29.39% <ø> (ø)
sdk/python/feast/metrics.py 75.00% <0.00%> (-0.52%) ⬇️
.../feast/permissions/auth/kubernetes_token_parser.py 41.00% <94.73%> (+2.90%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39d408d...ee1d6bb. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.