fix: Replace static RSA FIPS cipher suites with DHE-RSA for IBM Power ppc64le - #6728
Closed
jyejare wants to merge 439 commits into
Closed
fix: Replace static RSA FIPS cipher suites with DHE-RSA for IBM Power ppc64le#6728jyejare wants to merge 439 commits into
jyejare wants to merge 439 commits into
Conversation
Signed-off-by: Srihari <svenkata@redhat.com>
Signed-off-by: Srihari <svenkata@redhat.com>
Fix Ray Offline Store Notebook tests memory issues
Add disconnected readiness workflow
Merge remote-tracking branch 'upstream/master'
* feat: add upstream and downstream cherry-pick sync workflows Extend the label-based cherry-pick automation beyond stable: - sync_upstream.yml: on `sync-to-upstream` label, creates cross-repo PR against feast-dev/feast master - sync_downstream_branch.yml: on `sync-to-rhoai-*` labels, creates PRs against downstream release branches (e.g., rhoai-3.5, rhoai-3.4-ea.1) Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address coderabbitai review feedback on sync workflows - Add persist-credentials: false on checkout to limit PAT scope - Add least-privilege permissions (contents: read) to detect-targets job - Move event-derived values into env: vars to prevent template injection - Reorder branch-existence check before git fetch (was unreachable) - Fix doc inconsistency between label table and usage examples Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com> --------- Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…#166) * feat: Multi-arch publish for feast operator image Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * feat: Allow users to have protected project on shared registry Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock (feast-dev#6648) feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Carlos Sánchez <i52saarc@uco.es> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>
…rash on IBM Power RHOAIENG-78595 # What this PR does / why we need it: The Feast offline container crashes in CrashLoopBackOff on FIPS-enabled OpenShift clusters running IBM Power (ppc64le). The existing FIPS cipher suite fix (RHOAIENG-70153) set GRPC_SSL_CIPHER_SUITES in `start_server()`, but pyarrow.flight was already imported at module level. On IBM Power, gRPC reads this env var during shared-library initialization, so the late configuration had no effect. This fix moves the FIPS cipher configuration to module level — before the pyarrow.flight import — so the env var is present when gRPC initializes its SSL context. The `_configure_grpc_fips()` call in `start_server()` is retained as a safety net. # Which issue(s) this PR fixes: Fixes RHOAIENG-78595 # Checks - [x] I've made sure the tests are passing. - [x] My PR title follows conventional commits format ## Testing Strategy - [x] Unit tests Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: aipcc-bot <aipcc-bot@redhat.com>
# What this PR does / why we need it: Move FIPS log message below logger initialization so it is emitted at INFO level instead of being silently dropped under the default WARNING threshold. Replace importlib.reload-based import ordering test with a subprocess-based test that exercises fresh Python import from scratch, ensuring pyarrow.flight is not cached in sys.modules and the GRPC_SSL_CIPHER_SUITES ordering check is genuine. # Which issue(s) this PR fixes: Fixes RHOAIENG-78595 # Checks - [x] I've made sure the tests are passing. - [x] My PR title follows conventional commits format ## Testing Strategy - [x] Unit tests Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
…d errors The gh pr create commands used --label flags referencing labels that don't exist on target repositories (feast-dev/feast, red-hat-data-services/feast), causing sync workflows to fail with "label not found in stream". Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com>
chore: Rebase with upstream master(Auto Sync Failed)
… ppc64le The _FIPS_CIPHER_SUITES list contained two static RSA cipher suites (AES128-GCM-SHA256, AES256-GCM-SHA384) that are not available on IBM Power ppc64le in FIPS mode. When gRPC encounters any invalid cipher in the colon-separated GRPC_SSL_CIPHER_SUITES string, it rejects the entire list, causing the TLS context creation to fail with "SSL routines::library has no ciphers". Replace them with DHE-RSA-AES128-GCM-SHA256 and DHE-RSA-AES256-GCM-SHA384, which are FIPS-approved on IBM Power ppc64le and also provide forward secrecy. Signed-off-by: Jitendra Yejare <jyejare@redhat.com> Signed-off-by: Jitendra Yejare <11752425+jyejare@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
Author
|
Closing - PR was opened against the wrong base. Will re-create against opendatahub-io/feast. |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6728 +/- ##
=======================================
Coverage 46.79% 46.79%
=======================================
Files 415 415
Lines 50395 50406 +11
Branches 7215 7217 +2
=======================================
+ Hits 23581 23589 +8
- Misses 25162 25164 +2
- Partials 1652 1653 +1
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
The Feast offline server fails to start on FIPS-enabled IBM Power ppc64le clusters with:
The
_FIPS_CIPHER_SUITESlist inoffline_server.pycontained two static RSA cipher suites (AES128-GCM-SHA256andAES256-GCM-SHA384) that are not available on IBM Power ppc64le in FIPS mode. When gRPC encounters any invalid cipher in the colon-separatedGRPC_SSL_CIPHER_SUITESstring, it rejects the entire list, preventing the TLS context from being created.This PR replaces those two ciphers with their DHE-RSA equivalents:
AES128-GCM-SHA256DHE-RSA-AES128-GCM-SHA256AES256-GCM-SHA384DHE-RSA-AES256-GCM-SHA384The updated cipher list (6 suites) now uses exclusively forward-secrecy key exchanges (ECDHE-RSA, ECDHE-ECDSA, DHE-RSA) with AES-GCM, all of which are FIPS-approved and confirmed available on both x86_64 and IBM Power ppc64le FIPS systems.
Which issue(s) this PR fixes:
Fixes https://redhat.atlassian.net/browse/RHOAIENG-82588
Checks
git commit -s)Testing Strategy
Made with Cursor