Bytewax Materializer Security Context Capabilities #3572

Expected Behavior

Running the Bytewax materializer in a scaled and secured Kubernetes cluster should work, successfully running the materializer job.

Current Behavior

In a managed, secured cluster, setting the securityContext capabilities may not be permitted, as is enabled by default. This causes the materialization job/s to fail.

Steps to reproduce

Run the Bytewax materializer in an environment where securityContext capabilities cannot be specified/overriden in jobs.

Specifications

• Version: 0.30.2
• Platform: EKS
• Subsystem: Python, Bytewax, Snowflake, Dynamodb

Possible Solution

Add an optional boolean configuration parameter to the Bytewax materialization engine to dictate whether or not the (current default) security context options are included in the job spec. If set to False set the securityContext capabilities to None.