Skip to content

Fix Kubernetes TCP keepalive not being applied with urllib3 2.x - #74921

Open
shahar1 wants to merge 4 commits into
apache:mainfrom
shahar1:fix-k8s-tcp-keepalive-socket-options
Open

shahar1 wants to merge 4 commits into
apache:mainfrom
shahar1:fix-k8s-tcp-keepalive-socket-options

Conversation

@shahar1

@shahar1 shahar1 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

closes: #68396
replaces: #68557

This PR makes [kubernetes_executor] enable_tcp_keepalive effective for all synchronous Kubernetes clients by setting Configuration.socket_options, preserving existing options with urllib3 2.x.

AI Summary

[kubernetes_executor] enable_tcp_keepalive defaults to True, but it has had no effect since urllib3 2.x. The provider changed HTTPConnection.default_socket_options after import, but urllib3 2.x binds that list as a default argument of HTTPConnection.__init__ when the module is imported, so new connections never see the change. When a cloud load balancer or firewall silently drops idle connections to the Kubernetes API server, they stay in the pool in CLOSE_WAIT (see #68396).

This sets the options on the Kubernetes client's Configuration.socket_options instead. The Kubernetes client passes them to urllib3's pool manager (including the proxy path). Every sync client gets them: get_kube_client() (KubernetesExecutor and the pod watcher), every branch of KubernetesHook.get_conn(), and the GKE hook.

  • The keepalive options are merged into any socket_options the user already set, or into urllib3's defaults, so TCP_NODELAY is kept. Calling it twice adds no duplicates.
  • enable_tcp_keepalive(configuration) is now public because the google provider imports it. _enable_tcp_keepalive() stays as a deprecated wrapper. The google provider's dependency on cncf.kubernetes is marked # use next version.
  • kubernetes 35.0.0, which the current floor still allows, has no Configuration.socket_options. There a warning is logged and keepalive stays off, as it effectively already was.
  • The async (kubernetes_asyncio) clients are not changed.

I checked this on a local kubeadm cluster by reading getsockopt on the pooled sockets after a real API call. On main, every path above has SO_KEEPALIVE=0 with kernel defaults (TCP_KEEPIDLE=7200). With this change, SO_KEEPALIVE=1 and the configured tcp_keep_idle/tcp_keep_intvl/tcp_keep_cnt are used. disable_tcp_keepalive / enable_tcp_keepalive=False still leave them off.

This takes over #68557, which became inactive. The original commit and its authorship are kept, with review fixes in a follow-up commit.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

anmolxlight and others added 2 commits October 17, 2026 16:34
…n cncf-kubernetes

In urllib3 2.x, HTTPConnection.__init__ binds socket_options to
default_socket_options as a default argument at import time, so
reassigning the class attribute later never reaches new connections and
[kubernetes_executor] enable_tcp_keepalive was silently a no-op.

Pass the options through the Kubernetes client's Configuration.socket_options,
which RESTClientObject forwards to urllib3.PoolManager.

Closes: apache#68396
Supersedes: apache#68557
- Mark the google cncf.kubernetes floor with 'use next version'.
- enable_tcp_keepalive: drop the urllib3 monkey-patch fallback; warn and return when Configuration has no socket_options.
- Merge keepalive options into existing socket_options (or urllib3 defaults) without duplicates.
- Deprecated _enable_tcp_keepalive() now sets the default Configuration.
- Tidy docstrings and revert the spelling_wordlist additions.
- Add tests for merging, idempotency, unsupported clients, get_kube_client, _TimeoutK8sApiClient, the deprecated helper and the hook; use autospec in hook tests.
@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues provider:google Google (including GCP) related issues labels Oct 17, 2026
Configuration.socket_options, the only way to apply keepalive options with urllib3 2.x, does not exist in kubernetes 35.0.0, so on that floor keepalive could only ever be logged as unavailable and the lowest-dependency tests failed on it.
Comment thread providers/cncf/kubernetes/tests/unit/cncf/kubernetes/test_client.py Outdated
Comment thread providers/cncf/kubernetes/tests/unit/cncf/kubernetes/test_client.py Outdated
Cover custom tcp_keep_idle/intvl/cnt values, and exercise
_TimeoutK8sApiClient with and without a supplied configuration and
disable_verify_ssl, asserting the global default stays unchanged.

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending green CI.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kubernetes provider kubernetes_executor.enable_tcp_keepalive does not actually enable TCP keepalives

2 participants