Skip to content

feat: virtual thread support - #3664

Open
csviri wants to merge 35 commits into
nextfrom
virtual-threads-and-shared-executors
Open

csviri wants to merge 35 commits into
nextfrom
virtual-threads-and-shared-executors

Conversation

@csviri

@csviri csviri commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

feat: virtual thread support

Adds a single flag that switches the framework's concurrent work over to virtual threads:

Operator operator = new Operator(override -> override.withUseVirtualThreads(true));

It can also be set with josdk.use-virtual-threads. It is off by default, so nothing changes for
existing operators.

Why

Reconcilers spend almost all their time blocked on the Kubernetes API server or on external
systems. Virtual threads make that kind of blocking cheap, so concurrency limits can be raised far
above what is reasonable with platform threads, without sizing and tuning thread pools.

What changes when the flag is on

Area Behavior
Reconciliations Run on virtual threads. withConcurrentReconciliationThreads(n) still caps how many run at once (enforced with a Semaphore).
Dependent workflows Same: virtual threads, still capped by withConcurrentWorkflowExecutorThreads(n).
Internal housekeeping The unbounded caching executor (starting informers, etc.) uses a virtual thread per task.
Scheduled tasks Polling event sources and retried or rescheduled reconciliations run on virtual threads. See the next section.
Kubernetes client The default client created by the framework runs its internal task executor (informer event dispatch, watch event delivery) on virtual threads. A client provided by the user is used as is.
JVM lifetime Virtual threads are always daemon threads, so they don't keep the JVM alive. If main returns after starting the operator and nothing else keeps the JVM running, call Operator.keepAlive() (already available, see the docs). The framework does not start such a thread on its own.

Java 25 only

Virtual threads are supported on Java 25 and later (JEP 491
removed the carrier pinning inside synchronized blocks, which the codebase uses extensively; on
Java 21 to 24 it can starve the carrier pool and deadlock startup). This is checked at runtime: on
an older JVM a warning is logged and platform threads are used, so one configuration works on every
Java version the operator runs on. The compilation target stays Java 17.

Scheduled executors (the larger part of this PR)

Previously PollingEventSource and TimerEventSource each created a java.util.Timer, and
PerResourcePollingEventSource created its own ScheduledThreadPoolExecutor. Each cost a thread
that the operator neither sized nor shut down. They now use scheduled executors that are created
and shut down in a defined way:

  • Polling event sources get an executor of their own. PollingEventSource and
    PerResourcePollingEventSource do not share an executor, so a slow poll of one source never
    delays another. Unless one is provided with withExecutorService on the configuration builder,
    the source creates a single-thread daemon executor when it starts and shuts it down when it stops
    (a new one on every start). With useVirtualThreads on, that executor runs the polls on virtual
    threads. A provided executor is used as is, is never shut down by the source, and may be shared
    deliberately.
  • ConfigurationService.getRetryAndRescheduleExecutorService() backs the retry and reschedule
    timer of every controller. It is managed by ExecutorServiceManager, sized with
    concurrentRetryAndRescheduleThreads() (default 2) or
    josdk.retry-and-reschedule.concurrent-threads, and can be replaced through
    ConfigurationServiceOverrider. The operator shuts it down on stop and creates a new one when
    started again.
  • The default pools use daemon threads, drop tasks scheduled for later on shutdown (so far-out
    reschedules don't delay termination), remove cancelled tasks eagerly, and let idle threads time
    out after 60 seconds.
  • Event sources resolve their executor on every start() rather than at construction. The polling
    and timer sources let a poll or handler already in flight finish when stopped, as the Timer
    they replace did, and the timer source re-arms its pending timers on the operator's new executor
    when the operator is started again.
  • ExecutorServiceManager.scheduledExecutorService() stays available for compatibility,
    deprecated, as a general purpose executor created on first use. The framework itself no longer
    uses it.

VirtualThreadScheduledExecutor

The JDK has no scheduled executor backed by virtual threads, so with the flag on the defaults are a
new package-private VirtualThreadScheduledExecutor: one daemon platform thread does the timing
and hands each fired task to a virtual-thread-per-task executor.

  • The number of scheduled tasks running at once is not limited in this mode, because they are
    typically blocking calls. withConcurrentRetryAndRescheduleThreads only applies to platform
    threads.
  • It keeps the semantics of the default platform executor: waiting and periodic tasks are discarded
    on shutdown(), and tasks that have already started are let finish. shutdownNow() interrupts
    running tasks.
  • Fixed-rate tasks make up missed executions after an overrun, as ScheduledThreadPoolExecutor
    does. Fixed-delay tasks re-arm from completion time.

Other changes

  • Executor helpers live in an internal ExecutorUtilsInternal.
  • ConfigLoader and ConfigurationServiceOverrider expose the new settings
    (josdk.use-virtual-threads, josdk.retry-and-reschedule.concurrent-threads).
  • The webpage sample operator now runs with virtual threads.
  • spotbugs exclusion for the VirtualThreadScheduledExecutor handles. They are ordered by remaining
    delay but must stay distinct by identity, as the JDK's own scheduled futures.

Documentation

  • operations/configuration.md: new "Virtual Threads" section, which points to "Keeping the JVM
    Alive" for operators whose main returns, and the new configuration keys.
  • eventing.md: "Threading of the polling event sources", with the per-source executor and who
    owns it.

Testing

Unit tests, with the virtual thread ones gated by @EnabledForJreRange(min = JAVA_25):

  • VirtualThreadScheduledExecutorTest: virtual thread execution, cancellation, fixed delay and
    fixed rate, overrun catch-up, shutdown semantics, unbounded concurrency.
  • VirtualThreadsTest, BoundedVirtualThreadExecutorAuditTest: bounded virtual thread executor
    and fallback on old JVMs.
  • VirtualThreadsStartupDeadlockTest: startup with virtual threads does not deadlock.
  • ExecutorServiceManagerTest, ConfigurationServiceOverriderTest: lifecycle and overriding.
  • PerResourcePollingEventSourceTest, PollingEventSourceTest, TimerEventSourceTest: each source
    owning its executor (and not sharing it), provided executors left running, virtual threads,
    restart, stop and start ordering, polling resilience.
  • ExecutorUtilsInternalTest: idle pool threads time out and delayed tasks still run.

Integration test: VirtualThreadsIT runs a full operator with the flag on.

Known limitations and follow-ups

  • Only Java 25 and later is officially supported for virtual threads.

Copilot AI balanced review requested due to automatic review settings October 27, 2026 13:21
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 115c9d46-91d5-4639-8102-97120e8051a2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved executor lifecycle, polling startup, and event-validation issues can lose work or prevent events from being recorded.

4 open issues
What changed in this PR

Adds opt-in virtual threads to Java Operator SDK while preserving concurrency limits and Java 17 fallback. The PR also centralizes scheduling and includes event-recording and desired-state fixes.

Changes:

  • Adds virtual-thread configuration, executor implementations, client integration, documentation, and tests.
  • Moves polling and retry scheduling to separate configurable executors with lifecycle management.
  • Updates event aggregation, long event names, and desired-state aspect handling.
File Description
sample-operators/​webpage/​src/​main/​java/​io/​javaoperatorsdk/​operator/​sample/​WebPageOperator.java Enables virtual threads in the sample.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​dependent/​multiplemanagedexternaldependenttype/​MultipleManagedExternalDependentResourceReconciler.java Supplies context for shared polling.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​config/​loader/​ConfigLoaderTest.java Tests new configuration keys.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​baseapi/​virtualthreads/​VirtualThreadsTestReconciler.java Records thread type and concurrency.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​baseapi/​virtualthreads/​VirtualThreadsIT.java Tests reconciliation threading and limits.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​baseapi/​virtualthreads/​VirtualThreadsCustomResource.java Provides the integration-test resource.
operator-framework/​src/​main/​java/​io/​javaoperatorsdk/​operator/​config/​loader/​ConfigLoader.java Binds virtual-thread and scheduling settings.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​timer/​TimerEventSourceTest.java Tests executor ownership and restart.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingEventSourceTest.java Tests shared scheduling and cancellation.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PerResourcePollingEventSourceTest.java Tests polling restart behavior.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​dependent/​DesiredStateAspectAuditTest.java Adds desired-state aspect regression cases.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​event/​EventRecorderAuditTest.java Adds event-recording regression cases.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​event/​DefaultEventRecorderTest.java Tests collision-resistant name shortening.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​VirtualThreadsTest.java Tests executor and client threading.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ExecutorServiceManagerTest.java Tests scheduler separation and lifecycle.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ConfigurationServiceOverriderTest.java Tests threading configuration overrides.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​BoundedVirtualThreadExecutorAuditTest.java Tests limits and pending-task shutdown.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​timer/​TimerEventSource.java Replaces timers with supplied or owned executors.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingEventSource.java Adds shared scheduling and executor ownership.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingConfigurationBuilder.java Adds custom polling executor configuration.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingConfiguration.java Stores the optional polling executor.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PerResourcePollingEventSource.java Resolves executors on startup and cancels tasks.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PerResourcePollingConfiguration.java Defers default executor selection.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​EventSources.java Accepts a retry scheduler supplier.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​EventSourceManager.java Connects retries to the managed scheduler.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​dependent/​external/​PollingDependentResource.java Passes context to polling sources.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​dependent/​BulkDependentResourceReconciler.java Passes the user dependent to aspects.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​reconciler/​DefaultContext.java Handles reentrant desired-state aspect lookups.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​event/​DefaultEventSink.java Updates fields during named-event aggregation.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​event/​DefaultEventRecorder.java Adds event timestamps and hashed name suffixes.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​VirtualThreads.java Implements virtual executors and concurrency bounds.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​Utils.java Adds named daemon scheduler utilities.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ExecutorServiceManager.java Selects executors and manages scheduler lifecycle.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ConfigurationServiceOverrider.java Exposes threading and scheduler overrides.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ConfigurationService.java Defines threading defaults and client integration.
docs/​content/​en/​docs/​documentation/​operations/​configuration.md Documents virtual threads and configuration keys.
docs/​content/​en/​docs/​documentation/​eventing.md Documents shared polling and executor ownership.

Review effort: Balanced

@csviri csviri linked an issue Oct 27, 2026 that may be closed by this pull request
@csviri
csviri requested a balanced review from Copilot October 28, 2026 10:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

csviri and others added 14 commits November 3, 2026 20:19
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Publish the runner of a scheduled task before checking whether it was cancelled, make fixed-rate
tasks catch up after an overrun, drop the stale scheduled future of a primary missing from the
cache, and fix a race in TimerEventSourceTest.
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
ScheduledThreadPoolExecutor starts a core thread on every schedule call until the core size is
reached, regardless of the load, and never gives them up. Allow the core threads to time out so
that the pools shrink back when idle.
…ranch

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d handles from the waiting set

- a per resource poll that outlives a stop and start of the event source no longer reschedules itself next to the polling chain of the new run
- a periodic task cancelled while being re-armed is removed from the waiting set of the scheduled executor

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s an override changes their settings

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ecutors when started again while running

- a per resource registration that races stop() is buffered until the next start instead of leaving a stale entry that blocks polling after a restart
- the timer and per resource polling event sources resolve their executor again when started while running, which happens when the operator is started again after a failed start and replaces its executors

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s, and let in-flight work finish when stopping owned executors

- PollingEventSource schedules its polling task again when started while running after the operator replaced its executors
- TimerEventSource schedules the timers pending on the replaced executor again for their remaining delay
- stopping a polling or timer event source that owns its executor no longer interrupts a poll or handler in flight

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
csviri force-pushed the virtual-threads-and-shared-executors branch from aaa7b6c to 83cc809 Compare November 3, 2026 19:19
@csviri
csviri requested a balanced review from Copilot November 3, 2026 19:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

csviri and others added 2 commits November 3, 2026 14:39
…se races in the polling event sources

- ExecutorServiceManager keeps the public scheduledExecutorService() of previous versions, deprecated, created on first use and replaced after a restart
- PerResourcePollingEventSource drops the result of a fetch from a previous run instead of overwriting the cache of the current one
- PollingEventSource ends start() cleanly when stopped during the initial fetch instead of failing on the shut down executor it owns

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ck in the fetching task

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

shutdownNow() omits unstarted tasks from its return value, violating its contract, while a startup replay/deletion race can leave stale entries that prevent polling from restarting for recreated resources.

2 open issues
3 resolved since last review

Review effort: Balanced

…g the JVM alive

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Virtual-thread overrides can discard custom executors, while a startup deletion race can leave stale polling entries that prevent polling after resource recreation.

1 open issue
1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Executor replacement can silently discard custom retry scheduling and re-arm an already firing timer, causing duplicate events while the original handler is still running.

3 open issues

Review effort: Balanced

Comment on lines +411 to +415
} else if (!changes(
concurrentRetryAndRescheduleThreads, original.concurrentRetryAndRescheduleThreads())) {
return original.getRetryAndRescheduleExecutorService();
} else {
return super.getRetryAndRescheduleExecutorService();
Comment on lines +174 to +175
if (future.isCancelled() || !future.isDone()) {
scheduleOnce(resourceID, Math.max(0, future.getDelay(TimeUnit.MILLISECONDS)));

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for Virtual Threads

2 participants