Skip to content

feat: implement graceful degradation and timeout propagation in sync …#22977

Open
Aayushiii25 wants to merge 1 commit into
cloudquery:mainfrom
Aayushiii25:feat/graceful-sync-shutdown
Open

feat: implement graceful degradation and timeout propagation in sync …#22977
Aayushiii25 wants to merge 1 commit into
cloudquery:mainfrom
Aayushiii25:feat/graceful-sync-shutdown

Conversation

@Aayushiii25

Copy link
Copy Markdown
  1. Fixed Memory Leaks in TransformerPipeline:
    Replaced time.After(1 * time.Second) with a reusable time.NewTicker inside the Send() and startBlocking() loops. This prevents the allocation of thousands of lingering time.Timer objects during high-throughput syncs.

  2. Fixed Goroutine Leaks in TransformerPipeline:
    Introduced context.Context cancellation for the background Recv() goroutines. When the pipeline closes, the background goroutine now cleanly exits instead of being orphaned and blocking on a dead channel indefinitely.

  3. Best-Effort Parallel Destination Shutdown:
    Previously, destination cleanup in sync_v2.go and sync_v3.go was sequential and fail-fast. If the first destination failed to close, subsequent destinations were never closed (leaving dangling streams and missing summary data). This PR implements a parallel shutdown sequence using sync.WaitGroup, collecting all errors securely via sync.Mutex and errors.Join, ensuring all destinations attempt to close cleanly.

  4. Plugin Timeout Propagation:
    Previously, the CLI lacked timeout propagation for plugin lifecycle methods, meaning a stuck plugin would hang the entire CLI indefinitely. This PR introduces a --plugin-timeout CLI flag (defaulting to 0 for backward compatibility). It propagates a derived context to Init, Configure, Close, CloseAndRecv, and testPluginConnection gRPC calls to ensure the CLI can fail gracefully.

@Aayushiii25 Aayushiii25 requested a review from a team as a code owner June 19, 2026 07:19
@Aayushiii25 Aayushiii25 requested a review from erezrokah June 19, 2026 07:19

@erezrokah erezrokah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Aayushiii25, thank you for your contribution. Since this PR seems to try and fix 4 different distinct issues, do you mind opening a bug or feature report per fix first?

Please detail the impact of the bug/feature and share a reproduction so we can validate it on our end.

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants