Skip to content

ui: add pipeline duplicate controls#6520

Merged
Karakatiza666 merged 2 commits into
feldera:mainfrom
Dnreikronos:web_console/duplicate_pipeline
Jun 25, 2026
Merged

ui: add pipeline duplicate controls#6520
Karakatiza666 merged 2 commits into
feldera:mainfrom
Dnreikronos:web_console/duplicate_pipeline

Conversation

@Dnreikronos

@Dnreikronos Dnreikronos commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

fixes #6110

adds duplicate from the pipelines table and from the editor menu. imo keeping both paths on the same helper is the right call here, since the name lookup and copied fields should not drift.

the copy keeps sql, udfs, runtime config, program config, description, and tags. storage/state start fresh. idk if there is a cleaner split than shared helper plus two buttons, but this felt pretty tidy to me. lgtm locally with a mock api, plus bun --cwd=js-packages/web-console run check and bun --cwd=js-packages/web-console run vitest --run --project server src/lib/compositions/duplicatePipeline.spec.ts.

video

feldera.mp4

@Dnreikronos

Copy link
Copy Markdown
Contributor Author

hi, @Karakatiza666!
i can't assign you as reviewer because i don't have permissions. because of this i'm pinging you here directly

@mihaibudiu
mihaibudiu requested a review from Karakatiza666 June 22, 2026 17:11
@Karakatiza666

Copy link
Copy Markdown
Contributor

I am yet to review the code, but the showcase looks good! Please check the algorithm to derive the duplicate name - orders-copy2 was created before orders-copy - this looks like a deterministic bug in the name resolution.

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice scoping: one shared duplicatePipeline helper called from both the table row action and the editor menu. The optimisticDuplicatePipelineThumb shape lines up with the post-create thumb, the Queued: { cause: 'compile' } initial status is right (it'll get superseded by the server response), and storageStatus: 'Cleared' / connectors: undefined correctly forces the copy to start fresh.

A few thoughts:

  • resolveDuplicatePipelineName and Karakatiza's question. The algorithm is name-copy, name-copy2, name-copy3, ... walking from index=1. That's fine until the user deletes a name in the middle of the sequence — then the next duplicate fills the hole rather than appending. So if orders-copy2 exists but orders-copy doesn't (e.g. the user deleted the first copy), the next duplicate becomes orders-copy, not orders-copy3. Whether that's a bug depends on what you want: hole-filling is the principle-of-least-surprise answer ("smallest available name") and matches what most file managers do, but it does mean the visible suffix is no longer monotonic in creation order. I'd lean "this is fine, document the rule in duplicatePipelineTooltip or as a code comment" rather than switching to a max-suffix-plus-one strategy, but pick whichever is least confusing to your users.
  • for (let index = 1; ; index += 1). Unbounded loop (Power-of-Ten #2). A user with 100K pipelines named orders-copy* will hang the browser. A MAX_DUPLICATE_ATTEMPTS = 10_000 ceiling with a thrown error (could not allocate a unique duplicate name) is the right shape — covers any plausible real workload and turns the pathological case into a visible failure instead of a hang.
  • Test coverage. duplicatePipeline.spec.ts looks like it covers the payload (good), but I'd add one for resolveDuplicatePipelineName against a list with a hole (e.g. [orders-copy2, orders-copy3] → expect orders-copy) and one against [orders-copy, orders-copy2, orders-copy3] → expect orders-copy4, so the chosen semantics are pinned. Right now only the payload composition is tested, not the name-resolution rule that Karakatiza is asking about.
  • duplicatePipelineTooltip capitalization. Tooltips should start with a capital letter and end with a period — "Pipeline storage and state are not duplicated. The new pipeline starts fresh." is just a touch more polished than the lower-cased version.
  • Cargo.toml-style font asset changes (feldera-material-icons.{css,svg,ttf,woff,woff2} + generic-icons.*). ~700 lines of regenerated font output that shouldn't be part of this PR's mental model. Worth a one-line note in the PR body (or splitting into a separate commit) that the regeneration is mechanical and was needed to add the copy-plus glyph. Not a blocker.
  • Drop unrelated metrics formatting commit. Good cleanup — but the dirty-history comment in MEMORY.md applies: please squash before merging or rebase out so the final history is one logical commit per concern.

I'll wait for Karakatiza's review pass before approving formally — the naming-rule choice is the one open call that affects users.


{#if actions.length}
<div class="flex h-9 flex-wrap gap-2">
{#if selected.length === 1}

@Karakatiza666 Karakatiza666 Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be simpler, but let's follow the structure and add the "duplicate" button to the list of actions defined in const actions = $derived.by(.... This helps separating the logic and the presentation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, duplicate is in the same derived action list now. imo that reads better too because the render block no longer has a special case.

@@ -638,17 +676,44 @@ groups related actions into multi-action dropdowns when multiple options are ava
{/snippet} -->

{#snippet _delete()}

@Karakatiza666 Karakatiza666 Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename the snippet appropriately and hard-code to insert it at the end of the actions in the definition of const getRawActions or const active - this design solidifies it as an action independent of status, so it's redundant to keep it in the match expression with status-dependent actions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, moved it out of the status match, appended it in active, and renamed the old popup action to more. lgtm to me because duplicate is always about the current pipeline, not the runtime status.

onDeletePipeline?.(pipelineName)
goto(resolve('/'))
}
const duplicateCurrentPipeline = async (close: () => void) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reconcile and combine this duplicateCurrentPipeline, duplicateSelectedPipeline and duplicatePipeline in duplicatePipeline.ts - the procedure is the same, including discarding the refresh and updating the cache. The only difference is close() + await goto(...) vs selectedPipelines = [] - both can be implemented as a callback passed at the call site and called after await duplicatePipeline. Early exit if (unsavedChanges) can be done outside the combined function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, the shared helper handles name resolution, refresh discard, optimistic insert, server create, cache replace, and rollback. the call sites only do their local after-step now: reset selection or navigate.

import type { PipelineThumb } from '$lib/services/pipelineManager'

export const duplicatePipelineTooltip =
'pipeline storage and state are not duplicated. the new pipeline starts fresh.'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Start sentences with the capital letter

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, the tooltip copy is capitalized in the ui now.

const pipelineNames = new Set(pipelines.map((pipeline) => pipeline.name))

for (let index = 1; ; index += 1) {
const candidate = `${pipelineName}-copy${index === 1 ? '' : index}`

@Karakatiza666 Karakatiza666 Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct and the issue I described earlier is not a blocker, but I hope you can identify why *-copy2 appeared before *-copy in the screen capture. Or let me know if that was recorded on some earlier version of the code,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

found it. the video was from the older local behavior while i was duplicating a copy of orders, so the selected name became the next base and the sequence went weird. the resolver now strips a trailing -copy or -copy before picking the next name. imo that is the behavior users expect: orders, orders-copy, orders-copy2, orders-copy3. added tests for the hole case and for duplicating existing copies too.

@Dnreikronos

Copy link
Copy Markdown
Contributor Author

pushed f99506e. covered the non-inline review notes too: bounded the duplicate-name loop at 10k, added the hole/consecutive-copy tests, kept the smallest-available-name behavior, and added the font note to the pr body. imo smallest available name is still the better rule, as long as copies of copies normalize back to the original base.

@Karakatiza666 Karakatiza666 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Last fix:

Let's not introduce type PipelineListUpdates. Either do

updates: PipelineListUpdates ReturnType<typeof useUpdatePipelineList>

, or turn duplicatePipeline function into useDuplicatePipeline "composition" that calls useUpdatePipelineList itself (and relocate it in the source tree appropriately).
Either way it's fine to couple to useUpdatePipelineList directly; coupling to it through duck typing of PipelineListUpdates is unnecessary indirection.

After that - make sure to cd js-packages/web-console && bun run format, discard formatting changes to the files you did not touch, squash commits and enqueue to merge. Thank you for the contribution!

@Dnreikronos
Dnreikronos force-pushed the web_console/duplicate_pipeline branch from f99506e to bb830e5 Compare June 24, 2026 15:32
@Dnreikronos

Copy link
Copy Markdown
Contributor Author

done, pushed a squashed commit with the type alias removed. imo the direct returntype tie-in reads cleaner here too.

ran check + the focused duplicate pipeline test. i also ran bun run format, but idk, it still dies on existing prettier/biome stuff after touching a ton of unrelated files, so i kept the relevant files clean and dropped the unrelated formatting noise.

also tried gh pr merge 6520 --auto to enqueue, but github says i do not have permission to enable auto-merge on this repo. lgtm from my side, but ltm like someone with maintainer perms needs to hit the queue button.

@Karakatiza666
Karakatiza666 force-pushed the web_console/duplicate_pipeline branch from bb830e5 to 3611605 Compare June 24, 2026 16:26
@Karakatiza666
Karakatiza666 enabled auto-merge June 24, 2026 16:26
@Karakatiza666
Karakatiza666 added this pull request to the merge queue Jun 24, 2026
@Dnreikronos

Copy link
Copy Markdown
Contributor Author

ty for pairing with me here
@Karakatiza666

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three substantive things from my prior pass are addressed:

  • Unbounded loop in resolveDuplicatePipelineName is now bounded by MAX_DUPLICATE_ATTEMPTS = 10_000 with a thrown error on exhaustion — Power-of-Ten #2 satisfied, and the cap's pathological case is covered by 'fails when no duplicate name can be allocated'.
  • Naming-rule test coverage is now solid: hole-fill ([orders-copy2, orders-copy3] → orders-copy), consecutive ([copy, copy2, copy3] → copy4), mid-gap ([copy, copy3] → copy2), and re-duplicating both orders-copy and orders-copy2 to confirm the base-name strip is stable. The hole-filling semantics from Karakatiza's question is pinned in tests now, so any future change is a visible one.
  • History squashed to a single commit; the unrelated metrics-formatting drop is gone.

Couple of small remaining things, none blocking:

  • for (let index = 1; index <= MAX_DUPLICATE_ATTEMPTS; index += 1) — the constant lives in the same file and the test imports it, which is fine, but the error message ("Could not allocate a unique duplicate pipeline name.") could mention the cap (...after 10000 attempts) so the failure tells the user what number they hit. Tiny actionability tweak.
  • duplicatePipelineTooltip now reads "Pipeline storage and state are not duplicated. The new pipeline starts fresh." — good, capitalized and punctuated.
  • The font-asset regeneration is still in this PR; the new note in the PR body explaining why is what I asked for. Fine to keep.

APPROVE.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 24, 2026
@Karakatiza666

Copy link
Copy Markdown
Contributor

@Dnreikronos please address the unit test failures. You can run those with bun run test-unit (these failures are in --project client)

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review on ce62e31f (since prior APPROVE on 3611605e).

Sole new commit "Stabilize web console browser specs" is test-infra only — three *.svelte.spec.ts files. Replaces await locator.click() with synchronous clickButton(locator) that drills to the underlying HTMLButtonElement and clicks directly, and switches the pipeline-list search to dispatch a real input event followed by await tick() rather than .fill(). This bypasses the vitest-browser-svelte click/fill helpers that were apparently racing the Svelte reactivity in CI.

Reasonable workaround; the production code is untouched. APPROVE stands.

@Dnreikronos

Copy link
Copy Markdown
Contributor Author

hi, @Karakatiza666!
i just adjusted the tests. can you re-trigger the CI, please?

@Karakatiza666
Karakatiza666 force-pushed the web_console/duplicate_pipeline branch from ce62e31 to 033d1c6 Compare June 25, 2026 12:45
@Karakatiza666
Karakatiza666 enabled auto-merge June 25, 2026 12:47
@Karakatiza666
Karakatiza666 added this pull request to the merge queue Jun 25, 2026
Merged via the queue into feldera:main with commit 5c8ed75 Jun 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Button to duplicate a pipeline

3 participants