Skip to content

[migrations] Spark-to-Feldera migration tool PoC. #5837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wilmaontherun wants to merge 45 commits into
base: main
Choose a base branch
from
Draft

[migrations] Spark-to-Feldera migration tool PoC. #5837

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
046f795
[migrations] Spark-to-Feldera migration tool PoC.
wilmaontherun Mar 16, 2026
bb1b24c
[ci] apply automatic fixes
feldera-bot Mar 16, 2026
871d79c
Intermediate progress based on Mihai's comments.
wilmaontherun Mar 19, 2026
e856080
fixed comments on skills
wilmaontherun Mar 19, 2026
b672679
Fixed all comments before we refactor skills
wilmaontherun Mar 19, 2026
3d8536f
Merge skills
wilmaontherun Mar 19, 2026
219f8b7
Fixed the rest of the code w.r.t. new skill file
wilmaontherun Mar 19, 2026
83d7731
Revised doc indexing
wilmaontherun Mar 19, 2026
98f7dee
merged skills
wilmaontherun Mar 19, 2026
56bd911
add --verbose flag, translate-file, combined demos, and Feldera PK/qu…
wilmaontherun Mar 19, 2026
b10f7a4
more demo files
wilmaontherun Mar 19, 2026
7d56201
revised samples and skills
wilmaontherun Mar 20, 2026
9e17a33
[ci] apply automatic fixes
feldera-bot Mar 20, 2026
3a75739
add --compiler option, fix no-compiler handling, improve example list…
wilmaontherun Mar 20, 2026
ab4f746
[ci] apply automatic fixes
feldera-bot Mar 20, 2026
6672146
fixed readme
wilmaontherun Mar 20, 2026
9975f33
[ci] apply automatic fixes
feldera-bot Mar 20, 2026
14e7cc6
Add --model option, remove OpenAI support and hardcoded compiler path
wilmaontherun Mar 20, 2026
7ac2fb9
Use sqlparse for SQL splitting, fix README inconsistencies
wilmaontherun Mar 20, 2026
6655f7c
Add prompt caching and rate limit retry; skip examples on first pass
wilmaontherun Mar 20, 2026
8fa1210
Clean up code quality: fix imports, types, and consistency issues
wilmaontherun Mar 21, 2026
4358104
Fix spark_skills.md inconsistencies
wilmaontherun Mar 24, 2026
80a5e7f
[ci] apply automatic fixes
feldera-bot Mar 24, 2026
7da4c30
Verify and fix spark_skills.md against Apache Spark SQL reference
wilmaontherun Mar 24, 2026
589406c
Overhaul spark/data/samples: fix errors, add new patterns, remove tri…
wilmaontherun Mar 24, 2026
b53a97a
Fix skills inconsistencies: QUARTER unsupported, contains/binary, pmo…
wilmaontherun Mar 25, 2026
05388f9
Rename misnamed sample files to match their content
wilmaontherun Mar 25, 2026
643ba2b
Improve and expand sample demos
wilmaontherun Mar 25, 2026
d1b0c95
Fix demo files: remove unsupported patterns, add dates and arithmetic…
wilmaontherun Mar 25, 2026
d7057b9
Fix aggregations and arithmetic demos to use only supported Feldera f…
wilmaontherun Mar 25, 2026
b438037
split_part skill & base_url config
anandbraman Mar 27, 2026
aaf02dd
Update AVG(integer) rule: rewrite to AVG(CAST(col AS DOUBLE)) for int…
wilmaontherun Mar 30, 2026
9de37f9
Untrack known_unsupported.yaml (ignored by .gitignore)
wilmaontherun Mar 30, 2026
e088426
Add dialect subcommand structure: felderize spark <cmd>
wilmaontherun Mar 30, 2026
d596584
Move data/skills one level above spark/ to felderize root
wilmaontherun Mar 30, 2026
d5bf6e0
Move skills file to spark/skills/
wilmaontherun Mar 30, 2026
e714f19
Address review comments: lateral aliases, size/CARDINALITY, JSON, sem…
wilmaontherun Mar 30, 2026
621dd95
Merge branch 'gh-readonly-queue/main/pr-5953-63f28bb9543f28137c91d7a2…
wilmaontherun Mar 30, 2026
a84c82f
Reorganize spark_skills.md: fix section placement and add missing ent…
wilmaontherun Mar 31, 2026
06b7bc6
Fix and improve spark_skills.md translation rules
wilmaontherun Apr 2, 2026
425d5ab
Reorganize spark_skills.md: structured subsections with emoji markers
wilmaontherun Apr 2, 2026
2b98d4e
Verify spark_skills.md claims against test evidence; fix errors found
wilmaontherun Apr 2, 2026
4faca22
Fix skills: STDDEV rewrite rule, GBD-REGEX-ESCAPE for RLIKE, CAST(num…
wilmaontherun Apr 2, 2026
c1f740d
README: note that Spark is current dialect, more planned
wilmaontherun Apr 2, 2026
d8a1ed6
[ci] apply automatic fixes
feldera-bot Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge skills
  • Loading branch information
@wilmaontherun
wilmaontherun committed Mar 19, 2026
commit 3d8536f1bcf572813f9e18cf2432f49d5d9b9677
75 changes: 0 additions & 75 deletions python/felderize/spark/data/skills/array-translation/SKILL.md
View file
Open in desktop

This file was deleted.

265 changes: 0 additions & 265 deletions python/felderize/spark/data/skills/function-reference/SKILL.md
View file
Open in desktop

This file was deleted.

Loading