Skip to content

Uniformize monorepo structure to OrdinaryDiffEq canonical style#611

Merged
ChrisRackauckas merged 10 commits into
SciML:masterfrom
ChrisRackauckas-Claude:uniformize-monorepo-structure
Jun 8, 2026
Merged

Uniformize monorepo structure to OrdinaryDiffEq canonical style#611
ChrisRackauckas merged 10 commits into
SciML:masterfrom
ChrisRackauckas-Claude:uniformize-monorepo-structure

Conversation

@ChrisRackauckas-Claude
Copy link
Copy Markdown
Contributor

Uniformizes the RecursiveArrayTools.jl monorepo to the OrdinaryDiffEq.jl canonical structure.

Canonicalization done

  • [sources] path graph (leaf->root). Each sublibrary (RecursiveArrayToolsArrayPartitionAnyAll, RecursiveArrayToolsRaggedArrays, RecursiveArrayToolsShorthandConstructors) now declares RecursiveArrayTools = {path = "../.."} in [sources], so CI tests them against the PR-branch root rather than the registered release. This preserves the true dependency direction (the sublibs depend on the registered root). The root does not add cyclic root->leaf [sources] because it does not depend on its sublibs — Julia 1.11 rejects [sources] entries that are not in [deps]/[extras] (verified locally).
  • Removed the redundant double-dispatch. The root Tests.yml no longer GROUP-dispatches RaggedArrays/ArrayPartitionAnyAll/ShorthandConstructors, and the root test/runtests.jl no longer Pkg.activate/Pkg.tests them. Sublibraries are now covered exclusively by SublibraryCI's per-project model.
  • Root test/runtests.jl is now a _detect_sublibrary_group dispatcher mirroring OrdinaryDiffEq: a GROUP naming lib/<X> activates that sublib, develops its [sources] path deps on Julia < 1.11, sets RECURSIVEARRAYTOOLS_TEST_GROUP, and Pkg.tests it; otherwise the root runs its own @safetestset groups (Core / SymbolicIndexingInterface / Downstream / GPU / nopre).
  • SublibraryCI.yml now passes group-env-name: RECURSIVEARRAYTOOLS_TEST_GROUP and check-bounds: auto to sublibrary-project-tests.yml@v1.
  • DowngradeSublibraries.yml now passes group-env-name/group-env-value: Core, expands the skip list to the siblings (root + 3 sublibs) plus stdlibs, and continues to auto-discover lib/*. sublibrary-downgrade.yml@v1 has no allow-reresolve input, so none is passed.
  • Each sublibrary test/runtests.jl now reads get(ENV, "RECURSIVEARRAYTOOLS_TEST_GROUP", "Core") and gates its tests on the Core group — the env name is threaded consistently from every caller.
  • test/test_groups.toml added to each sublibrary declaring only [Core] on ["lts", "1.11", "1", "pre"]. None of the sublibs have QA-group tests, so the default group expansion (which would also emit QA on [1]) is overridden.

No [compat] floors were lowered; no tests/tolerances were loosened, skipped, or broken. There are no test/Project.toml files (the root's test/{downstream,gpu,nopre}/Project.toml are nested Pkg-environment subprojects, matching OrdinaryDiffEq's test/qa/test/gpu env subprojects — not test-target Project.tomls). No @testitem/ReTestItems were present in this repo, so item D is a no-op. GPU.yml is left as a root self-hosted job (the GPU tests are a root-level concern using the ArrayPartitionAnyAll env, not a sublibrary group).

Verified locally (isolated depot, Julia 1.11.9)

  • TOML-parsed every changed Project.toml; every [targets].test dep resolves from [extras] or [weakdeps] (ReverseDiff/Tables come from [weakdeps], as on master). YAML-parsed all changed workflows.
  • Instantiated the root and 2 representative sublibs; confirmed each sublib resolves RecursiveArrayTools from the local ../.. path (is_tracking_path=true) via [sources].
  • Ran RecursiveArrayToolsArrayPartitionAnyAll Core group via the new @testset/group dispatch: Optimized any 5/5, Optimized all 4/4, Matches AbstractArray default results 4/4 — all pass.
  • Ran RecursiveArrayToolsShorthandConstructors Core group: VA[...] shorthand 6/6, AP[...] shorthand 6/6 — all pass.
  • Ran the full root dispatcher end-to-end via Pkg.test() with GROUP=RecursiveArrayToolsArrayPartitionAnyAll: _detect_sublibrary_group activated the sublib, set the group env, and Pkg.tested it — "RecursiveArrayTools tests passed" (exit 0). This exercises the dispatcher, [sources] resolution, and env threading together.
  • Verified _detect_sublibrary_group mapping (X -> (X, Core), X_QA -> (X, QA), root group names fall through) and the group gate (default -> Core runs, QA -> skips).
  • Ran the root Core @safetestset files (named_array_partition_tests.jl passed; utils_test.jl 7/8 — the single error was a missing dep in the hand-built minimal env, present in the real [targets].test).
  • Ran Runic on all changed .jl; --check is clean.

What CI must confirm

  • The full root Core/Downstream/nopre groups on the [1, lts, pre] matrix (heavy deps: Mooncake, Aqua, ModelingToolkit, OrdinaryDiffEq, JET — not run locally).
  • SublibraryCI discovery + the RaggedArrays Core group (large suite, not fully run locally).
  • DowngradeSublibraries resolution on lts with the new skip list and group env.
  • The root and sublib paths on Julia < 1.11 (the manual [sources] develop branch — only the >= 1.11 native path was exercised locally).

Ignore until reviewed by @ChrisRackauckas.

ChrisRackauckas and others added 10 commits June 8, 2026 05:40
Align RecursiveArrayTools.jl with the OrdinaryDiffEq.jl canonical monorepo
layout:

- Add [sources] to each sublibrary pointing at the umbrella root via ../..
  (true leaf->root dependency direction; the 3 sublibs depend on the
  registered root today). No cyclic root->leaf [sources] are added because
  the root does not depend on its sublibs.
- Remove the redundant double-dispatch of the sublibraries: the root
  Tests.yml no longer GROUP-dispatches RaggedArrays/ArrayPartitionAnyAll/
  ShorthandConstructors, and the root test/runtests.jl no longer activates
  and Pkg.tests them. Sublibraries are now covered exclusively by
  SublibraryCI's per-project model.
- Rewrite root test/runtests.jl as a _detect_sublibrary_group dispatcher
  mirroring OrdinaryDiffEq: a GROUP naming lib/<X> activates that sublib,
  develops its [sources] path deps on Julia < 1.11, sets the sub group env,
  and Pkg.tests it; otherwise the root runs its own @safetestset groups.
- SublibraryCI.yml: thread group-env-name RECURSIVEARRAYTOOLS_TEST_GROUP and
  check-bounds auto.
- DowngradeSublibraries.yml: add group-env-name/group-env-value Core, expand
  the skip list to the siblings (root + 3 sublibs) plus stdlibs; auto-discovers
  lib/* (sublibrary-downgrade.yml@v1 has no allow-reresolve input).
- Each sublibrary test/runtests.jl now reads
  get(ENV, "RECURSIVEARRAYTOOLS_TEST_GROUP", "Core") and gates its tests on
  the Core group.
- Add test/test_groups.toml to each sublibrary declaring only [Core] on
  [lts, 1.11, 1, pre], since none of them have QA-group tests (the default
  expansion would otherwise emit an unsupported QA group on [1]).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the SciML/OrdinaryDiffEq.jl convention: the catch-all test group
is canonical Title-case All, not all-caps ALL. The sublib runtests.jl
gates compared TEST_GROUP against "ALL" while the default and the
test_groups.toml key are already canonical ("Core"). Align the gate
string casing so all group-name strings are consistent.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- TagBot.yml: adopt canonical OrdinaryDiffEq structure — top-level
  permissions block, workflow_dispatch.inputs.lookback default "3",
  named root TagBot job, and a TagBot-Subpackages matrix
  (fail-fast: false) over the three registered sublibraries, each
  invoking TagBot@v1 with subdir: lib/<package>.
- .gitignore: complete to canonical set (docs/build, LocalPreferences.toml,
  *.jl.*.mem, *.DS_Store, profile.pb.gz, .*.swp, .claude/, etc.).
- Casing: rename the special test group "nopre" -> "NoPre" on both
  sides of the workflow<->runtests contract (Tests.yml matrix.group +
  exclude, and runtests.jl GROUP comparison) so it is canonical
  Title-case and case-consistent.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ors test dep

- Sublibrary READMEs: prepend the canonical OrdinaryDiffEq-style badge
  block (Zulip, Global Docs -> RecursiveArrayTools, ColPrac, SciML Code
  Style) and the "<Name>.jl is a component of the RecursiveArrayTools.jl
  monorepo. <role>. While completely independent and usable on its own,
  users wanting the full functionality should use RecursiveArrayTools.jl."
  wording, substituting names. Existing content preserved below.
- Project.toml: declare RecursiveArrayToolsShorthandConstructors as a
  proper test dependency — added to [extras], [targets].test, [compat]
  ("1"), and a [sources] path entry so the Core test group resolves it
  from the in-repo lib/ on Julia >= 1.11 (runtests.jl Pkg.develop
  fallback still covers Julia < 1.11 where [sources] is ignored).

Verified: project resolves and `VA[...]`/`AP[...]` from the sublibrary
load in a Core-test-like environment.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the inline Aqua quality-assurance test out of the main test
environment and into a dedicated test/qa/ project, matching the
canonical OrdinaryDiffEq sublib layout (lib/*/test/qa/).

- Add test/qa/Project.toml: Aqua + RecursiveArrayTools (via [sources])
  + Pkg + Test, with [compat] for each dep and julia.
- Move test/qa.jl -> test/qa/qa.jl (Aqua testset + the existing
  downstream-package manifest sentinel; behavior unchanged).
- runtests.jl: add activate_qa_env() and, in the Core/All branch,
  activate the qa env, run the QA safetestset, then restore the main
  test project so the remaining Core functional tests keep their full
  dependency set.
- Remove Aqua from the main Project.toml [extras], [targets].test, and
  [compat]; it now lives only in the qa env.

JET (NoPre group) is already isolated in test/nopre/ and is left as-is.
The three lib/* sublibraries run no QA today and are left untouched.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… downstream-only NLsolve

The branch had added RecursiveArrayToolsShorthandConstructors to the umbrella
root [sources]. It is a test-only dependency (the VA[]/AP[] shorthand tests),
not a dependency of the RecursiveArrayTools package itself, so sourcing it at
the package level makes Pkg.test add it to the Core test env as a declared
package dependency. Aqua.test_stale_deps then flags it as a stale dep of
RecursiveArrayTools (the module never imports it), failing the QA group.

master never had this [sources] entry: its runtests.jl Pkg.develops the
sublibrary into the test env directly, which the current runtests.jl still
does. Removing the [sources] entry restores master's green QA behavior
(verified: Core+QA Pkg.test now passes — Quality Assurance 9 pass / 1 broken
(expected ambiguities @test_broken) / 0 fail; previously 1 fail).

Also drop NLsolve from the main [extras]/[targets].test/[compat]: it is used
only by test/downstream/odesolve.jl and is already declared in
test/downstream/Project.toml, so it does not belong in the base test env.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ject.tomls

Replace the root Tests.yml + bespoke GPU.yml with a single canonical CI.yml
(matching the OrdinaryDiffEq umbrella layout):

- CI.yml carries the version matrix for the root base groups
  (Core/SymbolicIndexingInterface/Downstream/NoPre on [lts, 1, pre], NoPre
  excluded on pre) via SciML/.github tests.yml@v1.
- The GPU tests become a dep-adding group on the self-hosted GPU runner: a
  dedicated CI.yml job passes group=GPU and
  runner=[self-hosted, Linux, X64, gpu] to tests.yml@v1 (which sets
  GROUP=GPU; runtests.jl then activates test/gpu and runs the GPU
  safetestsets). This reproduces the former GPU.yml (same runner labels,
  60-min timeout, src,ext coverage) so GPU.yml is deleted.

- Give the dep-adding root group Project.tomls the canonical
  [sources] = {path=../..} (+ sibling sublib paths) plus the package and Test
  deps so they resolve on Julia >= 1.11 against the PR-branch code:
  - test/gpu: add Adapt/ArrayInterface/RecursiveArrayTools/Test/Zygote +
    [sources] for the root and RecursiveArrayToolsArrayPartitionAnyAll.
  - test/nopre: add RecursiveArrayTools + [sources].
  - test/downstream: add Test + [sources] for root and
    RecursiveArrayToolsShorthandConstructors.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…olders)

Make the root test layout match the OrdinaryDiffEq canonical pattern: every
test lives in exactly one group, each group in its own capitalized test/<Group>/
folder, dep-adding groups have their own Project.toml (excluded from the light
main env), and the version matrix is consistent between CI and test_groups.toml.

- Split QA (Aqua) out of the Core branch into a standalone QA group keyed on
  GROUP == "QA" || GROUP == "All", running in its dep-adding test/QA env. Add a
  dedicated QA matrix row to CI.yml restricted to [lts, 1] (excluded on pre),
  so QA no longer runs inside the functional Core job.
- Normalize sublib test_groups.toml [Core] versions ["lts","1.11","1","pre"] ->
  ["lts","1","pre"] for all three sublibraries.
- Move the loose root Core test files into test/Core/, including the previously
  orphaned partitions_and_static_arrays.jl which was never wired into runtests
  and is now part of the Core group. Capitalize the remaining group folders
  (downstream->Downstream, gpu->GPU, nopre->NoPre, qa->QA) and route the
  SymbolicIndexingInterface and NoPre tests into their own group folders so all
  group folder names match their CI GROUP names.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QA is its own dep-adding test group (test/QA/Project.toml) and is run as
a separate CI matrix entry, so it should not also fire under the "All"
group. CI never runs "All", and a local "All" run should not pull the QA
tooling into the functional test pass.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace TagBot.yml with the canonical SciML/.github tagbot.yml@v1 thin
caller, plus a tagbot-subpackages matrix over the registered monorepo
sublibraries (RecursiveArrayToolsArrayPartitionAnyAll,
RecursiveArrayToolsRaggedArrays, RecursiveArrayToolsShorthandConstructors).

Drop hand-maintained skip lists and pinned julia-version floors from the
downgrade callers; the centralized workflow now auto-populates skip
(stdlibs union in-repo sublibs union caller package) and defaults to the
LTS floor.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude force-pushed the uniformize-monorepo-structure branch from 30d494c to 86dbcc1 Compare June 8, 2026 09:41
@ChrisRackauckas-Claude ChrisRackauckas-Claude marked this pull request as ready for review June 8, 2026 09:41
@ChrisRackauckas ChrisRackauckas merged commit f5a9eb1 into SciML:master Jun 8, 2026
16 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants