Skip to content

Mesh and Heightfield collision support for Kamino#2280

Merged
nvtw merged 16 commits intonewton-physics:mainfrom
nvtw:dev/tw2/kamino_mesh_heightfield
Apr 14, 2026
Merged

Mesh and Heightfield collision support for Kamino#2280
nvtw merged 16 commits intonewton-physics:mainfrom
nvtw:dev/tw2/kamino_mesh_heightfield

Conversation

@nvtw
Copy link
Copy Markdown
Member

@nvtw nvtw commented Mar 31, 2026

Description

Mesh and Heightfield collision support for Kamino

Checklist

  • New or existing tests cover these changes
  • The documentation is up to date with these changes
  • CHANGELOG.md has been updated (if user-facing change)

Test plan

TODO - testing is currently not sufficient

Summary by CodeRabbit

  • New Features

    • Full mesh/heightfield support with per-shape heightfield/voxel/collision parameters and improved collision-radius handling.
    • Optional Newton-driven collision mode and RL example enhancements: procedural terrain, configurable pushable balls, and actuated-joint control flow.
  • Refactor

    • Geometry type API extended with read-only type attributes; heightfield shape implemented; narrow-phase inputs renamed and conditionally wired for explicit geometries.
  • Tests

    • New end-to-end test suite validating mesh and heightfield collision behavior and conversion.

@nvtw nvtw self-assigned this Mar 31, 2026
@nvtw nvtw marked this pull request as draft March 31, 2026 15:02
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 31, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds GeoType properties for primitive/explicit classification; threads per-shape mesh/heightfield and collision buffers through Kamino (model, builder, pipeline); implements HFieldShape; switches checks to use GeoType.is_explicit; adds examples and extensive mesh/heightfield tests.

Changes

Cohort / File(s) Summary
Type System
newton/_src/geometry/types.py
Added GeoType.is_primitive and GeoType.is_explicit boolean properties.
Kamino datamodel & model conversion
newton/_src/solvers/kamino/_src/core/geometry.py, newton/_src/solvers/kamino/_src/core/model.py
Extended GeometriesModel with optional per-shape fields (heightfield_index, heightfield_data, heightfield_elevations, collision_aabb_lower, collision_aabb_upper, voxel_resolution, collision_radius); model construction now sets type directly from arrays and populates the new fields.
Shapes & builder
newton/_src/solvers/kamino/_src/core/shapes.py, newton/_src/solvers/kamino/_src/core/builder.py
Deprecated free-function checks now delegate to GeoType properties; HFieldShape implemented to hold a Heightfield and expose data; builder uses geom.shape.type.is_explicit to decide when to finalize explicit geometry data.
Collision pipeline (unified)
newton/_src/solvers/kamino/_src/geometry/unified.py
Detects explicit geometries (_has_explicit), wires narrow-phase inputs from model collision_*/heightfield_* buffers or placeholders, renames/uses collision_radius, and initializes backend with has_meshes=_has_explicit.
Examples & simulator
newton/_src/solvers/kamino/examples/rl/example_rl_bipedal.py, newton/_src/solvers/kamino/examples/rl/simulation.py
Added Heightfield terrain builder and pushable-ball scene; RigidBodySim API extended (terrain_fn, scene_callback, max_contacts_per_world); SimulatorFromNewton gains use_newton_collisions path that runs Newton collision and converts contacts for Kamino.
Tests
newton/_src/solvers/kamino/tests/test_geometry_mesh_heightfield.py, newton/_src/solvers/kamino/tests/test_geometry_unified.py
Added comprehensive end-to-end tests for mesh/heightfield collision and Newton→Kamino contact conversion; updated one test docstring to reference collision_radius.

Sequence Diagram(s)

sequenceDiagram
    participant NewtonModel as Newton Model
    participant ModelConv as ModelKamino.from_newton
    participant Pipeline as CollisionPipelineUnifiedKamino
    participant NarrowPhase as Narrow-phase Backend
    participant NewtonCD as Newton Collision (newton.collide)
    participant Converter as convert_contacts_newton_to_kamino
    participant KaminoSolver as Kamino Solver

    NewtonModel->>ModelConv: emit shape arrays (types, mesh/hfield buffers, AABBs, radii)
    ModelConv->>Pipeline: build GeometriesModel (includes collision/heightfield fields)
    Pipeline->>Pipeline: scan types -> set _has_explicit
    alt _has_explicit == true
        Pipeline->>NarrowPhase: provide collision_aabb_*/voxel_resolution/heightfield_* buffers
        NewtonCD->>Converter: (when using Newton collisions) produce contacts
        Converter->>KaminoSolver: convert to ContactsKamino
    else
        Pipeline->>NarrowPhase: provide placeholders for explicit inputs
    end
    KaminoSolver->>NarrowPhase: launch narrow-phase with collision_radius array
    NarrowPhase->>KaminoSolver: return contacts
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~55 minutes

Possibly related PRs

Suggested reviewers

  • camevor
  • eric-heiden
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 73.44% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'Mesh and Heightfield collision support for Kamino' directly and clearly summarizes the main objective of the changeset, which adds mesh and heightfield collision support to the Kamino solver across multiple files.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 31, 2026

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
newton/_src/geometry/types.py 66.66% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
newton/_src/solvers/kamino/_src/core/shapes.py (1)

626-699: ⚠️ Potential issue | 🟠 Major

Primitive–heightfield pairs can still allocate zero contact capacity.

The new GeoType.HFIELD handling only exists in the type_a == GeoType.HFIELD and mesh/convex branches. After the canonical sort, any primitive/heightfield pair that still enters one of the earlier primitive branches has no type_b == GeoType.HFIELD case and falls through to (0, 0). newton/_src/solvers/kamino/_src/core/conversions.py:235-260 uses this return value to size world_max_contacts, so the default ModelKamino.from_newton() path can still under-allocate and drop real contacts.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@newton/_src/solvers/kamino/_src/core/shapes.py` around lines 626 - 699, The
function that returns contact capacity for (type_a, type_b) can return (0,0) for
primitive vs HFIELD because only the final HFIELD and mesh branches handle
HFIELD; to fix, ensure any branch handling a primitive also checks for type_b ==
GeoType.HFIELD and returns the conservative capacity (e.g. _MESH_CONVEX_MAX, 0),
or perform a canonical swap early so GeoType.HFIELD is always handled by the
dedicated HFIELD branch; update the primitive branches (cases for
GeoType.SPHERE/CAPSULE/ELLIPSOID/CYLINDER/BOX and the mesh/convex branch) to
explicitly handle type_b == GeoType.HFIELD (returning _MESH_CONVEX_MAX, 0) so
ModelKamino.from_newton and conversions.py sizing no longer under-allocates.
🧹 Nitpick comments (1)
newton/_src/solvers/kamino/_src/core/model.py (1)

1324-1331: The private field coupling is safe and does not require fixes.

_shape_voxel_resolution is properly initialized during Model.finalize() in a loop that populates the array for all shapes, and it is always device-backed via wp.array(..., device=device). The field is guaranteed to be set before Kamino's from_newton() consumes it. While exposing a public accessor would reduce coupling, the current approach is functionally sound.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@newton/_src/solvers/kamino/_src/core/model.py` around lines 1324 - 1331, No
functional change required because Model.finalize() populates
model._shape_voxel_resolution before Kamino.from_newton() consumes it; if you
want a safety improvement, either add a short assertion at the start of
Kamino.from_newton() verifying model._shape_voxel_resolution is set and
device-backed (to fail fast with a clear message), or add a public accessor
property on Model (e.g., Model.shape_voxel_resolution) that returns the wp.array
and use that in from_newton() instead of the private attribute.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@newton/_src/solvers/kamino/examples/rl/example_rl_bipedal.py`:
- Around line 169-171: The _make_balls_fn function currently slices
BALL_POSITIONS with num_balls without validation leading to negative or
too-large inputs producing unexpected results; before using num_balls in the
slice, validate and clamp it to a safe range (e.g., clamp num_balls = max(0,
min(num_balls, min(5, len(BALL_POSITIONS)))) or alternatively raise a ValueError
for out-of-range values), and update any doc/help text if you change behavior so
callers know the enforced max; refer to the _make_balls_fn function and the
BALL_POSITIONS constant when making the change.

In `@newton/_src/solvers/kamino/examples/rl/simulation.py`:
- Around line 82-89: The single-world path can leave negative shape_world (wid
== -1) entries in Newton contact data which later feed
convert_contacts_newton_to_kamino; to fix, when use_newton_collisions is true
and newton_model.world_count == 1, sanitize the Newton contact/shape_world data
by replacing negative shape_world/wid values with 0 on the model/state/contacts
you store (referencing newton_model, self._newton_state, self._newton_contacts
and the converter convert_contacts_newton_to_kamino) before any conversion or
use — i.e., after assigning self._newton_state/self._newton_contacts (and before
creating ContactsKamino or calling convert_contacts_newton_to_kamino) iterate
the contact shape_world/wid entries and set negative values to 0.
- Around line 82-89: The Newton-CD branch currently sizes ContactsKamino using
newton_model.rigid_contact_max and ignores the configured cap, so update the
allocation logic in the use_newton_collisions branch (where self._newton_model,
self._newton_state, self._newton_contacts are set and per_world/world_max are
computed) to clamp per_world by the configured max_contacts_per_world (e.g.,
config.collision_detector.max_contacts_per_world or the corresponding variable
passed into the module) before building world_max and calling
ContactsKamino(capacity=world_max, ...); ensure the clamp applies per-world (use
min(rigid_contact_max-derived value, max_contacts_per_world)) so the configured
cap is honored for heightfield/mesh scenes that force
use_newton_collisions=True.

---

Outside diff comments:
In `@newton/_src/solvers/kamino/_src/core/shapes.py`:
- Around line 626-699: The function that returns contact capacity for (type_a,
type_b) can return (0,0) for primitive vs HFIELD because only the final HFIELD
and mesh branches handle HFIELD; to fix, ensure any branch handling a primitive
also checks for type_b == GeoType.HFIELD and returns the conservative capacity
(e.g. _MESH_CONVEX_MAX, 0), or perform a canonical swap early so GeoType.HFIELD
is always handled by the dedicated HFIELD branch; update the primitive branches
(cases for GeoType.SPHERE/CAPSULE/ELLIPSOID/CYLINDER/BOX and the mesh/convex
branch) to explicitly handle type_b == GeoType.HFIELD (returning
_MESH_CONVEX_MAX, 0) so ModelKamino.from_newton and conversions.py sizing no
longer under-allocates.

---

Nitpick comments:
In `@newton/_src/solvers/kamino/_src/core/model.py`:
- Around line 1324-1331: No functional change required because Model.finalize()
populates model._shape_voxel_resolution before Kamino.from_newton() consumes it;
if you want a safety improvement, either add a short assertion at the start of
Kamino.from_newton() verifying model._shape_voxel_resolution is set and
device-backed (to fail fast with a clear message), or add a public accessor
property on Model (e.g., Model.shape_voxel_resolution) that returns the wp.array
and use that in from_newton() instead of the private attribute.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: a9b75da9-ce53-4444-b594-7510aa768ef6

📥 Commits

Reviewing files that changed from the base of the PR and between 3284a2e and 88ea618.

📒 Files selected for processing (9)
  • newton/_src/geometry/types.py
  • newton/_src/solvers/kamino/_src/core/builder.py
  • newton/_src/solvers/kamino/_src/core/geometry.py
  • newton/_src/solvers/kamino/_src/core/model.py
  • newton/_src/solvers/kamino/_src/core/shapes.py
  • newton/_src/solvers/kamino/_src/geometry/unified.py
  • newton/_src/solvers/kamino/examples/rl/example_rl_bipedal.py
  • newton/_src/solvers/kamino/examples/rl/simulation.py
  • newton/_src/solvers/kamino/tests/test_geometry_mesh_heightfield.py

Comment thread newton/_src/solvers/kamino/examples/rl/example_rl_bipedal.py
Comment thread newton/_src/solvers/kamino/examples/rl/simulation.py
@preist-nvidia preist-nvidia added this to the 1.2 Release milestone Apr 7, 2026
@preist-nvidia preist-nvidia requested a review from vastsoun April 7, 2026 09:13
Copy link
Copy Markdown
Member

@vastsoun vastsoun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks @nvtw.

I have only minor comments, so I think this could be close to being mergable.

Comment thread newton/_src/solvers/kamino/_src/core/geometry.py Outdated
Comment thread newton/_src/solvers/kamino/examples/rl/simulation.py Outdated
@nvtw nvtw marked this pull request as ready for review April 8, 2026 10:19
@nvtw nvtw changed the title Draft: Mesh and Heightfield collision support for Kamino Mesh and Heightfield collision support for Kamino Apr 8, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
newton/_src/solvers/kamino/examples/rl/simulation.py (1)

862-863: ⚠️ Potential issue | 🟡 Minor

Docstring inconsistency: q_j_ref claims num_joint_coords but is shaped to num_joint_dofs.

The docstring says (num_worlds, num_joint_coords) but line 447 reshapes to (nw, njd) (DOF space), which is correct for PD position targets. The docstring should say num_joint_dofs.

📝 Proposed fix
     `@property`
     def q_j_ref(self) -> torch.Tensor:
-        """Joint position reference ``(num_worlds, num_joint_coords)`` for implicit PD."""
+        """Joint position reference ``(num_worlds, num_joint_dofs)`` for implicit PD."""
         return self._q_j_ref
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@newton/_src/solvers/kamino/examples/rl/simulation.py` around lines 862 - 863,
The docstring for the q_j_ref property incorrectly says "num_joint_coords" while
the stored shape is (nw, njd) in DOF space; update the q_j_ref docstring to
state "(num_worlds, num_joint_dofs)" (or use "num_joint_dofs" consistently) to
match the actual reshape and the attribute _q_j_ref so callers understand this
is DOF-space joint position references.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@newton/_src/solvers/kamino/examples/rl/simulation.py`:
- Around line 862-863: The docstring for the q_j_ref property incorrectly says
"num_joint_coords" while the stored shape is (nw, njd) in DOF space; update the
q_j_ref docstring to state "(num_worlds, num_joint_dofs)" (or use
"num_joint_dofs" consistently) to match the actual reshape and the attribute
_q_j_ref so callers understand this is DOF-space joint position references.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: a344b3df-189f-4690-a6fd-1533764b0858

📥 Commits

Reviewing files that changed from the base of the PR and between 3528ccb and 3eb1c29.

📒 Files selected for processing (7)
  • newton/_src/geometry/types.py
  • newton/_src/solvers/kamino/_src/core/geometry.py
  • newton/_src/solvers/kamino/_src/core/model.py
  • newton/_src/solvers/kamino/_src/geometry/unified.py
  • newton/_src/solvers/kamino/examples/rl/example_rl_bipedal.py
  • newton/_src/solvers/kamino/examples/rl/simulation.py
  • newton/_src/solvers/kamino/tests/test_geometry_unified.py
✅ Files skipped from review due to trivial changes (1)
  • newton/_src/solvers/kamino/tests/test_geometry_unified.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • newton/_src/solvers/kamino/_src/core/model.py
  • newton/_src/solvers/kamino/examples/rl/example_rl_bipedal.py

@nvtw nvtw requested a review from vastsoun April 8, 2026 11:14
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@newton/_src/solvers/kamino/_src/geometry/unified.py`:
- Around line 524-527: The current detection packs heightfields with meshes into
_has_explicit, causing NarrowPhase to be created with has_meshes=True even when
only heightfields exist; change the detection to compute two booleans: one (e.g.
_has_meshes) that checks for GeoType.MESH and GeoType.CONVEX_MESH, and another
(e.g. _has_heightfields) that checks for GeoType.HFIELD, then pass both
explicitly to NarrowPhase as has_meshes=_has_meshes and
has_heightfields=_has_heightfields so the mesh-mesh contact kernel is
enabled/disabled correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: d135857f-b5e9-4700-9b2a-7cad58dbe69e

📥 Commits

Reviewing files that changed from the base of the PR and between e8b3b62 and 272a8b8.

📒 Files selected for processing (1)
  • newton/_src/solvers/kamino/_src/geometry/unified.py

Comment thread newton/_src/solvers/kamino/_src/geometry/unified.py Outdated
vastsoun
vastsoun previously approved these changes Apr 13, 2026
Copy link
Copy Markdown
Member

@vastsoun vastsoun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @nvtw

@vastsoun vastsoun self-requested a review April 13, 2026 07:57
Copy link
Copy Markdown
Member

@vastsoun vastsoun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good, but I get SIGSEGV when running the unit tests on this branch. It seems to be the new Mesh + Heighfield UTs

@nvtw
Copy link
Copy Markdown
Member Author

nvtw commented Apr 13, 2026

Code looks good, but I get SIGSEGV when running the unit tests on this branch. It seems to be the new Mesh + Heighfield UTs

This should be addressed now

@chschuma-disney
Copy link
Copy Markdown
Contributor

Code looks good to me, just the following tests seem to fail:

  • test_01_sphere_on_flat_heightfield: AssertionError: 2 != 1 : Sphere-on-flat-heightfield should produce exactly 1 contact, got 2
  • test_04_multi_world_heightfield: AssertionError: 6 != 3 : Expected 3 contacts (1 per world), got 6

@nvtw
Copy link
Copy Markdown
Member Author

nvtw commented Apr 14, 2026

Code looks good to me, just the following tests seem to fail:

  • test_01_sphere_on_flat_heightfield: AssertionError: 2 != 1 : Sphere-on-flat-heightfield should produce exactly 1 contact, got 2
  • test_04_multi_world_heightfield: AssertionError: 6 != 3 : Expected 3 contacts (1 per world), got 6

Code looks good to me, just the following tests seem to fail:

  • test_01_sphere_on_flat_heightfield: AssertionError: 2 != 1 : Sphere-on-flat-heightfield should produce exactly 1 contact, got 2
  • test_04_multi_world_heightfield: AssertionError: 6 != 3 : Expected 3 contacts (1 per world), got 6

Indeed, should be fixed now

@nvtw nvtw enabled auto-merge April 14, 2026 14:21
@nvtw nvtw added this pull request to the merge queue Apr 14, 2026
Merged via the queue into newton-physics:main with commit f52ac3e Apr 14, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants