Skip to content

Scene.render_to_depth_image() returns metric depth instead of the documented normalized [0,1] range (regression since 0.19.0) #7611

Open

Description

@airalcorn2

Checklist

Describe the issue

Summary from Claude Code.

Scene.render_to_depth_image()'s own docstring states: "Pixels range from 0.0 (near plane) to 1.0 (far plane)." On 0.20.0 this is no longer true — depth values are raw/metric distances, and the "no hit" sentinel is inf instead of 1.0. This breaks any code following the documented contract, including Open3D's own example (examples/python/visualization/mouse_and_point_coord.py), which feeds this depth directly into Camera.unproject(x, y, depth, width, height) — that function still expects normalized depth, so the combination now produces wrong/degenerate world coordinates for mouse-picking.

Root cause

Commit 4216be84 ("3DGS rendering", PR #7490, merged 2026-05-16 by @ssheorey) changed FilamentScene::RenderToDepthImage:

 void FilamentScene::RenderToDepthImage(
         std::function<void(std::shared_ptr<geometry::Image>)> callback) {
     auto view = views_.begin()->second.view.get();
-    renderer_.RenderToDepthImage(view, this, callback);
+    renderer_.RenderToDepthImage(view, this, callback, true /*linear depth*/);
 }

This is the exact function backing Scene.render_to_depth_image() in Python. The PR added 3D Gaussian Splat rendering support and, as part of compositing GS depth with Filament's native depth, needed linear (metric) depth rather than normalized NDC depth — but the change was applied unconditionally to this method's only public entry point, which every other caller (including GUI mouse-picking, Open3D's own example mouse_and_point_coord.py, and our app) still relies on for normalized [0,1] depth. The docstring was never updated to reflect the new contract, and Camera.unproject() wasn't updated to accept the new depth format either.

Confirmed this is the sole cause: it's the only commit between v0.19.0 and v0.20.0 that touches RenderToDepthImage, and it exactly matches the observed break (metric values, inf sentinel instead of 1.0).

Steps to reproduce the bug

"""Minimal repro: Scene.render_to_depth_image() does not return normalized depth.

Scene.render_to_depth_image()'s own docstring says:
    "Pixels range from 0.0 (near plane) to 1.0 (far plane)"

Observed on open3d==0.20.0 (Linux): depth values are not in [0, 1] -- they look like
raw/metric distances (tens of units), and the "no hit" sentinel is `inf` rather than
`1.0`. This breaks any code that follows the documented contract, e.g. feeding the
returned depth directly into Camera.unproject(x, y, depth, width, height) (the
pattern used in Open3D's own example, mouse_and_point_coord.py) -- which still
expects normalized depth -- produces wrong/degenerate world coordinates.

Confirmed with open3d==0.19.0: depth values ARE in [0.995, 1.0] for this scene (as
expected -- a box ~4-5m away, rendered with a 60-degree vertical FOV, so most of the
depth range maps close to the far plane), and the far-plane sentinel is exactly 1.0.
So this is a regression introduced between 0.19.0 and 0.20.0, not a change in this
script's own logic.

Run with: python repro_depth_bug.py
"""

import threading
import time

import numpy as np
import open3d as o3d
from open3d.visualization import gui, rendering


def main():
    print(f"open3d version: {o3d.__version__}")

    app = gui.Application.instance
    app.initialize()

    w = app.create_window("render_to_depth_image depth-format repro", 640, 480)

    widget = gui.SceneWidget()
    widget.scene = rendering.Open3DScene(w.renderer)
    w.add_child(widget)

    def on_layout(layout_context):
        widget.frame = w.content_rect

    w.set_on_layout(on_layout)

    # A single box a few meters in front of the camera -- simple, unambiguous depth
    # structure, no external data files needed.
    box = o3d.geometry.TriangleMesh.create_box(1, 1, 1)
    box.translate([-0.5, -0.5, 4.5])
    box.compute_vertex_normals()
    mat = rendering.MaterialRecord()
    mat.shader = "defaultLit"
    widget.scene.add_geometry("box", box, mat)

    bbox = widget.scene.bounding_box
    widget.setup_camera(60.0, bbox, bbox.get_center())

    def check_depth():
        def depth_callback(depth_image):
            arr = np.asarray(depth_image)
            print(f"depth_image shape={arr.shape} dtype={arr.dtype}")
            print(f"min={arr.min()} max={arr.max()}")
            center = arr[arr.shape[0] // 2, arr.shape[1] // 2]
            corner = arr[2, 2]
            print(f"center pixel (hits the box) depth={center}")
            print(f"corner pixel (background/far plane) depth={corner}")
            print()
            if (0.0 <= arr.min()) and (arr.max() <= 1.0):
                print("OK: depth values are within the documented [0.0, 1.0] range.")
            else:
                print(
                    "BUG: depth values are NOT within the documented [0.0, 1.0] "
                    "range (docstring: 'Pixels range from 0.0 (near plane) to 1.0 "
                    "(far plane)')."
                )
            app.quit()

        widget.scene.scene.render_to_depth_image(depth_callback)

    def scheduler():
        time.sleep(1.5)
        app.post_to_main_thread(w, check_depth)

    threading.Thread(target=scheduler, daemon=True).start()
    app.run()


if __name__ == "__main__":
    main()

Error message

No response

Expected behavior

Expected (open3d==0.19.0): min=0.867 max=1.0, far-plane pixel depth=1.0.
Actual (open3d==0.20.0): min=0.75 max=inf, far-plane pixel depth=inf.

Open3D, Python and System information

- Operating system: Fedora Linux 44
- Python version: Python 3.11.18
- Open3D version: 0.20.0
- System architecture: Vulkan backend, Intel Iris Xe / Mesa
- Is this a remote workstation?: No
- How did you install Open3D?: pip
- Compiler version (if built from source): N/A

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugNot a build issue, this is likely a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions