Sensible default and the wiring looks right — I like that the per-host case in local_runner.rs splits the machine share by n_hosts and that pipeline-process fallback in parse_config runs before the logger, using eprintln!.

A few things worth a look before you merge:

1. available_memory_mb() semantics don't match the docstring / user doc.
The docstring says "min of the host's available memory and the cgroup budget", and docs/operations/memory.md says "min(host memory size, container memory limit)". The implementation, however, uses system.available_memory() (currently-free host memory) and limits.free_memory (currently-free within the cgroup) — those are point-in-time free amounts, not the host size or the cgroup limit. In a container that started with the pipeline as the first process the two are close, but in a co-tenant pod or a busy host they can diverge quite a bit and produce non-deterministic startup limits across restarts.

If you want a stable ceiling that survives across restarts, prefer total_memory for both:

let mut available = system.total_memory();
if let Some(limits) = system.cgroup_limits() {
    available = available.min(limits.total_memory);
}

Otherwise, please update the docstring and memory.md to say "currently available" rather than "size / limit", so operators know the default can drift restart-to-restart.

2. sysinfo MB vs MiB. sysinfo returns bytes; you divide by 1_000_000 (decimal MB). The max_rss_mb / memory_mb_max fields have historically been treated as MB throughout the config, so this is internally consistent — just flagging in case anyone thought it was MiB.

3. Cosmetic: System::new_with_specifics(RefreshKind::new().with_memory()) avoids allocating the full snapshot, but the cost here is negligible.

4. running_in_kubernetes() doc. The comment references pipeline-manager rejecting user-supplied KUBERNETES_* variables — worth linking to that validation code (or referencing the type/function) so future readers can confirm the env-var isn't user-forgeable.

5. Test coverage. reports_nonzero_memory is Linux-only and doesn't exercise the cgroup path or the mb == 0 fallback. A unit test that injects a mock available/limit pair through a thin seam would guard the min-selection logic. Not blocking.

Not a blocker overall; the heuristic is fine for "no config supplied" mode. Approving on the assumption that (1) is either a docstring fix or a total_memory swap.

Thanks for addressing (1) — swapping to total_memory() + limits.total_memory gives the stable ceiling I was after, and the parse_config fallback in server.rs now runs before the logger via eprintln!. The controller warning split (running_in_kubernetes() → pod-limit wording vs local info-level) reads well too.

One small naming/docstring inconsistency to clean up before/after merge:

• The function is called available_memory_mb and its docstring says "min of the host's available memory and the cgroup budget", but the implementation uses system.total_memory() and limits.total_memory — i.e. installed RAM and the cgroup limit, not currently-free memory. Consider renaming to total_memory_mb (matches sysinfo's terminology) and updating the docstring plus the call sites in local_runner.rs / server.rs. As written, a future reader will assume this reports free memory and reach for it in places where it does not do what they want.

• docs/operations/memory.md: "min(host memory size, container memory limit)" is now accurate — good.

Non-blocking; APPROVE stands.

I think it is a good idea to always have (1) a memory limit on the pipeline, and (2) the pipeline knowing this limit and doing all in its ability to stay under it to not cause an OOM. Whether the memory limit (1) is enforced depends on the form factor.

In the act of keeping under this limit, ideally performance degrades and a warning bubbles up saying "limited memory is causing large amount of spillover to disk, negatively impacting performance" or even quantifying the negative impact. It might even suggest a new memory limit.

I'm not sure if using the host or container memory size as the default is the way to go, or instead having just a default memory size limit (e.g., 2 GiB). The latter would be more predictable, especially if the underlying cluster on which is scheduled is heterogeneous / has other pipelines running in it. When you create a new pipeline, the amount of resources you set for it could be a mandatory thing to think about (e.g., the same as for AWS Lambda when you spin up workers).

I think @ryzhyk / @blp would be best to chime in on this.

Follow-up on tip b48c34e5: rebase onto main plus my non-blocking rename nit addressed — available_memory_mb is now total_memory_megabyte, with the docstring clarified ("in megabytes (MB)"). Everything else unchanged from the reviewed tip. APPROVE stands.

Sensible default and the wiring looks right — I like that the per-host case in local_runner.rs splits the machine share by n_hosts and that pipeline-process fallback in parse_config runs before the logger, using eprintln!.

A few things worth a look before you merge:

1. available_memory_mb() semantics don't match the docstring / user doc.
The docstring says "min of the host's available memory and the cgroup budget", and docs/operations/memory.md says "min(host memory size, container memory limit)". The implementation, however, uses system.available_memory() (currently-free host memory) and limits.free_memory (currently-free within the cgroup) — those are point-in-time free amounts, not the host size or the cgroup limit. In a container that started with the pipeline as the first process the two are close, but in a co-tenant pod or a busy host they can diverge quite a bit and produce non-deterministic startup limits across restarts.

If you want a stable ceiling that survives across restarts, prefer total_memory for both:

let mut available = system.total_memory();
if let Some(limits) = system.cgroup_limits() {
    available = available.min(limits.total_memory);
}

Otherwise, please update the docstring and memory.md to say "currently available" rather than "size / limit", so operators know the default can drift restart-to-restart.

2. sysinfo MB vs MiB. sysinfo returns bytes; you divide by 1_000_000 (decimal MB). The max_rss_mb / memory_mb_max fields have historically been treated as MB throughout the config, so this is internally consistent — just flagging in case anyone thought it was MiB.

3. Cosmetic: System::new_with_specifics(RefreshKind::new().with_memory()) avoids allocating the full snapshot, but the cost here is negligible.

4. running_in_kubernetes() doc. The comment references pipeline-manager rejecting user-supplied KUBERNETES_* variables — worth linking to that validation code (or referencing the type/function) so future readers can confirm the env-var isn't user-forgeable.

5. Test coverage. reports_nonzero_memory is Linux-only and doesn't exercise the cgroup path or the mb == 0 fallback. A unit test that injects a mock available/limit pair through a thin seam would guard the min-selection logic. Not blocking.

Not a blocker overall; the heuristic is fine for "no config supplied" mode. Approving on the assumption that (1) is either a docstring fix or a total_memory swap.

Re-review at 8c8e8d3a — APPROVE.

Two things landed since my last comment:

1. The memory_memory_max=1024 typo in python/tests/workloads/test_now.py is fixed to memory_mb_max=1024. That was the real issue on the last commit.
2. Two small stacked commits: 5d17958e (Balancer::set_hint returns NotRegisteredWithBalancer instead of panicking on unregistered streams) and 8c8e8d3a (compiler wraps the balancer-hint block in if circuit.adaptive_joins_enabled() { ... }, with DBSPHandle::adaptive_joins_enabled() / Runtime::adaptive_joins() exposing the dev-tweaks flag and a warning when hints are skipped). Both belt-and-suspenders against the panic path — I already looked at the same pattern in my 4d62f1c1 review and it's still fine.

Apologies again for the stale-payload mishap on review 4741092000; the actual finding was the typo which is now addressed.