Runtime Python API Reference

The Python executorch.runtime module wraps the C++ ExecuTorch runtime. It can load and execute serialized .pte program files: see the Export to ExecuTorch Tutorial for how to convert a PyTorch nn.Module to an ExecuTorch .pte program file. Execution accepts and returns torch.Tensor values, making it a quick way to validate the correctness of the program. To run a program without PyTorch installed, see Without PyTorch below.

For detailed information on how APIs evolve and the deprecation process, please refer to the ExecuTorch API Life Cycle and Deprecation Policy.

Example usage:

from pathlib import Path

import torch
from executorch.runtime import Runtime, Program, Method

et_runtime: Runtime = Runtime.get()
program: Program = et_runtime.load_program(Path("/tmp/program.pte"))
print("Program methods:", program.method_names)
forward: Method = program.load_method("forward")

inputs = (torch.ones(2, 2), torch.ones(2, 2))
outputs = forward.execute(inputs)
print(f"Ran forward({inputs})")
print(f"  outputs: {outputs}")

Example output:

Program methods: {'forward'}
Ran forward((tensor([[1., 1.],
        [1., 1.]]), tensor([[1., 1.],
        [1., 1.]])))
  outputs: [tensor([[2., 2.],
        [2., 2.]])]

Example usage with ETDump generation:

Note: ETDump requires building ExecuTorch with event tracing enabled (CMake option EXECUTORCH_ENABLE_EVENT_TRACER=ON).

from pathlib import Path
import os

import torch
from executorch.runtime import Runtime, Program, Method

# Create program with etdump generation enabled
et_runtime: Runtime = Runtime.get()
program: Program = et_runtime.load_program(
    Path("/tmp/program.pte"),
    enable_etdump=True,
    debug_buffer_size=int(1e7),  # 10MB buffer to capture all debug info
)

# Load method and execute
forward: Method = program.load_method("forward")
inputs = (torch.ones(2, 2), torch.ones(2, 2))
outputs = forward.execute(inputs)

# Write etdump result to file
etdump_file = "/tmp/etdump_output.etdp"
debug_file = "/tmp/debug_output.bin"
program.write_etdump_result_to_file(etdump_file, debug_file)

# Check that files were created
print(f"ETDump file created: {os.path.exists(etdump_file)}")
print(f"Debug file created: {os.path.exists(debug_file)}")
print("Directory contents:", os.listdir("/tmp"))

Example output:

ETDump file created: True
Debug file created: True
Directory contents: ['program.pte', 'etdump_output.etdp', 'debug_output.bin']

Example usage with backend and operator introspection:

from executorch.runtime import Runtime

runtime = Runtime.get()

# Check available backends
backends = runtime.backend_registry.registered_backend_names
print(f"Available backends: {backends}")

# Check if a specific backend is available
if runtime.backend_registry.is_available("XnnpackBackend"):
    print("XNNPACK backend is available")

# List all registered operators
operators = runtime.operator_registry.operator_names
print(f"Number of registered operators: {len(operators)}")

Example output:

Available backends: ['XnnpackBackend', ...]  # Depends on your build configuration
XNNPACK backend is available
Number of registered operators: 247  # Depends on linked kernels

class executorch.runtime.Runtime(*, legacy_module)

An instance of the ExecuTorch runtime environment.

This can be used to concurrently load and execute any number of ExecuTorch programs and methods.

backend_registry

Registry for querying available hardware backends.

operator_registry

Registry for querying available operators/kernels.

static get()

Gets the Runtime singleton.

load_program(data, *, verification=<Verification.InternalConsistency: 1>, enable_etdump=False, debug_buffer_size=0, data_path=None, backend_options=None)

Loads an ExecuTorch program from a PTE binary.

Parameters:

• data – The binary program data to load. Can be a file path (str or Path), bytes/bytearray, or a file-like object.

• verification – Level of program verification to perform (Minimal or InternalConsistency). Default is InternalConsistency.

• enable_etdump – If True, enables ETDump profiling for runtime performance analysis. Default is False.

• debug_buffer_size – Size of the debug buffer in bytes for ETDump data. Only used when enable_etdump=True. Default is 0.

• data_path – Path to a .ptd file holding data the program keeps outside the PTE, such as the weights a CUDA export writes to a separate file.

• backend_options – Load-time options for each backend, keyed by backend name, used by every method loaded from this program.

Returns:

The loaded Program instance.

class executorch.runtime.BackendRegistry(legacy_module)

The registry of backends that are available to the runtime.

get_option(backend_name, options)

Returns the backend’s current values for the keys of options, whose values set the expected types.

is_available(backend_name)

Checks if a specific backend is available in the runtime.

Parameters:

backend_name – The name of the backend to check (e.g., “XnnpackBackend”).

Returns:

True if the backend is available, False otherwise.

property registered_backend_names

Returns the names of all registered backends as a list of strings.

set_option(backend_name, options)

Sets process-wide options of a backend, for example set_option("XnnpackBackend", {"weight_cache_enabled": True}).

class executorch.runtime.OperatorRegistry(legacy_module)

The registry of operators that are available to the runtime.

property operator_names

Returns the names of all registered operators as a set of strings.

class executorch.runtime.Program(program, data)

An ExecuTorch program, loaded from binary PTE data.

This can be used to load the methods/models defined by the program.

load_method(name)

Loads a method from the program.

Memory-planned buffers are allocated differently depending on where the program placed them. A method whose buffers are all on the host shares one set of arenas with every other host-only method of this program, sized to the largest of them, so two such methods overwrite each other’s intermediate values. By default execute copies its outputs out, so that sharing does not change what it returns. With clone_outputs=False the outputs share that memory, so the next execution of any method of this program may overwrite them. A method with any buffer placed on an accelerator gets its own arenas instead, claimed when that method is first loaded and owned by that method until the method itself is released. This program caches every method it loads, so in normal use those arenas live as long as the program does. A missing or exhausted accelerator therefore fails that one load rather than the whole program, but whatever it does claim stays claimed for every method loaded after it.

A program exported with share_mutable_buffers=True relies on every method reading its shared mutable state from one allocation. A method with accelerator-placed buffers gets its own arenas instead, so it also gets its own copy of that state and will not observe writes made through another method. Export without share_mutable_buffers if a method needs both accelerator memory and shared state.

Parameters:

name – The name of the method to load.

Returns:

The loaded method.

property method_names

Returns method names of the Program as a set of strings.

class executorch.runtime.Method(method)

An ExecuTorch method, loaded from a Program. This can be used to execute the method with inputs.

execute(inputs, clone_outputs=True)

Executes the method with the given inputs.

Outputs are copies, except for an output bound with set_output(), which is returned as the bound tensor and overwritten by later calls. An input that is not memory planned (e.g. exported with alloc_graph_input=False) is used in place, so a mutation of it is written into the tensor passed in for it, while an output that is that input is still returned as a copy. A memory planned input is copied into the method, so the tensor passed in for it is not updated.

Parameters:

• inputs – A sequence of input values, typically torch.Tensor objects. Without PyTorch, pass NumPy arrays, other buffers such as memoryview, DLPack objects such as CuPy arrays, or ExecuTorchResult values. Scalar inputs take Python numbers.

• clone_outputs – If False, the outputs share the method’s memory and a later execution may overwrite them, which avoids a copy.

Returns:

A list of output values, typically torch.Tensor objects. Without PyTorch, tensor outputs are ExecuTorchResult objects.

property metadata

Gets the metadata for the method.

The metadata includes information about input and output specifications, such as tensor shapes, data types, and memory requirements.

Returns:

The MethodMeta object containing method specifications.

set_output(tensor, index)

Binds output index to tensor’s storage for every later call.

The method then writes that output straight into tensor and returns it uncloned (viewed at the written shape when that differs) instead of a copy of its own storage. Every later call overwrites tensor, so clone a returned output to keep it.

The output must not be memory planned (export with alloc_graph_output=False), must not be returned at another output index too, and must not be an input, which already lives in the tensor passed in for it. That last case is only known once inputs have been set: a binding made before the first call is removed by that call, which raises. tensor must have the output’s dtype and device, be contiguous, hold at least as many bytes as the output’s largest shape, and not share memory with another bound output or with an input. The output itself must be contiguous, not e.g. channels-last. A rejected binding raises ValueError. The ATen bindings don’t support it.

Parameters:

• tensor – The tensor that stores the output from now on.

• index – The index of the output to bind.

class executorch.runtime.ExecuTorchResult

A tensor result of an ExecuTorch method, returned when PyTorch is not used. Read a CPU result through the buffer protocol, NumPy, or DLPack, and a CUDA result through DLPack or to_host().

Warning

This API is experimental and subject to change without notice.

property device

N”.

Type:

Where the data lives

Type:

“cpu” or “cuda

property dtype

Name of the dtype, such as “float32” or “bfloat16”, as torch spells it.

static from_host(source: object, device_index: SupportsInt | SupportsIndex = 0) → executorch.extension.pybindings._C.ExecuTorchResult

Copy a dense CPU buffer, or a CPU ExecuTorchResult, to a CUDA device, for use as a method input. Requires bindings built with CUDA support.

property nbytes

Size of the data in bytes.

property shape

Size of each dimension.

property strides

Byte step of each dimension.

to_host(self: executorch.extension.pybindings._C.ExecuTorchResult) → executorch.extension.pybindings._C.ExecuTorchResult

Return a copy of this result in CPU memory.

Without PyTorch

The runtime also works when PyTorch is not installed. Then Method.execute takes NumPy arrays as inputs and returns ExecuTorchResult values, which NumPy can read. For a model that adds its two inputs:

import numpy as np
from executorch.runtime import Runtime

method = Runtime.get().load_program("model.pte").load_method("forward")
out = method.execute([np.ones((2, 2), np.float32), np.ones((2, 2), np.float32)])[0]
print(out.dtype)          # float32
result = np.asarray(out)  # a NumPy view of the output, no copy
print(result.tolist())    # [[2.0, 2.0], [2.0, 2.0]]

Other buffers, such as memoryview, work as inputs too. NumPy has no bfloat16 type, so it reads a bfloat16 result as 16-bit integers that hold the raw bits.

For data on a GPU, use DLPack, the standard way to share tensors between Python libraries. This needs ExecuTorch built with CUDA support. Pass a CuPy array to a method that takes CUDA inputs. Read a CUDA result with cupy.from_dlpack(out), or copy it to the CPU with out.to_host(). ExecuTorchResult.from_host() copies CPU data to a CUDA device.

To get ExecuTorchResult values even when PyTorch is installed, set EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT=executorch before importing ExecuTorch.

CUDA streams

With ExecuTorch built with CUDA support, executorch.runtime.cuda makes the CUDA and TensorRT delegates of any number of programs run on one stream you choose, for example a CUDA green-context stream. It does not need PyTorch. The module documentation below gives an example and the rules.

CUDA streams for running delegates on a stream the caller chooses.

Warning

This API is experimental and subject to change without notice.

Example:

from executorch.runtime import cuda

s = cuda.Stream()
with cuda.stream(s):
    outputs = first.execute(inputs)  # e.g. a TensorRT delegate
    second.execute(outputs)  # e.g. a CUDA delegate in another program
s.close()

Inside the block, the CUDA and TensorRT delegates queue their work on s instead of their own streams, and so do a program’s copies between host and GPU memory. A call then waits for s. If s is non-blocking, as a Stream is, work that other streams still have queued keeps running. A blocking s, such as a cupy.cuda.Stream() with default flags, also waits for work on the legacy default stream. Some steps still wait for more: the CUDA delegate copies an output whose memory layout has to change through the legacy default stream, and loading a method’s CUDA weights copies each one with a plain cudaMemcpy, which waits for the legacy default stream and for blocking streams. Weights an older export keeps in one blob wait for the whole GPU. A call with GPU inputs or outputs, and ExecuTorchResult.to_host(), also wait for the whole GPU, so GPU data written on other streams stays safe.

Limits for now: a stream known to be on a CUDA device other than 0 is refused, so is the legacy default stream (handle 0, or 1 for cudaStreamLegacy), and the CUDA delegate refuses a method that uses CUDA graphs inside a block. A Stream is checked on the device it was created on, whatever wrapper selects it. Any other stream is refused if its wrapper reports another device. In builds with CUDA 13 or newer, ExecuTorch also asks CUDA for the stream’s device, so it is refused even if its wrapper does not say so, except while the stream captures a CUDA graph, when CUDA cannot be asked. This module does not need PyTorch; torch.cuda.Stream works because it has __cuda_stream__.

As in PyTorch, the selection belongs to the thread, not to an asyncio task, so do not await or yield inside a block. Blocks end on the thread that started them, innermost first. A block also ends when its thread ends, or when its scope object is dropped on that thread. A scope object dropped on another thread cannot end the block, so the stream stays selected until the thread that started the block ends. A Stream cannot be closed while any block selects its handle, and a closed one is refused. A stream given only by its handle, for example through ExternalStream, must stay alive while selected: CUDA has no safe way to tell a destroyed handle from a live one, so this module cannot refuse one, and using one can crash the process.

class executorch.runtime.cuda.ExternalStream(stream_ptr, device=None)

A CUDA stream someone else created and keeps alive; never destroyed here.

device is the CUDA device it belongs to, if known, as an index or a string such as "cuda:1"; stream() refuses a device other than 0.

class executorch.runtime.cuda.Stream(device=None)

A non-blocking CUDA stream, destroyed by close() or garbage collection.

device is the CUDA device to create it on; None means the current one.

close()

Destroys the stream. Refused while a block selects its handle.

property cuda_stream

The stream handle, or 0 once closed.

executorch.runtime.cuda.stream(s)

Makes s the CUDA stream for delegates run by this thread in the block.

An exit out of order is refused, and that block then ends when the blocks started after it end. The module docstring lists the other rules.

Parameters:

s – A Stream, an ExternalStream, or any object with __cuda_stream__.

Returns:

A context manager whose as value is s.