Skip to content

Explainer: image input #1638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kvark merged 2 commits into from
Apr 15, 2021
Merged

Explainer: image input #1638

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
36cd679
Explainer: image input
kainino0x Apr 15, 2021
40d7b4e
address comments
kainino0x Apr 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions explainer/index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -793,6 +793,44 @@ Some alternatives are mentioned in issue [#747](https://github.com/gpuweb/gpuweb

## Image, Video, and Canvas input ## {#image-input}

Issue: Exact API still in flux as of this writing.

WebGPU is largely isolated from the rest of the Web platform, but has several interop points.
One of these is image data input into the API.
Aside from the general data read/write mechanisms (`writeTexture`, `writeBuffer`, and `mapAsync`),
data can also come from `<img>`/`ImageBitmap`, canvases, and videos.
There are many use-cases that require these, including:

- Initializing textures from encoded images (JPEG, PNG, etc.)
- Rendering text with 2D canvas for use in WebGPU.
- Video element and video camera input for image processing, ML, 3D scenes, etc.

There are two paths:

- `copyExternalImageToTexture()` copies color data from a sub-rectangle of an
image/video/canvas object into an equally-sized sub-rectangle of a `GPUTexture`.
The input data is captured at the moment of the call.
- `importTexture()` takes a video or canvas and creates a `GPUExternalTexture` object which *can*
provide direct read access to an underlying resource if it exists on the (same) GPU already,
avoiding unnecessary copies or CPU-GPU bandwidth.
This is typically true of hardware-decoded videos and most canvas elements.

Issue: Update both names to whatever we settle on.

### GPUExternalTexture ### {#image-input-external-texture}

A `GPUExternalTexture` is a sampleable texture object which can be used in similar ways to normal
sampleable `GPUTexture` objects.
In particular, it can be bound as a texture resource to a shader and used directly from the GPU:
when it is bound, additional metadata is attached that allows WebGPU to "automagically"
transform the data from its underlying representation (e.g. YUV) to RGB sampled data.

A `GPUExternalTexture` represents a particular imported image, so the underlying data must not
change after import, either from internal (WebGPU) or external (Web platform) access.

Issue:
Describe how this is achieved for video element, VideoFrame, canvas element, and OffscreenCanvas.


## Canvas Output ## {#canvas-output}

Expand Down